You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yu-iskw <gi...@git.apache.org> on 2015/03/30 15:40:13 UTC
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
GitHub user yu-iskw opened a pull request:
https://github.com/apache/spark/pull/5267
[SPARK-6517][mllib] Implement the Algorithm of Hierarchical Clustering
I implemented a hierarchical clustering algorithm again. This PR doesn't include examples, documentation and spark.ml APIs. I am going to send another PRs later.
https://issues.apache.org/jira/browse/SPARK-6517
- This implementation based on a bi-sectiong K-means clustering.
- It derives from the @freeman-lab 's implementation
- The basic idea is not changed from the previous version. (#2906)
- However, It is 1000x faster than the previous version through parallel processing.
Thank you for your great cooperation, RJ Nowling(@rnowling), Jeremy Freeman(@freeman-lab), Xiangrui Meng(@mengxr) and Sean Owen(@srowen).
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yu-iskw/spark new-hierarchical-clustering
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5267.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5267
----
commit af0f65bb4726315c076b827d37276616c5218010
Author: Yu ISHIKAWA <yu...@gmail.com>
Date: 2015-03-30T11:29:12Z
[SPARK-6517][mllib] Implement the Algorithm of Hierarchical Clustering
Thank you for your great cooperation, RJ Nowling(@rnowling), Jeremy Freeman(@freeman-lab), Xiangrui Meng(@mengxr) and Sean Owen(@srowen).
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155174569
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42820998
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
--- End diff --
* better to use bit shift instead of `math.pow`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42820995
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
--- End diff --
need a more descriptive name
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-150270450
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r44321863
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,489 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.mllib.linalg.{BLAS, Vector, Vectors}
+import org.apache.spark.mllib.util.MLUtils
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage.StorageLevel
+
+/**
+ * A bisecting k-means algorithm based on the paper "A comparison of document clustering techniques"
+ * by Steinbach, Karypis, and Kumar, with modification to fit Spark.
+ * The algorithm starts from a single cluster that contains all points.
+ * Iteratively it finds divisible clusters on the bottom level and bisects each of them using
+ * k-means, until there are `k` leaf clusters in total or no leaf clusters are divisible.
+ * The bisecting steps of clusters on the same level are grouped together to increase parallelism.
+ * If bisecting all divisible clusters on the bottom level would result more than `k` leaf clusters,
+ * larger clusters get higher priority.
+ *
+ * @param k the desired number of leaf clusters (default: 4). The actual number could be smaller if
+ * there are no divisible leaf clusters.
+ * @param maxIterations the max number of k-means iterations to split clusters (default: 20)
+ * @param minDivisibleClusterSize the minimum number of points (if >= 1.0) or the minimum proportion
+ * of points (if < 1.0) of a divisible cluster (default: 1)
+ * @param seed a random seed (default: hash value of the class name)
+ *
+ * @see [[http://glaros.dtc.umn.edu/gkhome/fetch/papers/docclusterKDDTMW00.pdf
+ * Steinbach, Karypis, and Kumar, A comparison of document clustering techniques,
+ * KDD Workshop on Text Mining, 2000.]]
+ */
+@Since("1.6.0")
+@Experimental
+class BisectingKMeans private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var minDivisibleClusterSize: Double,
+ private var seed: Long) extends Logging {
+
+ import BisectingKMeans._
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(4, 20, 1.0, classOf[BisectingKMeans].getName.##)
+
+ /**
+ * Sets the desired number of leaf clusters (default: 4).
+ * The actual number could be smaller if there are no divisible leaf clusters.
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ require(k > 0, s"k must be positive but got $k.")
+ this.k = k
+ this
+ }
+
+ /**
+ * Gets the desired number of leaf clusters.
+ */
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the max number of k-means iterations to split clusters (default: 20).
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ require(maxIterations > 0, s"maxIterations must be positive but got $maxIterations.")
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /**
+ * Gets the max number of k-means iterations to split clusters.
+ */
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the minimum number of points (if >= `1.0`) or the minimum proportion of points
+ * (if < `1.0`) of a divisible cluster (default: 1).
+ */
+ @Since("1.6.0")
+ def setMinDivisibleClusterSize(minDivisibleClusterSize: Double): this.type = {
+ require(minDivisibleClusterSize > 0.0,
+ s"minDivisibleClusterSize must be positive but got $minDivisibleClusterSize.")
+ this.minDivisibleClusterSize = minDivisibleClusterSize
+ this
+ }
+
+ /**
+ * Gets the minimum number of points (if >= `1.0`) or the minimum proportion of points
+ * (if < `1.0`) of a divisible cluster.
+ */
+ @Since("1.6.0")
+ def getMinDivisibleClusterSize: Double = minDivisibleClusterSize
+
+ /**
+ * Sets the random seed (default: hash value of the class name).
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Gets the random seed.
+ */
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm.
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ if (input.getStorageLevel == StorageLevel.NONE) {
+ logWarning(s"The input RDD ${input.id} is not directly cached, which may hurt performance if"
+ + " its parent RDDs are also not cached.")
+ }
+ val d = input.map(_.size).first()
+ logInfo(s"Feature dimension: $d.")
+ // Compute and cache vector norms for fast distance computation.
+ val norms = input.map(v => Vectors.norm(v, 2.0)).persist(StorageLevel.MEMORY_AND_DISK)
+ val vectors = input.zip(norms).map { case (x, norm) => new VectorWithNorm(x, norm) }
+ var assignments = vectors.map(v => (ROOT_INDEX, v))
+ var activeClusters = summarize(d, assignments)
+ val rootSummary = activeClusters(ROOT_INDEX)
+ val n = rootSummary.size
+ logInfo(s"Number of points: $n.")
+ logInfo(s"Initial cost: ${rootSummary.cost}.")
+ val minSize = if (minDivisibleClusterSize >= 1.0) {
+ math.ceil(minDivisibleClusterSize).toLong
+ } else {
+ math.ceil(minDivisibleClusterSize * n).toLong
+ }
+ logInfo(s"The minimum number of points of a divisible cluster is $minSize.")
+ var inactiveClusters = mutable.Seq.empty[(Long, ClusterSummary)]
+ val random = new Random(seed)
+ var numLeafClustersNeeded = k - 1
+ var level = 1
+ while (activeClusters.nonEmpty && numLeafClustersNeeded > 0 && level < 63) {
--- End diff --
We should use `val levelLimit = log10(Long.MaxValue) / Log10(2)`, instead of `63`. It's a minor issue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155231942
@freeman-lab thank you for your great support!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151334153
**[Test build #44387 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44387/consoleFull)** for PR 5267 at commit [`165e191`](https://github.com/apache/spark/commit/165e191755bb641f10c604f1ec214248f5a104e7).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class BisectingKMeansModel @Since(\"1.6.0\") (`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42821001
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = BisectingKMeans.initData(input).cache()
+ while (clusterStats.size < maxAllNodesInTree && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ val leafClusters = BisectingKMeans.summarizeClusters(data)
--- End diff --
* `leafCluster` -> `leafClusterStats`
* add a TODO to remove non-leaf cluster stats from the broadcast data
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-88280539
Hi @jkbradley, @mengxr,
Would you review this PR, especially from the point of view about the design? It supports a saving and loading model functions in both of Scala and Python. I want to make sure that my implementation is suitable in terms of the concept of save/load.
Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42821578
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = BisectingKMeans.initData(input).cache()
+ while (clusterStats.size < maxAllNodesInTree && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ val leafClusters = BisectingKMeans.summarizeClusters(data)
+ val dividableLeafClusters = leafClusters.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusters
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ else {
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided =
+ BisectingKMeans.divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = BisectingKMeans.updateClusterIndex(data, divided).cache()
+ rddArray = rddArray ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (rddArray.length > 1) {
+ val head = rddArray.head
+ head.unpersist()
+ rddArray = rddArray.filterNot(_.hashCode() == head.hashCode())
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+ }
+ }
+ // unpersist kept RDDs
+ rddArray.foreach(_.unpersist())
+ // create a map of cluster node with their criterions
+ val nodes = BisectingKMeans.createClusterNodes(data, clusterStats)
+
+ // build a cluster tree by Map class which is expressed
+ logInfo(s"Building the cluster tree is started in ${sc.appName}")
+ val root = BisectingKMeans.buildTree(nodes, BisectingKMeans.ROOT_INDEX_KEY, this.k)
+ if (root.isEmpty) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusterStats")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ logInfo(s"Elapsed Time for ${this.getClass.getSimpleName} Training: ${finishTime} [sec]")
+
+ // make a bisecting kmeans model
+ val model = new BisectingKMeansModel(root.get)
+ val leavesNodes = model.getClusters
+ if (leavesNodes.length < this.k) {
+ logWarning(s"# clusters is less than you want: ${leavesNodes.length} / ${k}")
+ }
+ model
+ }
+}
+
+
+private[clustering] object BisectingKMeans {
+
+ val ROOT_INDEX_KEY: BigInt = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ def findClosestCenter(metric: (BV[Double], BV[Double]) => Double)
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)
+ closestIndex
+ }
+
+ /**
+ * Summarizes data by each cluster as Map
+ *
+ * @param data pairs of point and its cluster index
+ */
+ def summarizeClusters(data: RDD[(BigInt, BV[Double])]): Map[BigInt, BisectingClusterStat] = {
+
+ val stats = data.mapPartitions { iter =>
+ // calculate the accumulation of the all point in a partition and count the rows
+ val map = mutable.Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx: BigInt, point: BV[Double]) =>
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(idx, (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size)))
+ map(idx) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ map.toIterator
+ }.reduceByKey { case ((sum1, n1, sumOfSquares1), (sum2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sum1 + sum2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.collect().toMap
+
+ stats.map {case (i, stat) => i -> new BisectingClusterStat(stat._2.toLong, stat._1, stat._3)}
+ }
+
+ /**
+ * Assigns the initial cluster index id to all data
+ */
+ def initData(data: RDD[Vector]): RDD[(BigInt, BV[Double])] = {
+ data.map { v: Vector => (BisectingKMeans.ROOT_INDEX_KEY, v.toBreeze)}
+ }
+
+ /**
+ * Gets the initial centers for bisect k-means
+ *
+ * @param data pairs of point and its cluster index
+ * @param stats pairs of cluster index and cluster statistics
+ */
+ def initNextCenters(
+ data: RDD[(BigInt, BV[Double])],
+ stats: Map[BigInt, BisectingClusterStat]): Map[BigInt, BV[Double]] = {
+
+ // Since the combination sampleByKey and groupByKey is more expensive,
+ // this as follows would be better.
+ val bcIndeces = data.sparkContext.broadcast(stats.keySet)
+ val samples = data.mapPartitions { iter =>
+ val map = mutable.Map.empty[BigInt, mutable.ArrayBuffer[BV[Double]]]
+
+ bcIndeces.value.foreach {i => map(i) = mutable.ArrayBuffer.empty[BV[Double]]}
+ val LOCAL_SAMPLE_SIZE = 100
+ iter.foreach { case (i, point) =>
+ map(i).append(point)
+ // to avoid to increase the memory usage on each map thread,
+ // the number of elements is cut off at the right time.
+ if (map(i).size > LOCAL_SAMPLE_SIZE) {
+ val elements = map(i).sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ map(i) = mutable.ArrayBuffer(elements.head, elements.last)
+ }
+ }
+
+ // in order to reduce the shuffle size, take only two elements
+ map.filterNot(_._2.isEmpty).map { case (i, points) =>
+ val elements = map(i).toSeq.sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ i -> mutable.ArrayBuffer(elements.head, elements.last)
+ }.toIterator
+ }.reduceByKey { case (points1, points2) =>
+ points1.union(points2)
+ }.collect()
+
+ val nextCenters = samples.flatMap { case (i, points) =>
+ val elements = points.toSeq.sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ Array((2 * i, elements.head), (2 * i + 1, elements.last))
+ }.toMap
+ if (!stats.keySet.flatMap(idx => Array(2 * idx, 2 * idx + 1)).forall(nextCenters.contains(_))) {
+ throw new SparkException("Failed to initialize centers for next step")
+ }
+ nextCenters
+ }
+
+ /**
+ * Updates the indexes of clusters which is divided to its children indexes
+ *
+ * @param data pairs of point and its cluster index
+ * @param dividedClusters pairs of cluster index and cluster statistics
+ */
+ def updateClusterIndex(
+ data: RDD[(BigInt, BV[Double])],
+ dividedClusters: Map[BigInt, BisectingClusterStat]): RDD[(BigInt, BV[Double])] = {
+
+ // extract the centers of the clusters
+ val sc = data.sparkContext
+ var centers = dividedClusters.map { case (idx, cluster) => (idx, cluster.center)}
+ val bcCenters = sc.broadcast(centers)
+
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val bcMetric = sc.broadcast(metric)
+
+ // update the indexes to their children indexes
+ data.map { case (idx, point) =>
+ val childrenIndexes = Array(2 * idx, 2 * idx + 1).filter(c => bcCenters.value.contains(c))
+ childrenIndexes.length match {
+ // stay the index if the number of children is not enough
+ case s if s < 2 => (idx, point)
+ // update the indexes
+ case _ => {
+ val nextCenters = childrenIndexes.map(bcCenters.value(_))
+ val closestIndex = BisectingKMeans
+ .findClosestCenter(bcMetric.value)(nextCenters)(point)
+ val nextIndex = 2 * idx + closestIndex
+ (nextIndex, point)
+ }
+ }
+ }
+ }
+
+ /**
+ * Divides clusters according to their statistics
+ *
+ * @param data pairs of point and its cluster index
+ * @param targetClusters target clusters to divide
+ * @param maxIterations the maximum iterations to calculate clusters statistics
+ */
+ def divideClusters(
+ data: RDD[(BigInt, BV[Double])],
+ targetClusters: Map[BigInt, BisectingClusterStat],
+ maxIterations: Int): Map[BigInt, BisectingClusterStat] = {
+ val sc = data.sparkContext
+ val appName = sc.appName
+
+ // get keys of dividable clusters
+ val dividableClusters = targetClusters.filter { case (idx, cluster) => cluster.isDividable }
+ if (dividableClusters.isEmpty) {
+ return Map.empty[BigInt, BisectingClusterStat]
+ }
+ // extract dividable input data
+ val dividableData = data.filter { case (idx, point) => dividableClusters.contains(idx)}
+
+ var newCenters = BisectingKMeans.initNextCenters(dividableData, dividableClusters)
+ var bcNewCenters = sc.broadcast(newCenters)
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val bcMetric = sc.broadcast(metric)
+ var stats = Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+
+ var subIter = 0
+ var totalStd = Double.MaxValue
+ var oldTotalStd = Double.MaxValue
+ var relativeError = Double.MaxValue
+ while (subIter < maxIterations && relativeError > 10E-4) {
+ // calculate summary of each cluster
+ val eachStats = dividableData.mapPartitions { iter =>
+ val map = mutable.Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx, point) =>
+ // calculate next index number
+ val childrenCenters = Array(2 * idx, 2 * idx + 1)
+ .filter(x => bcNewCenters.value.contains(x)).map(bcNewCenters.value(_))
+ if (childrenCenters.length == 2) {
+ val closestIndex =
+ BisectingKMeans.findClosestCenter(bcMetric.value)(childrenCenters)(point)
+ val nextIndex = 2 * idx + closestIndex
+
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(
+ nextIndex,
+ (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size))
+ )
+ map(nextIndex) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ }
+ map.toIterator
+ }.reduceByKey { case ((sv1, n1, sumOfSquares1), (sv2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sv1 + sv2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.collect().toMap
+
+ // calculate the center of each cluster
+ newCenters = eachStats.map { case (idx, (sum, n, sumOfSquares)) => (idx, sum :/ n)}
+ bcNewCenters = sc.broadcast(newCenters)
+
+ // update summary of each cluster
+ stats = eachStats.toMap
+
+ totalStd = stats.map { case (idx, (sum, n, sumOfSquares)) =>
+ breezeSum((sumOfSquares :/ n) :- breezeNorm(sum :/ n, 2.0))
+ }.sum
+ relativeError = math.abs(oldTotalStd - totalStd) / totalStd
+ oldTotalStd = totalStd
+ subIter += 1
+ }
+ stats.map { case (i, stat) => i -> new BisectingClusterStat(stat._2.toLong, stat._1, stat._3) }
+ }
+
+ /**
+ * Creates the map of cluster stats to the map of cluster nodes with their criterions
+ *
+ * @param data input data
+ * @param stats map of cluster stats which is described as a binary tree
+ */
+ def createClusterNodes(
+ data: RDD[(BigInt, BV[Double])],
+ stats: Map[BigInt, BisectingClusterStat]): Map[BigInt, BisectingClusterNode] = {
+
+ // TODO: support other criterion, such as entropy
+ createClusterNodesWithAverageCost(data, stats)
+ }
+
+ /**
+ * Creates the map of cluster stats to the map of cluster nodes with their average costs
+ */
+ private def createClusterNodesWithAverageCost(
+ data: RDD[(BigInt, BV[Double])],
+ stats: Map[BigInt, BisectingClusterStat]): Map[BigInt, BisectingClusterNode] = {
+
+ // calculate average costs of all clusters
+ val bcCenters = data.sparkContext.broadcast(stats.map { case (i, stat) => i -> stat.center })
+ val costs = data.mapPartitions { iter =>
+ val counters = mutable.Map.empty[BigInt, (Long, Double)]
+ bcCenters.value.foreach {case (i, center) => counters(i) = (0L, 0.0)}
+ iter.foreach { case (i, point) =>
+ val cost = breezeNorm(bcCenters.value.apply(i) - point, 2.0)
+ counters(i) = (counters(i)._1 + 1, counters(i)._2 + cost)
+ }
+ counters.toIterator
+ }.reduceByKey { case((n1, cost1), (n2, cost2)) =>
+ (n1 + n2, cost1 + cost2)
+ }.collectAsMap()
+
+ stats.map { case (i, stat) =>
+ val avgCost = costs(i)._1 match {
+ case x if x == 0.0 => 0.0
+ case _ => costs(i)._2 / costs(i)._1
+ }
+ i -> new BisectingClusterNode(Vectors.fromBreeze(stat.center), stat.rows, avgCost)
+ }
+ }
+
+ /**
+ * Builds a cluster tree from a Map of clusters
+ *
+ * @param treeMap divided clusters as a Map class
+ * @param rootIndex index you want to start
+ * @param numClusters the number of clusters you want
+ * @return a built cluster tree
+ */
+ private def buildTree(
+ treeMap: Map[BigInt, BisectingClusterNode],
+ rootIndex: BigInt,
+ numClusters: Int): Option[BisectingClusterNode] = {
+
+ // if there is no index in the Map
+ if (!treeMap.contains(rootIndex)) return None
+
+ // build a cluster tree if the queue is empty or until the number of leaf clusters is enough
+ var numLeavesClusters = 1
+ val root = treeMap(rootIndex)
+ var leavesQueue = Map(rootIndex -> root)
+ while (leavesQueue.nonEmpty && numLeavesClusters < numClusters) {
+ // pick up the largest cluster by the maximum criterion of all the clusters
+ val mostScattered = leavesQueue.maxBy(_._2.criterion)
+ val mostScatteredKey = mostScattered._1
+ val mostScatteredCluster = mostScattered._2
+
+ // relate the most scattered cluster to its children clusters
+ val childrenIndexes = Array(2 * mostScatteredKey, 2 * mostScatteredKey + 1)
+ if (childrenIndexes.forall(i => treeMap.contains(i))) {
+ // insert children to the most scattered cluster
+ val children = childrenIndexes.map(i => treeMap(i))
+ mostScatteredCluster.insert(children)
+
+ // calculate the local dendrogram height
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val localHeight = children
+ .map(child => metric(child.center.toBreeze, mostScatteredCluster.center.toBreeze)).max
+ mostScatteredCluster.setLocalHeight(localHeight)
+
+ // update the queue
+ leavesQueue = leavesQueue ++ childrenIndexes.map(i => i -> treeMap(i)).toMap
+ numLeavesClusters += 1
+ }
+
+ // remove the cluster which is involved to the cluster tree
+ leavesQueue = leavesQueue.filterNot(_ == mostScattered)
+ }
+ Some(root)
+ }
+}
+
+/**
+ * A cluster as a tree node which can have its sub nodes
+ *
+ * @param center the center of the cluster
+ * @param rows the number of rows in the cluster
+ * @param criterion how large a cluster is
--- End diff --
`cost`? (`impurity`?)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151325995
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by freeman-lab <gi...@git.apache.org>.
Github user freeman-lab commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-96512684
@yu-iskw I'm still going through the patch, but so far it's looking good! I've also been testing it locally.
Is there a reason you removed the `toMergeList` method from the previous version of this submission? That seemed quite useful to me, as it's a common way to describe the output of hierarchical clustering, both in formal treatments as well as in other analysis libraries (though I do suggest naming it `toLinkageMatrix`). What do you think about bringing it back?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42821004
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = BisectingKMeans.initData(input).cache()
+ while (clusterStats.size < maxAllNodesInTree && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ val leafClusters = BisectingKMeans.summarizeClusters(data)
+ val dividableLeafClusters = leafClusters.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusters
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ else {
--- End diff --
merge to the line above
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r27716372
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/HierarchicalClustering.scala ---
@@ -0,0 +1,610 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseVector => BDV, SparseVector => BSV, Vector => BV, norm => breezeNorm}
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.random.XORShiftRandom
+import org.apache.spark.{Logging, SparkException}
+
+import scala.collection.{Map, mutable}
+
+
+/**
+ * Top-level methods for calling the hierarchical clustering algorithm
+ */
+object HierarchicalClustering extends Logging {
--- End diff --
Sorry, one more question. How about consistency between Scala and Python?
MLlib in PySpark doesn't support builder methods. How to call a train() method in PySpark should similar to that in Scala. I think that is a static train() method.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-90865882
Sorry for modifying the code before your feedback. The main points of the difference are like below. Thank you for your continuous support.
- Remove the static train() method from HierarchicalClustering object
- Remove the parentheses from the setter/getter methods
- Add a method to calculate Within Set Sum of Squared Error (WSSSE) into HierarchicalClusteringModel
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151334247
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by freeman-lab <gi...@git.apache.org>.
Github user freeman-lab commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r29121409
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/HierarchicalClusteringModel.scala ---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseVector => BDV, Vector => BV, norm => breezeNorm}
+import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.mllib.util.{Loader, Saveable}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{Logging, SparkContext}
+
+/**
+ * This class is used for the model of the hierarchical clustering
+ *
+ * @param tree a cluster as a tree node
+ */
+class HierarchicalClusteringModel(val tree: ClusterTree)
+ extends Serializable with Saveable with Logging {
+
+ /** Current version of model save/load format. */
+ override protected def formatVersion: String = "1.0"
+
+ override def save(sc: SparkContext, path: String) {
+ val oos = new java.io.ObjectOutputStream(new java.io.FileOutputStream(path))
+ try {
+ oos.writeObject(this)
+ } finally {
+ oos.close()
+ }
+ }
+
+ def getClusters(): Array[ClusterTree] = this.tree.getLeavesNodes()
+
+ def getCenters(): Array[Vector] = this.getClusters().map(_.center)
--- End diff --
Remove parentheses after `getCenters`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151327314
**[Test build #44387 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44387/consoleFull)** for PR 5267 at commit [`165e191`](https://github.com/apache/spark/commit/165e191755bb641f10c604f1ec214248f5a104e7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42821656
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = BisectingKMeans.initData(input).cache()
+ while (clusterStats.size < maxAllNodesInTree && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ val leafClusters = BisectingKMeans.summarizeClusters(data)
+ val dividableLeafClusters = leafClusters.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusters
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ else {
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided =
+ BisectingKMeans.divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = BisectingKMeans.updateClusterIndex(data, divided).cache()
+ rddArray = rddArray ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (rddArray.length > 1) {
+ val head = rddArray.head
+ head.unpersist()
+ rddArray = rddArray.filterNot(_.hashCode() == head.hashCode())
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+ }
+ }
+ // unpersist kept RDDs
+ rddArray.foreach(_.unpersist())
+ // create a map of cluster node with their criterions
+ val nodes = BisectingKMeans.createClusterNodes(data, clusterStats)
+
+ // build a cluster tree by Map class which is expressed
+ logInfo(s"Building the cluster tree is started in ${sc.appName}")
+ val root = BisectingKMeans.buildTree(nodes, BisectingKMeans.ROOT_INDEX_KEY, this.k)
+ if (root.isEmpty) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusterStats")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ logInfo(s"Elapsed Time for ${this.getClass.getSimpleName} Training: ${finishTime} [sec]")
+
+ // make a bisecting kmeans model
+ val model = new BisectingKMeansModel(root.get)
+ val leavesNodes = model.getClusters
+ if (leavesNodes.length < this.k) {
+ logWarning(s"# clusters is less than you want: ${leavesNodes.length} / ${k}")
+ }
+ model
+ }
+}
+
+
+private[clustering] object BisectingKMeans {
+
+ val ROOT_INDEX_KEY: BigInt = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ def findClosestCenter(metric: (BV[Double], BV[Double]) => Double)
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)
+ closestIndex
+ }
+
+ /**
+ * Summarizes data by each cluster as Map
+ *
+ * @param data pairs of point and its cluster index
+ */
+ def summarizeClusters(data: RDD[(BigInt, BV[Double])]): Map[BigInt, BisectingClusterStat] = {
+
+ val stats = data.mapPartitions { iter =>
+ // calculate the accumulation of the all point in a partition and count the rows
+ val map = mutable.Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx: BigInt, point: BV[Double]) =>
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(idx, (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size)))
+ map(idx) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ map.toIterator
+ }.reduceByKey { case ((sum1, n1, sumOfSquares1), (sum2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sum1 + sum2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.collect().toMap
+
+ stats.map {case (i, stat) => i -> new BisectingClusterStat(stat._2.toLong, stat._1, stat._3)}
+ }
+
+ /**
+ * Assigns the initial cluster index id to all data
+ */
+ def initData(data: RDD[Vector]): RDD[(BigInt, BV[Double])] = {
+ data.map { v: Vector => (BisectingKMeans.ROOT_INDEX_KEY, v.toBreeze)}
+ }
+
+ /**
+ * Gets the initial centers for bisect k-means
+ *
+ * @param data pairs of point and its cluster index
+ * @param stats pairs of cluster index and cluster statistics
+ */
+ def initNextCenters(
+ data: RDD[(BigInt, BV[Double])],
+ stats: Map[BigInt, BisectingClusterStat]): Map[BigInt, BV[Double]] = {
+
+ // Since the combination sampleByKey and groupByKey is more expensive,
+ // this as follows would be better.
+ val bcIndeces = data.sparkContext.broadcast(stats.keySet)
+ val samples = data.mapPartitions { iter =>
+ val map = mutable.Map.empty[BigInt, mutable.ArrayBuffer[BV[Double]]]
+
+ bcIndeces.value.foreach {i => map(i) = mutable.ArrayBuffer.empty[BV[Double]]}
+ val LOCAL_SAMPLE_SIZE = 100
+ iter.foreach { case (i, point) =>
+ map(i).append(point)
+ // to avoid to increase the memory usage on each map thread,
+ // the number of elements is cut off at the right time.
+ if (map(i).size > LOCAL_SAMPLE_SIZE) {
+ val elements = map(i).sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ map(i) = mutable.ArrayBuffer(elements.head, elements.last)
+ }
+ }
+
+ // in order to reduce the shuffle size, take only two elements
+ map.filterNot(_._2.isEmpty).map { case (i, points) =>
+ val elements = map(i).toSeq.sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ i -> mutable.ArrayBuffer(elements.head, elements.last)
+ }.toIterator
+ }.reduceByKey { case (points1, points2) =>
+ points1.union(points2)
+ }.collect()
+
+ val nextCenters = samples.flatMap { case (i, points) =>
+ val elements = points.toSeq.sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ Array((2 * i, elements.head), (2 * i + 1, elements.last))
+ }.toMap
+ if (!stats.keySet.flatMap(idx => Array(2 * idx, 2 * idx + 1)).forall(nextCenters.contains(_))) {
+ throw new SparkException("Failed to initialize centers for next step")
+ }
+ nextCenters
+ }
+
+ /**
+ * Updates the indexes of clusters which is divided to its children indexes
+ *
+ * @param data pairs of point and its cluster index
+ * @param dividedClusters pairs of cluster index and cluster statistics
+ */
+ def updateClusterIndex(
+ data: RDD[(BigInt, BV[Double])],
+ dividedClusters: Map[BigInt, BisectingClusterStat]): RDD[(BigInt, BV[Double])] = {
+
+ // extract the centers of the clusters
+ val sc = data.sparkContext
+ var centers = dividedClusters.map { case (idx, cluster) => (idx, cluster.center)}
+ val bcCenters = sc.broadcast(centers)
+
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val bcMetric = sc.broadcast(metric)
+
+ // update the indexes to their children indexes
+ data.map { case (idx, point) =>
+ val childrenIndexes = Array(2 * idx, 2 * idx + 1).filter(c => bcCenters.value.contains(c))
+ childrenIndexes.length match {
+ // stay the index if the number of children is not enough
+ case s if s < 2 => (idx, point)
+ // update the indexes
+ case _ => {
+ val nextCenters = childrenIndexes.map(bcCenters.value(_))
+ val closestIndex = BisectingKMeans
+ .findClosestCenter(bcMetric.value)(nextCenters)(point)
+ val nextIndex = 2 * idx + closestIndex
+ (nextIndex, point)
+ }
+ }
+ }
+ }
+
+ /**
+ * Divides clusters according to their statistics
+ *
+ * @param data pairs of point and its cluster index
+ * @param targetClusters target clusters to divide
+ * @param maxIterations the maximum iterations to calculate clusters statistics
+ */
+ def divideClusters(
+ data: RDD[(BigInt, BV[Double])],
+ targetClusters: Map[BigInt, BisectingClusterStat],
+ maxIterations: Int): Map[BigInt, BisectingClusterStat] = {
+ val sc = data.sparkContext
+ val appName = sc.appName
+
+ // get keys of dividable clusters
+ val dividableClusters = targetClusters.filter { case (idx, cluster) => cluster.isDividable }
+ if (dividableClusters.isEmpty) {
+ return Map.empty[BigInt, BisectingClusterStat]
+ }
+ // extract dividable input data
+ val dividableData = data.filter { case (idx, point) => dividableClusters.contains(idx)}
+
+ var newCenters = BisectingKMeans.initNextCenters(dividableData, dividableClusters)
+ var bcNewCenters = sc.broadcast(newCenters)
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val bcMetric = sc.broadcast(metric)
+ var stats = Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+
+ var subIter = 0
+ var totalStd = Double.MaxValue
+ var oldTotalStd = Double.MaxValue
+ var relativeError = Double.MaxValue
+ while (subIter < maxIterations && relativeError > 10E-4) {
+ // calculate summary of each cluster
+ val eachStats = dividableData.mapPartitions { iter =>
+ val map = mutable.Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx, point) =>
+ // calculate next index number
+ val childrenCenters = Array(2 * idx, 2 * idx + 1)
+ .filter(x => bcNewCenters.value.contains(x)).map(bcNewCenters.value(_))
+ if (childrenCenters.length == 2) {
+ val closestIndex =
+ BisectingKMeans.findClosestCenter(bcMetric.value)(childrenCenters)(point)
+ val nextIndex = 2 * idx + closestIndex
+
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(
+ nextIndex,
+ (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size))
+ )
+ map(nextIndex) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ }
+ map.toIterator
+ }.reduceByKey { case ((sv1, n1, sumOfSquares1), (sv2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sv1 + sv2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.collect().toMap
+
+ // calculate the center of each cluster
+ newCenters = eachStats.map { case (idx, (sum, n, sumOfSquares)) => (idx, sum :/ n)}
+ bcNewCenters = sc.broadcast(newCenters)
+
+ // update summary of each cluster
+ stats = eachStats.toMap
+
+ totalStd = stats.map { case (idx, (sum, n, sumOfSquares)) =>
+ breezeSum((sumOfSquares :/ n) :- breezeNorm(sum :/ n, 2.0))
+ }.sum
+ relativeError = math.abs(oldTotalStd - totalStd) / totalStd
+ oldTotalStd = totalStd
+ subIter += 1
+ }
+ stats.map { case (i, stat) => i -> new BisectingClusterStat(stat._2.toLong, stat._1, stat._3) }
+ }
+
+ /**
+ * Creates the map of cluster stats to the map of cluster nodes with their criterions
+ *
+ * @param data input data
+ * @param stats map of cluster stats which is described as a binary tree
+ */
+ def createClusterNodes(
+ data: RDD[(BigInt, BV[Double])],
+ stats: Map[BigInt, BisectingClusterStat]): Map[BigInt, BisectingClusterNode] = {
+
+ // TODO: support other criterion, such as entropy
+ createClusterNodesWithAverageCost(data, stats)
+ }
+
+ /**
+ * Creates the map of cluster stats to the map of cluster nodes with their average costs
+ */
+ private def createClusterNodesWithAverageCost(
+ data: RDD[(BigInt, BV[Double])],
+ stats: Map[BigInt, BisectingClusterStat]): Map[BigInt, BisectingClusterNode] = {
+
+ // calculate average costs of all clusters
+ val bcCenters = data.sparkContext.broadcast(stats.map { case (i, stat) => i -> stat.center })
+ val costs = data.mapPartitions { iter =>
+ val counters = mutable.Map.empty[BigInt, (Long, Double)]
+ bcCenters.value.foreach {case (i, center) => counters(i) = (0L, 0.0)}
+ iter.foreach { case (i, point) =>
+ val cost = breezeNorm(bcCenters.value.apply(i) - point, 2.0)
+ counters(i) = (counters(i)._1 + 1, counters(i)._2 + cost)
+ }
+ counters.toIterator
+ }.reduceByKey { case((n1, cost1), (n2, cost2)) =>
+ (n1 + n2, cost1 + cost2)
+ }.collectAsMap()
+
+ stats.map { case (i, stat) =>
+ val avgCost = costs(i)._1 match {
+ case x if x == 0.0 => 0.0
+ case _ => costs(i)._2 / costs(i)._1
+ }
+ i -> new BisectingClusterNode(Vectors.fromBreeze(stat.center), stat.rows, avgCost)
+ }
+ }
+
+ /**
+ * Builds a cluster tree from a Map of clusters
+ *
+ * @param treeMap divided clusters as a Map class
+ * @param rootIndex index you want to start
+ * @param numClusters the number of clusters you want
+ * @return a built cluster tree
+ */
+ private def buildTree(
+ treeMap: Map[BigInt, BisectingClusterNode],
+ rootIndex: BigInt,
+ numClusters: Int): Option[BisectingClusterNode] = {
+
+ // if there is no index in the Map
+ if (!treeMap.contains(rootIndex)) return None
+
+ // build a cluster tree if the queue is empty or until the number of leaf clusters is enough
+ var numLeavesClusters = 1
+ val root = treeMap(rootIndex)
+ var leavesQueue = Map(rootIndex -> root)
+ while (leavesQueue.nonEmpty && numLeavesClusters < numClusters) {
+ // pick up the largest cluster by the maximum criterion of all the clusters
+ val mostScattered = leavesQueue.maxBy(_._2.criterion)
+ val mostScatteredKey = mostScattered._1
+ val mostScatteredCluster = mostScattered._2
+
+ // relate the most scattered cluster to its children clusters
+ val childrenIndexes = Array(2 * mostScatteredKey, 2 * mostScatteredKey + 1)
+ if (childrenIndexes.forall(i => treeMap.contains(i))) {
+ // insert children to the most scattered cluster
+ val children = childrenIndexes.map(i => treeMap(i))
+ mostScatteredCluster.insert(children)
+
+ // calculate the local dendrogram height
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val localHeight = children
+ .map(child => metric(child.center.toBreeze, mostScatteredCluster.center.toBreeze)).max
+ mostScatteredCluster.setLocalHeight(localHeight)
+
+ // update the queue
+ leavesQueue = leavesQueue ++ childrenIndexes.map(i => i -> treeMap(i)).toMap
+ numLeavesClusters += 1
+ }
+
+ // remove the cluster which is involved to the cluster tree
+ leavesQueue = leavesQueue.filterNot(_ == mostScattered)
+ }
+ Some(root)
+ }
+}
+
+/**
+ * A cluster as a tree node which can have its sub nodes
+ *
+ * @param center the center of the cluster
+ * @param rows the number of rows in the cluster
+ * @param criterion how large a cluster is
+ * @param localHeight the maximal distance between this node and its children
+ * @param parent the parent cluster of the cluster
+ * @param children the children nodes of the cluster
+ */
+@Since("1.6.0")
+class BisectingClusterNode private (
+ @Since("1.6.0") val center: Vector,
+ @Since("1.6.0") val rows: Long,
+ @Since("1.6.0") val criterion: Double,
+ private var localHeight: Double,
+ private var parent: Option[BisectingClusterNode],
+ private var children: Seq[BisectingClusterNode]) extends Serializable {
+
+ require(!criterion.isNaN)
+
+ @Since("1.6.0")
+ def this(center: Vector, rows: Long, criterion: Double) =
+ this(center, rows, criterion, 0.0, None, Array.empty[BisectingClusterNode])
+
+ /**
+ * Inserts a sub node as its child
+ *
+ * @param child inserted sub node
+ */
+ @Since("1.6.0")
+ def insert(child: BisectingClusterNode) {
+ insert(Array(child))
+ }
+
+ /**
+ * Inserts sub nodes as its children
+ *
+ * @param children inserted sub nodes
+ */
+ @Since("1.6.0")
+ def insert(children: Array[BisectingClusterNode]) {
+ this.children = this.children ++ children
+ children.foreach(child => child.parent = Some(this))
+ }
+
+ /**
+ * Converts the tree into Array class
+ * the sub nodes are recursively expanded
+ *
+ * @return an Array class which the cluster tree is expanded
+ */
+ @Since("1.6.0")
+ def toArray: Array[BisectingClusterNode] = {
+ val array = this.children.size match {
+ case 0 => Array(this)
+ case _ => Array(this) ++ this.children.flatMap(child => child.toArray.toIterator)
+ }
+ array.sortWith { case (a, b) =>
+ a.getDepth < b.getDepth && a.criterion < b.criterion && a.rows < b.rows
+ }
+ }
+
+ /**
+ * Gets the depth of the cluster in the tree
+ *
+ * @return the depth from the root
+ */
+ @Since("1.6.0")
+ def getDepth: Int = {
+ this.parent match {
+ case None => 0
+ case _ => 1 + this.parent.get.getDepth
+ }
+ }
+
+ /**
+ * Gets the leaves nodes in the cluster tree
+ */
+ @Since("1.6.0")
+ def getLeavesNodes: Array[BisectingClusterNode] = {
+ this.toArray.filter(_.isLeaf).sortBy(_.center.toArray.sum)
+ }
+
+ @Since("1.6.0")
+ def isLeaf: Boolean = this.children.isEmpty
+
+ @Since("1.6.0")
+ def getParent: Option[BisectingClusterNode] = this.parent
+
+ @Since("1.6.0")
+ def getChildren: Seq[BisectingClusterNode] = this.children
+
+ /**
+ * Gets the dendrogram height of the cluster at the cluster tree.
+ * A dendrogram height is different from a local height.
+ * A dendrogram height means a total height of a node in a tree.
+ * A local height means a maximum distance between a node and its children.
+ *
+ * @return the dendrogram height
+ */
+ @Since("1.6.0")
+ def getHeight: Double = {
+ this.children.size match {
+ case 0 => 0.0
+ case _ => this.localHeight + this.children.map(_.getHeight).max
+ }
+ }
+
+ @Since("1.6.0")
+ def setLocalHeight(height: Double): Unit = this.localHeight = height
+
+ /**
+ * Converts to an adjacency list
+ *
+ * @return List[(fromNodeId, toNodeId, distance)]
+ */
+ @Since("1.6.0")
+ def toAdjacencyList: Array[(Int, Int, Double)] = {
+ val nodes = toArray
+
+ var adjacencyList = Array.empty[(Int, Int, Double)]
+ nodes.foreach { parent =>
+ if (parent.children.size > 1) {
+ val parentIndex = nodes.indexOf(parent)
+ parent.children.foreach { child =>
+ val childIndex = nodes.indexOf(child)
+ adjacencyList = adjacencyList :+(parentIndex, childIndex, parent.localHeight)
+ }
+ }
+ }
+ adjacencyList
+ }
+
+ /**
+ * Converts to a linkage matrix
+ * Returned data format is fit for scipy's dendrogram function
+ *
+ * @return List[(node1, node2, distance, tree size)]
+ */
+ @Since("1.6.0")
+ def toLinkageMatrix: Array[(Int, Int, Double, Int)] = {
+ val nodes = toArray.sortWith { case (a, b) => a.getHeight < b.getHeight}
+ val leaves = nodes.filter(_.isLeaf)
+ val notLeaves = nodes.filterNot(_.isLeaf).filter(_.getChildren.size > 1)
+ val clusters = leaves ++ notLeaves
+ val treeMap = clusters.zipWithIndex.map { case (node, idx) => node -> idx}.toMap
+
+ // If a node only has one-child, the child is regarded as the cluster of the child.
+ // Cluster A has cluster B and Cluster B. B is a leaf. C only has cluster D.
+ // ==> A merge list is (B, D), not (B, C).
+ def getIndex(map: Map[BisectingClusterNode, Int], node: BisectingClusterNode): Int = {
+ node.children.size match {
+ case 1 => getIndex(map, node.children.head)
+ case _ => map(node)
+ }
+ }
+ clusters.filterNot(_.isLeaf).map { node =>
+ (getIndex(treeMap, node.children.head),
+ getIndex(treeMap, node.children(1)),
+ node.getHeight,
+ node.toArray.filter(_.isLeaf).length)
+ }
+ }
+}
+
+
+/**
+ * This class is used for maneging a cluster statistics
+ *
+ * @param rows the number of points
+ * @param sums the sum of points
+ * @param sumOfSquares the sum of squares of points
+ */
+private[clustering] case class BisectingClusterStat (
+ rows: Long,
+ sums: BV[Double],
+ sumOfSquares: BV[Double]) extends Serializable {
+
+ // initialization
+ val center: BV[Double] = sums :/ rows.toDouble
+ val variances: BV[Double] = rows match {
--- End diff --
* use a scalar variance instead of coordinate variance
* change `sums` and `sumOfSquares` to `mean: BV[Double]` and `variance: Double`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r27705976
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/HierarchicalClustering.scala ---
@@ -0,0 +1,610 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseVector => BDV, SparseVector => BSV, Vector => BV, norm => breezeNorm}
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.random.XORShiftRandom
+import org.apache.spark.{Logging, SparkException}
+
+import scala.collection.{Map, mutable}
+
+
+/**
+ * Top-level methods for calling the hierarchical clustering algorithm
+ */
+object HierarchicalClustering extends Logging {
--- End diff --
I understand. I agree with that we should start deprecating the old ones.
I am removing the train() static method from HierarchicalClustering object.
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155156753
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152355068
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by freeman-lab <gi...@git.apache.org>.
Github user freeman-lab commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r29121114
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/HierarchicalClustering.scala ---
@@ -0,0 +1,574 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseVector => BDV, SparseVector => BSV, Vector => BV, norm => breezeNorm}
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.random.XORShiftRandom
+import org.apache.spark.{Logging, SparkException}
+
+import scala.collection.{Map, mutable}
+
+
+object HierarchicalClustering extends Logging {
+
+ private[clustering] val ROOT_INDEX_KEY: Long = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ private[mllib]
+ def findClosestCenter(metric: Function2[BV[Double], BV[Double], Double])
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)
+ closestIndex
+ }
+}
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bi-sect k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * @param numClusters tne number of clusters you want
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations
+ * @param maxRetries the number of maximum retries
+ * @param seed a random seed
+ */
+class HierarchicalClustering private (
+ private var numClusters: Int,
+ private var clusterMap: Map[Long, ClusterTree],
+ private var maxIterations: Int,
+ private var maxRetries: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ def this() = this(20, mutable.ListMap.empty[Long, ClusterTree], 20, 10, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ def setNumClusters(numClusters: Int): this.type = {
+ this.numClusters = numClusters
+ this
+ }
+
+ def getNumClusters: Int = this.numClusters
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ def getSubIterations: Int = this.maxIterations
+
+ /**
+ * Sets the number of maximum retries of each clustering step
+ */
+ def setMaxRetries(maxRetries: Int): this.type = {
+ this.maxRetries = maxRetries
+ this
+ }
+
+ def getMaxRetries: Int = this.maxRetries
+
+ /**
+ * Sets the random seed
+ */
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the hierarchical clustering algorithm
+ * @param input RDD of vectors
+ * @return model for the hierarchical clustering
+ */
+ def run(input: RDD[Vector]): HierarchicalClusteringModel = {
+ val sc = input.sparkContext
+ log.info(s"${sc.appName} starts a hierarchical clustering algorithm")
+
+ var data = initData(input).cache()
+ val startTime = System.currentTimeMillis()
+
+ // `clusters` is described as binary tree structure
+ // `clusters(1)` means the root of a binary tree
+ var clusters = summarizeAsClusters(data)
+ var leafClusters = clusters
+ var step = 1
+ var numDividedClusters = 0
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(Long, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.numClusters) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ while (clusters.size < maxAllNodesInTree && noMoreDividable == false) {
+ log.info(s"${sc.appName} starts step ${step}")
+
+ // enough to be clustered if the number of divided clusters is equal to 0
+ val divided = getDividedClusters(data, leafClusters)
+ if (divided.size == 0) {
+ noMoreDividable = true
+ }
+ else {
+ // update each index
+ val newData = updateClusterIndex(data, divided).cache()
+ rddArray = rddArray ++ Array(data)
+ data = newData
+
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (rddArray.size > 1) {
+ val head = rddArray.head
+ head.unpersist()
+ rddArray = rddArray.filterNot(_.hashCode() == head.hashCode())
+ }
+
+ // merge the divided clusters with the map as the cluster tree
+ clusters = clusters ++ divided
+ numDividedClusters = data.map(_._1).distinct().count().toInt
+ leafClusters = divided
+ step += 1
+
+ log.info(s"${sc.appName} adding ${divided.size} new clusters at step:${step}")
+ }
+ }
+ // unpersist kept RDDs
+ rddArray.foreach(_.unpersist())
+
+ // build a cluster tree by Map class which is expressed
+ log.info(s"Building the cluster tree is started in ${sc.appName}")
+ val root = buildTree(clusters, HierarchicalClustering.ROOT_INDEX_KEY, this.numClusters)
+ if (root == None) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusters")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ log.info(s"Elapsed Time for Hierarchical Clustering Training: ${finishTime} [sec]")
+
+ // make a hierarchical clustering model
+ val model = new HierarchicalClusteringModel(root.get)
+ val leavesNodes = model.getClusters()
+ if (leavesNodes.size < this.numClusters) {
+ log.warn(s"# clusters is less than you have expected: ${leavesNodes.size} / ${numClusters}. ")
+ }
+ model
+ }
+
+ /**
+ * Assigns the initial cluster index id to all data
+ */
+ private[clustering]
+ def initData(data: RDD[Vector]): RDD[(Long, BV[Double])] = {
+ data.map { v: Vector => (HierarchicalClustering.ROOT_INDEX_KEY, v.toBreeze)}.cache
+ }
+
+ /**
+ * Summarizes data by each cluster as ClusterTree2 classes
+ */
+ private[clustering]
+ def summarizeAsClusters(data: RDD[(Long, BV[Double])]): Map[Long, ClusterTree] = {
+ // summarize input data
+ val stats = summarize(data)
+
+ // convert statistics to ClusterTree class
+ stats.map { case (i, (sum, n, sumOfSquares)) =>
+ val center = Vectors.fromBreeze(sum :/ n)
+ val variances = n match {
+ case n if n > 1 => Vectors.fromBreeze(sumOfSquares.:*(n) - (sum :* sum) :/ (n * (n - 1.0)))
+ case _ => Vectors.zeros(sum.size)
+ }
+ (i, new ClusterTree(center, n.toLong, variances))
+ }.toMap
+ }
+
+ /**
+ * Summarizes data by each cluster as Map
+ */
+ private[clustering]
+ def summarize(data: RDD[(Long, BV[Double])]): Map[Long, (BV[Double], Double, BV[Double])] = {
+ data.mapPartitions { iter =>
+ // calculate the accumulation of the all point in a partition and count the rows
+ val map = mutable.Map.empty[Long, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx: Long, point: BV[Double]) =>
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map.get(idx)
+ .getOrElse(BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size))
+ map(idx) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ map.toIterator
+ }.reduceByKey { case ((sum1, n1, sumOfSquares1), (sum2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sum1 + sum2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.collect().toMap
+ }
+
+ /**
+ * Gets the initial centers for bi-sect k-means
+ */
+ private[clustering]
+ def initChildrenCenter(clusters: Map[Long, BV[Double]]): Map[Long, BV[Double]] = {
+ val rand = new XORShiftRandom()
+ rand.setSeed(this.seed)
+
+ clusters.flatMap { case (idx, center) =>
+ val childrenIndexes = Array(2 * idx, 2 * idx + 1)
+ val relativeErrorCoefficient = 0.001
+ Array(
+ (2 * idx, center.map(elm => elm - (elm * relativeErrorCoefficient * rand.nextDouble()))),
+ (2 * idx + 1, center.map(elm => elm + (elm * relativeErrorCoefficient * rand.nextDouble())))
+ )
+ }.toMap
+ }
+
+ /**
+ * Gets the new divided centers
+ */
+ private[clustering]
+ def getDividedClusters(data: RDD[(Long, BV[Double])],
+ dividedClusters: Map[Long, ClusterTree]): Map[Long, ClusterTree] = {
+ val sc = data.sparkContext
+ val appName = sc.appName
+
+ // get keys of dividable clusters
+ val dividableKeys = dividedClusters.filter { case (idx, cluster) =>
+ cluster.variances.toArray.sum > 0.0 && cluster.records >= 2
+ }.keySet
+ if (dividableKeys.size == 0) {
+ log.info(s"There is no dividable clusters in ${appName}.")
+ return Map.empty[Long, ClusterTree]
+ }
+
+ // divide input data
+ var dividableData = data.filter { case (idx, point) => dividableKeys.contains(idx)}
+ var dividableClusters = dividedClusters.filter { case (k, v) => dividableKeys.contains(k)}
+ val idealIndexes = dividableKeys.flatMap(idx => Array(2 * idx, 2 * idx + 1).toIterator)
+ var stats = divide(data, dividableClusters)
+
+ // if there is clusters which is failed to be divided,
+ // retry to divide only failed clusters again and again
+ var tryTimes = 1
+ while (stats.size < dividableKeys.size * 2 && tryTimes <= this.maxRetries) {
+ // get the indexes of clusters which is failed to be divided
+ val failedIndexes = idealIndexes.filterNot(stats.keySet.contains).map(idx => (idx / 2).toLong)
+ val failedCenters = dividedClusters.filter { case (idx, clstr) => failedIndexes.contains(idx)}
+ log.info(s"# failed clusters is ${failedCenters.size} of ${dividableKeys.size}" +
+ s"at ${tryTimes} times in ${appName}")
+
+ // divide the failed clusters again
+ sc.broadcast(failedIndexes)
+ dividableData = data.filter { case (idx, point) => failedIndexes.contains(idx)}
+ val missingStats = divide(dividableData, failedCenters)
+ stats = stats ++ missingStats
+ tryTimes += 1
+ }
+
+ // make children clusters
+ stats.filter { case (i, (sum, n, sumOfSquares)) => n > 0}
+ .map { case (i, (sum, n, sumOfSquares)) =>
+ val center = Vectors.fromBreeze(sum :/ n)
+ val variances = n match {
+ case 1 => Vectors.sparse(sum.size, Array(), Array())
+ case _ => Vectors.fromBreeze(sumOfSquares.:*(n) - (sum :* sum) :/ (n * (n - 1.0)))
+ }
+ val child = new ClusterTree(center, n.toLong, variances)
+ (i, child)
+ }.toMap
+ }
+
+ /**
+ * Builds a cluster tree from a Map of clusters
+ *
+ * @param treeMap divided clusters as a Map class
+ * @param rootIndex index you want to start
+ * @param numClusters the number of clusters you want
+ * @return a built cluster tree
+ */
+ private[clustering]
+ def buildTree(treeMap: Map[Long, ClusterTree],
+ rootIndex: Long,
+ numClusters: Int): Option[ClusterTree] = {
+
+ // if there is no index in the Map
+ if (!treeMap.contains(rootIndex)) return None
+
+ // build a cluster tree if the queue is empty or until the number of leaves clusters is enough
+ var numLeavesClusters = 1
+ val root = treeMap(rootIndex)
+ var leavesQueue = Map(rootIndex -> root)
+ while (leavesQueue.size > 0 && numLeavesClusters < numClusters) {
+ // pick up the cluster whose variance is the maximum in the queue
+ val mostScattered = leavesQueue.maxBy(_._2.variancesNorm)
+ val mostScatteredKey = mostScattered._1
+ val mostScatteredCluster = mostScattered._2
+
+ // relate the most scattered cluster to its children clusters
+ val childrenIndexes = Array(2 * mostScatteredKey, 2 * mostScatteredKey + 1)
+ if (childrenIndexes.forall(i => treeMap.contains(i))) {
+ // insert children to the most scattered cluster
+ val children = childrenIndexes.map(i => treeMap(i))
+ mostScatteredCluster.insert(children)
+
+ // calculate the local dendrogram height
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val localHeight = children
+ .map(child => metric(child.center.toBreeze, mostScatteredCluster.center.toBreeze)).max
+ mostScatteredCluster.setLocalHeight(localHeight)
+
+ // update the queue
+ leavesQueue = leavesQueue ++ childrenIndexes.map(i => (i -> treeMap(i))).toMap
+ numLeavesClusters += 1
+ }
+
+ // remove the cluster which is involved to the cluster tree
+ leavesQueue = leavesQueue.filterNot(_ == mostScattered)
+
+ log.info(s"Total Leaves Clusters: ${numLeavesClusters} / ${numClusters}. " +
+ s"Cluster ${childrenIndexes.mkString(",")} are merged.")
+ }
+ Some(root)
+ }
+
+ /**
+ * Divides the input data
+ *
+ * @param data the pairs of cluster index and point which you want to divide
+ * @param clusters the clusters you want to divide AS a Map class
+ * @return divided clusters as Map
+ */
+ private[clustering]
+ def divide(data: RDD[(Long, BV[Double])],
+ clusters: Map[Long, ClusterTree]): Map[Long, (BV[Double], Double, BV[Double])] = {
+
+ val sc = data.sparkContext
+ val centers = clusters.map { case (idx, cluster) => (idx, cluster.center.toBreeze)}
+ var newCenters = initChildrenCenter(centers)
+ if (newCenters.size == 0) {
+ return Map.empty[Long, (BV[Double], Double, BV[Double])]
+ }
+ sc.broadcast(newCenters)
--- End diff --
This use of `sc.broadcast` has no effect because the output isn't assigned or used. Instead, you want something like val `bcNewCenters = sc.broadcast(newCenters)` and then access within the map as bcCenters.value. This appears in a few other places as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-150285291
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152355087
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r43071633
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,744 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
--- End diff --
`Map` is not needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r44322215
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,489 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import java.util.Random
+
+import scala.collection.mutable
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.{Experimental, Since}
+import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.mllib.linalg.{BLAS, Vector, Vectors}
+import org.apache.spark.mllib.util.MLUtils
+import org.apache.spark.rdd.RDD
+import org.apache.spark.storage.StorageLevel
+
+/**
+ * A bisecting k-means algorithm based on the paper "A comparison of document clustering techniques"
+ * by Steinbach, Karypis, and Kumar, with modification to fit Spark.
+ * The algorithm starts from a single cluster that contains all points.
+ * Iteratively it finds divisible clusters on the bottom level and bisects each of them using
+ * k-means, until there are `k` leaf clusters in total or no leaf clusters are divisible.
+ * The bisecting steps of clusters on the same level are grouped together to increase parallelism.
+ * If bisecting all divisible clusters on the bottom level would result more than `k` leaf clusters,
+ * larger clusters get higher priority.
+ *
+ * @param k the desired number of leaf clusters (default: 4). The actual number could be smaller if
+ * there are no divisible leaf clusters.
+ * @param maxIterations the max number of k-means iterations to split clusters (default: 20)
+ * @param minDivisibleClusterSize the minimum number of points (if >= 1.0) or the minimum proportion
+ * of points (if < 1.0) of a divisible cluster (default: 1)
+ * @param seed a random seed (default: hash value of the class name)
+ *
+ * @see [[http://glaros.dtc.umn.edu/gkhome/fetch/papers/docclusterKDDTMW00.pdf
+ * Steinbach, Karypis, and Kumar, A comparison of document clustering techniques,
+ * KDD Workshop on Text Mining, 2000.]]
+ */
+@Since("1.6.0")
+@Experimental
+class BisectingKMeans private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var minDivisibleClusterSize: Double,
+ private var seed: Long) extends Logging {
+
+ import BisectingKMeans._
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(4, 20, 1.0, classOf[BisectingKMeans].getName.##)
+
+ /**
+ * Sets the desired number of leaf clusters (default: 4).
+ * The actual number could be smaller if there are no divisible leaf clusters.
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ require(k > 0, s"k must be positive but got $k.")
+ this.k = k
+ this
+ }
+
+ /**
+ * Gets the desired number of leaf clusters.
+ */
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the max number of k-means iterations to split clusters (default: 20).
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ require(maxIterations > 0, s"maxIterations must be positive but got $maxIterations.")
+ this.maxIterations = maxIterations
+ this
+ }
+
+ /**
+ * Gets the max number of k-means iterations to split clusters.
+ */
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the minimum number of points (if >= `1.0`) or the minimum proportion of points
+ * (if < `1.0`) of a divisible cluster (default: 1).
+ */
+ @Since("1.6.0")
+ def setMinDivisibleClusterSize(minDivisibleClusterSize: Double): this.type = {
+ require(minDivisibleClusterSize > 0.0,
+ s"minDivisibleClusterSize must be positive but got $minDivisibleClusterSize.")
+ this.minDivisibleClusterSize = minDivisibleClusterSize
+ this
+ }
+
+ /**
+ * Gets the minimum number of points (if >= `1.0`) or the minimum proportion of points
+ * (if < `1.0`) of a divisible cluster.
+ */
+ @Since("1.6.0")
+ def getMinDivisibleClusterSize: Double = minDivisibleClusterSize
+
+ /**
+ * Sets the random seed (default: hash value of the class name).
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ /**
+ * Gets the random seed.
+ */
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm.
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ if (input.getStorageLevel == StorageLevel.NONE) {
+ logWarning(s"The input RDD ${input.id} is not directly cached, which may hurt performance if"
+ + " its parent RDDs are also not cached.")
+ }
+ val d = input.map(_.size).first()
+ logInfo(s"Feature dimension: $d.")
+ // Compute and cache vector norms for fast distance computation.
+ val norms = input.map(v => Vectors.norm(v, 2.0)).persist(StorageLevel.MEMORY_AND_DISK)
+ val vectors = input.zip(norms).map { case (x, norm) => new VectorWithNorm(x, norm) }
+ var assignments = vectors.map(v => (ROOT_INDEX, v))
+ var activeClusters = summarize(d, assignments)
+ val rootSummary = activeClusters(ROOT_INDEX)
+ val n = rootSummary.size
+ logInfo(s"Number of points: $n.")
+ logInfo(s"Initial cost: ${rootSummary.cost}.")
+ val minSize = if (minDivisibleClusterSize >= 1.0) {
+ math.ceil(minDivisibleClusterSize).toLong
+ } else {
+ math.ceil(minDivisibleClusterSize * n).toLong
+ }
+ logInfo(s"The minimum number of points of a divisible cluster is $minSize.")
+ var inactiveClusters = mutable.Seq.empty[(Long, ClusterSummary)]
+ val random = new Random(seed)
+ var numLeafClustersNeeded = k - 1
+ var level = 1
+ while (activeClusters.nonEmpty && numLeafClustersNeeded > 0 && level < 63) {
+ // Divisible clusters are sufficiently large and have non-trivial cost.
+ var divisibleClusters = activeClusters.filter { case (_, summary) =>
+ (summary.size >= minSize) && (summary.cost > MLUtils.EPSILON * summary.size)
+ }
+ // If we don't need all divisible clusters, take the larger ones.
+ if (divisibleClusters.size > numLeafClustersNeeded) {
+ divisibleClusters = divisibleClusters.toSeq.sortBy { case (_, summary) =>
+ -summary.size
+ }.take(numLeafClustersNeeded)
+ .toMap
+ }
+ if (divisibleClusters.nonEmpty) {
+ val divisibleIndices = divisibleClusters.keys.toSet
+ logInfo(s"Dividing ${divisibleIndices.size} clusters on level $level.")
+ var newClusterCenters = divisibleClusters.flatMap { case (index, summary) =>
+ val (left, right) = splitCenter(summary.center, random)
+ Iterator((leftChildIndex(index), left), (rightChildIndex(index), right))
+ }.map(identity) // workaround for a Scala bug (SI-7005) that produces a not serializable map
+ var newClusters: Map[Long, ClusterSummary] = null
+ var newAssignments: RDD[(Long, VectorWithNorm)] = null
+ for (iter <- 0 until maxIterations) {
--- End diff --
I think we should stop the iteration, when the total cost is enough saturated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155175092
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155158286
**[Test build #45396 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45396/consoleFull)** for PR 5267 at commit [`75ca2a0`](https://github.com/apache/spark/commit/75ca2a000936fe62fb53293d805b2888c67cb563).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42820987
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
--- End diff --
* remove it from constructor args, move it to `run` method instead
* `BigInt` -> `Long`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151326036
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/5267
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-150285114
**[Test build #44157 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44157/consoleFull)** for PR 5267 at commit [`31623ea`](https://github.com/apache/spark/commit/31623ead10d4664c7c5d98b5c72731bcc804bcba).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class BisectingKMeansModel @Since(\"1.6.0\") (`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42820980
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = BisectingKMeans.initData(input).cache()
+ while (clusterStats.size < maxAllNodesInTree && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ val leafClusters = BisectingKMeans.summarizeClusters(data)
+ val dividableLeafClusters = leafClusters.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusters
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ else {
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided =
+ BisectingKMeans.divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = BisectingKMeans.updateClusterIndex(data, divided).cache()
+ rddArray = rddArray ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (rddArray.length > 1) {
+ val head = rddArray.head
+ head.unpersist()
+ rddArray = rddArray.filterNot(_.hashCode() == head.hashCode())
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+ }
+ }
+ // unpersist kept RDDs
+ rddArray.foreach(_.unpersist())
+ // create a map of cluster node with their criterions
+ val nodes = BisectingKMeans.createClusterNodes(data, clusterStats)
+
+ // build a cluster tree by Map class which is expressed
+ logInfo(s"Building the cluster tree is started in ${sc.appName}")
+ val root = BisectingKMeans.buildTree(nodes, BisectingKMeans.ROOT_INDEX_KEY, this.k)
+ if (root.isEmpty) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusterStats")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ logInfo(s"Elapsed Time for ${this.getClass.getSimpleName} Training: ${finishTime} [sec]")
+
+ // make a bisecting kmeans model
+ val model = new BisectingKMeansModel(root.get)
+ val leavesNodes = model.getClusters
+ if (leavesNodes.length < this.k) {
+ logWarning(s"# clusters is less than you want: ${leavesNodes.length} / ${k}")
+ }
+ model
+ }
+}
+
+
+private[clustering] object BisectingKMeans {
+
+ val ROOT_INDEX_KEY: BigInt = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ def findClosestCenter(metric: (BV[Double], BV[Double]) => Double)
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)
+ closestIndex
+ }
+
+ /**
+ * Summarizes data by each cluster as Map
+ *
+ * @param data pairs of point and its cluster index
+ */
+ def summarizeClusters(data: RDD[(BigInt, BV[Double])]): Map[BigInt, BisectingClusterStat] = {
+
+ val stats = data.mapPartitions { iter =>
+ // calculate the accumulation of the all point in a partition and count the rows
+ val map = mutable.Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx: BigInt, point: BV[Double]) =>
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(idx, (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size)))
+ map(idx) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ map.toIterator
+ }.reduceByKey { case ((sum1, n1, sumOfSquares1), (sum2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sum1 + sum2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.collect().toMap
+
+ stats.map {case (i, stat) => i -> new BisectingClusterStat(stat._2.toLong, stat._1, stat._3)}
+ }
+
+ /**
+ * Assigns the initial cluster index id to all data
+ */
+ def initData(data: RDD[Vector]): RDD[(BigInt, BV[Double])] = {
+ data.map { v: Vector => (BisectingKMeans.ROOT_INDEX_KEY, v.toBreeze)}
+ }
+
+ /**
+ * Gets the initial centers for bisect k-means
+ *
+ * @param data pairs of point and its cluster index
+ * @param stats pairs of cluster index and cluster statistics
+ */
+ def initNextCenters(
--- End diff --
maybe we can do this:
1. select a random (with pre-determined seed) vector `p` of length in-cluster standard deviation for perturbation
2. use `c + p` and `c - p` as the new initial centers
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42820990
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
--- End diff --
`import BisectingKMeans._`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151319812
**[Test build #44382 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44382/consoleFull)** for PR 5267 at commit [`2e8226d`](https://github.com/apache/spark/commit/2e8226d509ead4c14940335bc44c110d4c02c52f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class BisectingKMeansModel @Since(\"1.6.0\") (`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r27632117
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/HierarchicalClustering.scala ---
@@ -0,0 +1,610 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseVector => BDV, SparseVector => BSV, Vector => BV, norm => breezeNorm}
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.random.XORShiftRandom
+import org.apache.spark.{Logging, SparkException}
+
+import scala.collection.{Map, mutable}
+
+
+/**
+ * Top-level methods for calling the hierarchical clustering algorithm
+ */
+object HierarchicalClustering extends Logging {
+
+ private[clustering] val ROOT_INDEX_KEY: Long = 1
+
+ /**
+ * Trains a hierarchical clustering model with the given data
+ *
+ * @param data trained data
+ * @param numClusters the maximum number of clusters you want
+ * @return a hierarchical clustering model
+ */
+ def train(data: RDD[Vector], numClusters: Int): HierarchicalClusteringModel = {
+ val algo = new HierarchicalClustering().setNumClusters(numClusters)
+ algo.run(data)
+ }
+
+ /**
+ * Trains a hierarchical clustering model with the given data
+ *
+ * @param data training data
+ * @param numClusters the maximum number of clusters you want
+ * @param maxIterations the number of maximal iterations
+ * @param maxRetries the number of maximum retries when the clustering can't be succeeded
+ * @param seed the randomseed to generate the initial vectors for each bisecting
+ * @return a hierarchical clustering model
+ */
+ def train(data: RDD[Vector],
+ numClusters: Int,
+ maxIterations: Int,
+ maxRetries: Int,
+ seed: Int): HierarchicalClusteringModel = {
+
+ val algo = new HierarchicalClustering().setNumClusters(numClusters)
+ .setMaxIterations(maxIterations)
+ .setMaxRetries(maxRetries)
+ .setSeed(seed)
+ algo.run(data)
+ }
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ private[mllib]
+ def findClosestCenter(metric: Function2[BV[Double], BV[Double], Double])
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)
+ closestIndex
+ }
+}
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bi-sect k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * @param numClusters tne number of clusters you want
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations
+ * @param maxRetries the number of maximum retries
+ * @param seed a random seed
+ */
+class HierarchicalClustering(
+ private var numClusters: Int,
+ private var clusterMap: Map[Long, ClusterTree],
+ private var maxIterations: Int,
+ private var maxRetries: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ def this() = this(20, mutable.ListMap.empty[Long, ClusterTree], 20, 10, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ def setNumClusters(numClusters: Int): this.type = {
+ this.numClusters = numClusters
+ this
+ }
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ def getSubIterations(): Int = this.maxIterations
--- End diff --
Got it. I am going to remove them. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152342812
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152362400
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151313829
**[Test build #44384 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44384/consoleFull)** for PR 5267 at commit [`622499e`](https://github.com/apache/spark/commit/622499e95e2717625eecd11f5c43e0c9eddc820e).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r43071613
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,744 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg.{SparseVector => BSV, Vector => BV, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the costs, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The costs means how large the cluster is. That is, the cluster
+ * whose cost is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ import BisectingKMeans._
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[Long, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var updatedDataHistory = Array.empty[RDD[(Long, BV[Double])]]
+ // the minimum number of nodes of a binary tree by given parameter
+ val numNodeLimit = getMinimumNumNodesInTree(this.k)
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = initData(input).cache()
+ while (clusterStats.size < numNodeLimit && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ // TODO Remove non-leaf cluster stats from `leafClusterStats`
+ val leafClusterStats = summarizeClusters(data)
+ val dividableLeafClusters = leafClusterStats.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusterStats
+
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided = divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = updateClusterIndex(data, divided).cache()
+ updatedDataHistory = updatedDataHistory ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (updatedDataHistory.length > 1) {
+ val head = updatedDataHistory.head
+ updatedDataHistory = updatedDataHistory.tail
+ head.unpersist()
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ }
+ // create a map of cluster node with their costs
+ val nodes = createClusterNodes(data, clusterStats)
+ // unpersist RDDs
+ data.unpersist()
+ updatedDataHistory.foreach(_.unpersist())
+
+ // build a cluster tree by Map class which is expressed
+ logInfo(s"Building the cluster tree is started in ${sc.appName}")
+ val root = buildTree(nodes, ROOT_INDEX_KEY, this.k)
+ if (root.isEmpty) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusterStats")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ logInfo(s"Elapsed Time for ${this.getClass.getSimpleName} Training: ${finishTime} [sec]")
+
+ // make a bisecting kmeans model
+ val model = new BisectingKMeansModel(root.get)
+ val leavesNodes = model.getClusters
+ if (leavesNodes.length < this.k) {
+ logWarning(s"# clusters is less than you want: ${leavesNodes.length} / ${k}")
+ }
+ model
+ }
+}
+
+
+private[clustering] object BisectingKMeans {
+
+ import BisectingClusterStat._
+
+ val ROOT_INDEX_KEY: Long = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ def findClosestCenter(metric: (BV[Double], BV[Double]) => Double)
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ // get the closest index
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)._2
+ }
+
+ /**
+ * Gets the minimum number of nodes in a tree by the number of leaves
+ *
+ * @param k: the number of leaf nodes
+ */
+ def getMinimumNumNodesInTree(k: Int): Int = {
+ val multiplier = math.ceil(math.log(k) / math.log(2.0))
+ // the calculation is same as `math.pow(2, multiplier)`
+ var numNodes = 2
+ (1 to multiplier.toInt).foreach (i => numNodes = numNodes << 1)
+ numNodes
+ }
+
+ /**
+ * Summarizes data by each cluster as Map
+ *
+ * @param data pairs of point and its cluster index
+ */
+ def summarizeClusters(data: RDD[(Long, BV[Double])]): Map[Long, BisectingClusterStat] = {
+
+ data.mapPartitions { iter =>
+ // calculate the accumulation of the all point in a partition and count the rows
+ val map = mutable.Map.empty[Long, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx: Long, point: BV[Double]) =>
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(idx, (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size)))
+ map(idx) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
--- End diff --
We don't need a vector for `sumOfSquares` because we only uses its sum to compute in-cluster variance. A single scalar should work.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155175125
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42821017
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = BisectingKMeans.initData(input).cache()
+ while (clusterStats.size < maxAllNodesInTree && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ val leafClusters = BisectingKMeans.summarizeClusters(data)
+ val dividableLeafClusters = leafClusters.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusters
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ else {
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided =
+ BisectingKMeans.divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = BisectingKMeans.updateClusterIndex(data, divided).cache()
+ rddArray = rddArray ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (rddArray.length > 1) {
+ val head = rddArray.head
+ head.unpersist()
+ rddArray = rddArray.filterNot(_.hashCode() == head.hashCode())
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+ }
+ }
+ // unpersist kept RDDs
+ rddArray.foreach(_.unpersist())
+ // create a map of cluster node with their criterions
+ val nodes = BisectingKMeans.createClusterNodes(data, clusterStats)
+
+ // build a cluster tree by Map class which is expressed
+ logInfo(s"Building the cluster tree is started in ${sc.appName}")
+ val root = BisectingKMeans.buildTree(nodes, BisectingKMeans.ROOT_INDEX_KEY, this.k)
+ if (root.isEmpty) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusterStats")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ logInfo(s"Elapsed Time for ${this.getClass.getSimpleName} Training: ${finishTime} [sec]")
+
+ // make a bisecting kmeans model
+ val model = new BisectingKMeansModel(root.get)
+ val leavesNodes = model.getClusters
+ if (leavesNodes.length < this.k) {
+ logWarning(s"# clusters is less than you want: ${leavesNodes.length} / ${k}")
+ }
+ model
+ }
+}
+
+
+private[clustering] object BisectingKMeans {
+
+ val ROOT_INDEX_KEY: BigInt = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ def findClosestCenter(metric: (BV[Double], BV[Double]) => Double)
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
--- End diff --
`closetCenter` is not used
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42822007
--- Diff: mllib/src/test/java/org/apache/spark/mllib/clustering/JavaBisectingKMeansSuite.java ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering;
+
+import com.google.common.collect.Lists;
+import org.apache.spark.api.java.JavaRDD;
--- End diff --
reorganize imports
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152275043
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-150136685
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42821607
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = BisectingKMeans.initData(input).cache()
+ while (clusterStats.size < maxAllNodesInTree && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ val leafClusters = BisectingKMeans.summarizeClusters(data)
+ val dividableLeafClusters = leafClusters.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusters
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ else {
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided =
+ BisectingKMeans.divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = BisectingKMeans.updateClusterIndex(data, divided).cache()
+ rddArray = rddArray ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (rddArray.length > 1) {
+ val head = rddArray.head
+ head.unpersist()
+ rddArray = rddArray.filterNot(_.hashCode() == head.hashCode())
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+ }
+ }
+ // unpersist kept RDDs
+ rddArray.foreach(_.unpersist())
+ // create a map of cluster node with their criterions
+ val nodes = BisectingKMeans.createClusterNodes(data, clusterStats)
+
+ // build a cluster tree by Map class which is expressed
+ logInfo(s"Building the cluster tree is started in ${sc.appName}")
+ val root = BisectingKMeans.buildTree(nodes, BisectingKMeans.ROOT_INDEX_KEY, this.k)
+ if (root.isEmpty) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusterStats")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ logInfo(s"Elapsed Time for ${this.getClass.getSimpleName} Training: ${finishTime} [sec]")
+
+ // make a bisecting kmeans model
+ val model = new BisectingKMeansModel(root.get)
+ val leavesNodes = model.getClusters
+ if (leavesNodes.length < this.k) {
+ logWarning(s"# clusters is less than you want: ${leavesNodes.length} / ${k}")
+ }
+ model
+ }
+}
+
+
+private[clustering] object BisectingKMeans {
+
+ val ROOT_INDEX_KEY: BigInt = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ def findClosestCenter(metric: (BV[Double], BV[Double]) => Double)
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)
+ closestIndex
+ }
+
+ /**
+ * Summarizes data by each cluster as Map
+ *
+ * @param data pairs of point and its cluster index
+ */
+ def summarizeClusters(data: RDD[(BigInt, BV[Double])]): Map[BigInt, BisectingClusterStat] = {
+
+ val stats = data.mapPartitions { iter =>
+ // calculate the accumulation of the all point in a partition and count the rows
+ val map = mutable.Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx: BigInt, point: BV[Double]) =>
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(idx, (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size)))
+ map(idx) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ map.toIterator
+ }.reduceByKey { case ((sum1, n1, sumOfSquares1), (sum2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sum1 + sum2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.collect().toMap
+
+ stats.map {case (i, stat) => i -> new BisectingClusterStat(stat._2.toLong, stat._1, stat._3)}
+ }
+
+ /**
+ * Assigns the initial cluster index id to all data
+ */
+ def initData(data: RDD[Vector]): RDD[(BigInt, BV[Double])] = {
+ data.map { v: Vector => (BisectingKMeans.ROOT_INDEX_KEY, v.toBreeze)}
+ }
+
+ /**
+ * Gets the initial centers for bisect k-means
+ *
+ * @param data pairs of point and its cluster index
+ * @param stats pairs of cluster index and cluster statistics
+ */
+ def initNextCenters(
+ data: RDD[(BigInt, BV[Double])],
+ stats: Map[BigInt, BisectingClusterStat]): Map[BigInt, BV[Double]] = {
+
+ // Since the combination sampleByKey and groupByKey is more expensive,
+ // this as follows would be better.
+ val bcIndeces = data.sparkContext.broadcast(stats.keySet)
+ val samples = data.mapPartitions { iter =>
+ val map = mutable.Map.empty[BigInt, mutable.ArrayBuffer[BV[Double]]]
+
+ bcIndeces.value.foreach {i => map(i) = mutable.ArrayBuffer.empty[BV[Double]]}
+ val LOCAL_SAMPLE_SIZE = 100
+ iter.foreach { case (i, point) =>
+ map(i).append(point)
+ // to avoid to increase the memory usage on each map thread,
+ // the number of elements is cut off at the right time.
+ if (map(i).size > LOCAL_SAMPLE_SIZE) {
+ val elements = map(i).sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ map(i) = mutable.ArrayBuffer(elements.head, elements.last)
+ }
+ }
+
+ // in order to reduce the shuffle size, take only two elements
+ map.filterNot(_._2.isEmpty).map { case (i, points) =>
+ val elements = map(i).toSeq.sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ i -> mutable.ArrayBuffer(elements.head, elements.last)
+ }.toIterator
+ }.reduceByKey { case (points1, points2) =>
+ points1.union(points2)
+ }.collect()
+
+ val nextCenters = samples.flatMap { case (i, points) =>
+ val elements = points.toSeq.sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ Array((2 * i, elements.head), (2 * i + 1, elements.last))
+ }.toMap
+ if (!stats.keySet.flatMap(idx => Array(2 * idx, 2 * idx + 1)).forall(nextCenters.contains(_))) {
+ throw new SparkException("Failed to initialize centers for next step")
+ }
+ nextCenters
+ }
+
+ /**
+ * Updates the indexes of clusters which is divided to its children indexes
+ *
+ * @param data pairs of point and its cluster index
+ * @param dividedClusters pairs of cluster index and cluster statistics
+ */
+ def updateClusterIndex(
+ data: RDD[(BigInt, BV[Double])],
+ dividedClusters: Map[BigInt, BisectingClusterStat]): RDD[(BigInt, BV[Double])] = {
+
+ // extract the centers of the clusters
+ val sc = data.sparkContext
+ var centers = dividedClusters.map { case (idx, cluster) => (idx, cluster.center)}
+ val bcCenters = sc.broadcast(centers)
+
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val bcMetric = sc.broadcast(metric)
+
+ // update the indexes to their children indexes
+ data.map { case (idx, point) =>
+ val childrenIndexes = Array(2 * idx, 2 * idx + 1).filter(c => bcCenters.value.contains(c))
+ childrenIndexes.length match {
+ // stay the index if the number of children is not enough
+ case s if s < 2 => (idx, point)
+ // update the indexes
+ case _ => {
+ val nextCenters = childrenIndexes.map(bcCenters.value(_))
+ val closestIndex = BisectingKMeans
+ .findClosestCenter(bcMetric.value)(nextCenters)(point)
+ val nextIndex = 2 * idx + closestIndex
+ (nextIndex, point)
+ }
+ }
+ }
+ }
+
+ /**
+ * Divides clusters according to their statistics
+ *
+ * @param data pairs of point and its cluster index
+ * @param targetClusters target clusters to divide
+ * @param maxIterations the maximum iterations to calculate clusters statistics
+ */
+ def divideClusters(
+ data: RDD[(BigInt, BV[Double])],
+ targetClusters: Map[BigInt, BisectingClusterStat],
+ maxIterations: Int): Map[BigInt, BisectingClusterStat] = {
+ val sc = data.sparkContext
+ val appName = sc.appName
+
+ // get keys of dividable clusters
+ val dividableClusters = targetClusters.filter { case (idx, cluster) => cluster.isDividable }
+ if (dividableClusters.isEmpty) {
+ return Map.empty[BigInt, BisectingClusterStat]
+ }
+ // extract dividable input data
+ val dividableData = data.filter { case (idx, point) => dividableClusters.contains(idx)}
+
+ var newCenters = BisectingKMeans.initNextCenters(dividableData, dividableClusters)
+ var bcNewCenters = sc.broadcast(newCenters)
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val bcMetric = sc.broadcast(metric)
+ var stats = Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+
+ var subIter = 0
+ var totalStd = Double.MaxValue
+ var oldTotalStd = Double.MaxValue
+ var relativeError = Double.MaxValue
+ while (subIter < maxIterations && relativeError > 10E-4) {
+ // calculate summary of each cluster
+ val eachStats = dividableData.mapPartitions { iter =>
+ val map = mutable.Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx, point) =>
+ // calculate next index number
+ val childrenCenters = Array(2 * idx, 2 * idx + 1)
+ .filter(x => bcNewCenters.value.contains(x)).map(bcNewCenters.value(_))
+ if (childrenCenters.length == 2) {
+ val closestIndex =
+ BisectingKMeans.findClosestCenter(bcMetric.value)(childrenCenters)(point)
+ val nextIndex = 2 * idx + closestIndex
+
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(
+ nextIndex,
+ (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size))
+ )
+ map(nextIndex) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ }
+ map.toIterator
+ }.reduceByKey { case ((sv1, n1, sumOfSquares1), (sv2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sv1 + sv2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.collect().toMap
+
+ // calculate the center of each cluster
+ newCenters = eachStats.map { case (idx, (sum, n, sumOfSquares)) => (idx, sum :/ n)}
+ bcNewCenters = sc.broadcast(newCenters)
+
+ // update summary of each cluster
+ stats = eachStats.toMap
+
+ totalStd = stats.map { case (idx, (sum, n, sumOfSquares)) =>
+ breezeSum((sumOfSquares :/ n) :- breezeNorm(sum :/ n, 2.0))
+ }.sum
+ relativeError = math.abs(oldTotalStd - totalStd) / totalStd
+ oldTotalStd = totalStd
+ subIter += 1
+ }
+ stats.map { case (i, stat) => i -> new BisectingClusterStat(stat._2.toLong, stat._1, stat._3) }
+ }
+
+ /**
+ * Creates the map of cluster stats to the map of cluster nodes with their criterions
+ *
+ * @param data input data
+ * @param stats map of cluster stats which is described as a binary tree
+ */
+ def createClusterNodes(
+ data: RDD[(BigInt, BV[Double])],
+ stats: Map[BigInt, BisectingClusterStat]): Map[BigInt, BisectingClusterNode] = {
+
+ // TODO: support other criterion, such as entropy
+ createClusterNodesWithAverageCost(data, stats)
+ }
+
+ /**
+ * Creates the map of cluster stats to the map of cluster nodes with their average costs
+ */
+ private def createClusterNodesWithAverageCost(
+ data: RDD[(BigInt, BV[Double])],
+ stats: Map[BigInt, BisectingClusterStat]): Map[BigInt, BisectingClusterNode] = {
+
+ // calculate average costs of all clusters
+ val bcCenters = data.sparkContext.broadcast(stats.map { case (i, stat) => i -> stat.center })
+ val costs = data.mapPartitions { iter =>
+ val counters = mutable.Map.empty[BigInt, (Long, Double)]
+ bcCenters.value.foreach {case (i, center) => counters(i) = (0L, 0.0)}
+ iter.foreach { case (i, point) =>
+ val cost = breezeNorm(bcCenters.value.apply(i) - point, 2.0)
+ counters(i) = (counters(i)._1 + 1, counters(i)._2 + cost)
+ }
+ counters.toIterator
+ }.reduceByKey { case((n1, cost1), (n2, cost2)) =>
+ (n1 + n2, cost1 + cost2)
+ }.collectAsMap()
+
+ stats.map { case (i, stat) =>
+ val avgCost = costs(i)._1 match {
+ case x if x == 0.0 => 0.0
+ case _ => costs(i)._2 / costs(i)._1
+ }
+ i -> new BisectingClusterNode(Vectors.fromBreeze(stat.center), stat.rows, avgCost)
+ }
+ }
+
+ /**
+ * Builds a cluster tree from a Map of clusters
+ *
+ * @param treeMap divided clusters as a Map class
+ * @param rootIndex index you want to start
+ * @param numClusters the number of clusters you want
+ * @return a built cluster tree
+ */
+ private def buildTree(
+ treeMap: Map[BigInt, BisectingClusterNode],
+ rootIndex: BigInt,
+ numClusters: Int): Option[BisectingClusterNode] = {
+
+ // if there is no index in the Map
+ if (!treeMap.contains(rootIndex)) return None
+
+ // build a cluster tree if the queue is empty or until the number of leaf clusters is enough
+ var numLeavesClusters = 1
+ val root = treeMap(rootIndex)
+ var leavesQueue = Map(rootIndex -> root)
+ while (leavesQueue.nonEmpty && numLeavesClusters < numClusters) {
+ // pick up the largest cluster by the maximum criterion of all the clusters
+ val mostScattered = leavesQueue.maxBy(_._2.criterion)
+ val mostScatteredKey = mostScattered._1
+ val mostScatteredCluster = mostScattered._2
+
+ // relate the most scattered cluster to its children clusters
+ val childrenIndexes = Array(2 * mostScatteredKey, 2 * mostScatteredKey + 1)
+ if (childrenIndexes.forall(i => treeMap.contains(i))) {
+ // insert children to the most scattered cluster
+ val children = childrenIndexes.map(i => treeMap(i))
+ mostScatteredCluster.insert(children)
+
+ // calculate the local dendrogram height
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val localHeight = children
+ .map(child => metric(child.center.toBreeze, mostScatteredCluster.center.toBreeze)).max
+ mostScatteredCluster.setLocalHeight(localHeight)
+
+ // update the queue
+ leavesQueue = leavesQueue ++ childrenIndexes.map(i => i -> treeMap(i)).toMap
+ numLeavesClusters += 1
+ }
+
+ // remove the cluster which is involved to the cluster tree
+ leavesQueue = leavesQueue.filterNot(_ == mostScattered)
+ }
+ Some(root)
+ }
+}
+
+/**
+ * A cluster as a tree node which can have its sub nodes
+ *
+ * @param center the center of the cluster
+ * @param rows the number of rows in the cluster
+ * @param criterion how large a cluster is
+ * @param localHeight the maximal distance between this node and its children
+ * @param parent the parent cluster of the cluster
+ * @param children the children nodes of the cluster
+ */
+@Since("1.6.0")
+class BisectingClusterNode private (
--- End diff --
Maybe it is useful to use the same parameter names as `TreeNode`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152262191
**[Test build #44615 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44615/consoleFull)** for PR 5267 at commit [`a876ba2`](https://github.com/apache/spark/commit/a876ba233e197bb5228a05fe2946523af1b40ee6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-87686119
[Test build #29402 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29402/consoleFull) for PR 5267 at commit [`af0f65b`](https://github.com/apache/spark/commit/af0f65bb4726315c076b827d37276616c5218010).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42821747
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = BisectingKMeans.initData(input).cache()
+ while (clusterStats.size < maxAllNodesInTree && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ val leafClusters = BisectingKMeans.summarizeClusters(data)
+ val dividableLeafClusters = leafClusters.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusters
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ else {
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided =
+ BisectingKMeans.divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = BisectingKMeans.updateClusterIndex(data, divided).cache()
+ rddArray = rddArray ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (rddArray.length > 1) {
+ val head = rddArray.head
+ head.unpersist()
+ rddArray = rddArray.filterNot(_.hashCode() == head.hashCode())
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+ }
+ }
+ // unpersist kept RDDs
+ rddArray.foreach(_.unpersist())
+ // create a map of cluster node with their criterions
+ val nodes = BisectingKMeans.createClusterNodes(data, clusterStats)
+
+ // build a cluster tree by Map class which is expressed
+ logInfo(s"Building the cluster tree is started in ${sc.appName}")
+ val root = BisectingKMeans.buildTree(nodes, BisectingKMeans.ROOT_INDEX_KEY, this.k)
+ if (root.isEmpty) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusterStats")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ logInfo(s"Elapsed Time for ${this.getClass.getSimpleName} Training: ${finishTime} [sec]")
+
+ // make a bisecting kmeans model
+ val model = new BisectingKMeansModel(root.get)
+ val leavesNodes = model.getClusters
+ if (leavesNodes.length < this.k) {
+ logWarning(s"# clusters is less than you want: ${leavesNodes.length} / ${k}")
+ }
+ model
+ }
+}
+
+
+private[clustering] object BisectingKMeans {
+
+ val ROOT_INDEX_KEY: BigInt = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ def findClosestCenter(metric: (BV[Double], BV[Double]) => Double)
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)
+ closestIndex
+ }
+
+ /**
+ * Summarizes data by each cluster as Map
+ *
+ * @param data pairs of point and its cluster index
+ */
+ def summarizeClusters(data: RDD[(BigInt, BV[Double])]): Map[BigInt, BisectingClusterStat] = {
+
+ val stats = data.mapPartitions { iter =>
+ // calculate the accumulation of the all point in a partition and count the rows
+ val map = mutable.Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx: BigInt, point: BV[Double]) =>
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(idx, (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size)))
+ map(idx) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ map.toIterator
+ }.reduceByKey { case ((sum1, n1, sumOfSquares1), (sum2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sum1 + sum2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.collect().toMap
+
+ stats.map {case (i, stat) => i -> new BisectingClusterStat(stat._2.toLong, stat._1, stat._3)}
+ }
+
+ /**
+ * Assigns the initial cluster index id to all data
+ */
+ def initData(data: RDD[Vector]): RDD[(BigInt, BV[Double])] = {
+ data.map { v: Vector => (BisectingKMeans.ROOT_INDEX_KEY, v.toBreeze)}
+ }
+
+ /**
+ * Gets the initial centers for bisect k-means
+ *
+ * @param data pairs of point and its cluster index
+ * @param stats pairs of cluster index and cluster statistics
+ */
+ def initNextCenters(
+ data: RDD[(BigInt, BV[Double])],
+ stats: Map[BigInt, BisectingClusterStat]): Map[BigInt, BV[Double]] = {
+
+ // Since the combination sampleByKey and groupByKey is more expensive,
+ // this as follows would be better.
+ val bcIndeces = data.sparkContext.broadcast(stats.keySet)
+ val samples = data.mapPartitions { iter =>
+ val map = mutable.Map.empty[BigInt, mutable.ArrayBuffer[BV[Double]]]
+
+ bcIndeces.value.foreach {i => map(i) = mutable.ArrayBuffer.empty[BV[Double]]}
+ val LOCAL_SAMPLE_SIZE = 100
+ iter.foreach { case (i, point) =>
+ map(i).append(point)
+ // to avoid to increase the memory usage on each map thread,
+ // the number of elements is cut off at the right time.
+ if (map(i).size > LOCAL_SAMPLE_SIZE) {
+ val elements = map(i).sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ map(i) = mutable.ArrayBuffer(elements.head, elements.last)
+ }
+ }
+
+ // in order to reduce the shuffle size, take only two elements
+ map.filterNot(_._2.isEmpty).map { case (i, points) =>
+ val elements = map(i).toSeq.sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ i -> mutable.ArrayBuffer(elements.head, elements.last)
+ }.toIterator
+ }.reduceByKey { case (points1, points2) =>
+ points1.union(points2)
+ }.collect()
+
+ val nextCenters = samples.flatMap { case (i, points) =>
+ val elements = points.toSeq.sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ Array((2 * i, elements.head), (2 * i + 1, elements.last))
+ }.toMap
+ if (!stats.keySet.flatMap(idx => Array(2 * idx, 2 * idx + 1)).forall(nextCenters.contains(_))) {
+ throw new SparkException("Failed to initialize centers for next step")
+ }
+ nextCenters
+ }
+
+ /**
+ * Updates the indexes of clusters which is divided to its children indexes
+ *
+ * @param data pairs of point and its cluster index
+ * @param dividedClusters pairs of cluster index and cluster statistics
+ */
+ def updateClusterIndex(
+ data: RDD[(BigInt, BV[Double])],
+ dividedClusters: Map[BigInt, BisectingClusterStat]): RDD[(BigInt, BV[Double])] = {
+
+ // extract the centers of the clusters
+ val sc = data.sparkContext
+ var centers = dividedClusters.map { case (idx, cluster) => (idx, cluster.center)}
+ val bcCenters = sc.broadcast(centers)
+
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val bcMetric = sc.broadcast(metric)
+
+ // update the indexes to their children indexes
+ data.map { case (idx, point) =>
+ val childrenIndexes = Array(2 * idx, 2 * idx + 1).filter(c => bcCenters.value.contains(c))
+ childrenIndexes.length match {
+ // stay the index if the number of children is not enough
+ case s if s < 2 => (idx, point)
+ // update the indexes
+ case _ => {
+ val nextCenters = childrenIndexes.map(bcCenters.value(_))
+ val closestIndex = BisectingKMeans
+ .findClosestCenter(bcMetric.value)(nextCenters)(point)
+ val nextIndex = 2 * idx + closestIndex
+ (nextIndex, point)
+ }
+ }
+ }
+ }
+
+ /**
+ * Divides clusters according to their statistics
+ *
+ * @param data pairs of point and its cluster index
+ * @param targetClusters target clusters to divide
+ * @param maxIterations the maximum iterations to calculate clusters statistics
+ */
+ def divideClusters(
+ data: RDD[(BigInt, BV[Double])],
+ targetClusters: Map[BigInt, BisectingClusterStat],
+ maxIterations: Int): Map[BigInt, BisectingClusterStat] = {
+ val sc = data.sparkContext
+ val appName = sc.appName
+
+ // get keys of dividable clusters
+ val dividableClusters = targetClusters.filter { case (idx, cluster) => cluster.isDividable }
+ if (dividableClusters.isEmpty) {
+ return Map.empty[BigInt, BisectingClusterStat]
+ }
+ // extract dividable input data
+ val dividableData = data.filter { case (idx, point) => dividableClusters.contains(idx)}
+
+ var newCenters = BisectingKMeans.initNextCenters(dividableData, dividableClusters)
+ var bcNewCenters = sc.broadcast(newCenters)
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val bcMetric = sc.broadcast(metric)
+ var stats = Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+
+ var subIter = 0
+ var totalStd = Double.MaxValue
+ var oldTotalStd = Double.MaxValue
+ var relativeError = Double.MaxValue
+ while (subIter < maxIterations && relativeError > 10E-4) {
+ // calculate summary of each cluster
+ val eachStats = dividableData.mapPartitions { iter =>
+ val map = mutable.Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx, point) =>
+ // calculate next index number
+ val childrenCenters = Array(2 * idx, 2 * idx + 1)
+ .filter(x => bcNewCenters.value.contains(x)).map(bcNewCenters.value(_))
+ if (childrenCenters.length == 2) {
+ val closestIndex =
+ BisectingKMeans.findClosestCenter(bcMetric.value)(childrenCenters)(point)
+ val nextIndex = 2 * idx + closestIndex
+
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(
+ nextIndex,
+ (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size))
+ )
+ map(nextIndex) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ }
+ map.toIterator
+ }.reduceByKey { case ((sv1, n1, sumOfSquares1), (sv2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sv1 + sv2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.collect().toMap
+
+ // calculate the center of each cluster
+ newCenters = eachStats.map { case (idx, (sum, n, sumOfSquares)) => (idx, sum :/ n)}
+ bcNewCenters = sc.broadcast(newCenters)
+
+ // update summary of each cluster
+ stats = eachStats.toMap
+
+ totalStd = stats.map { case (idx, (sum, n, sumOfSquares)) =>
+ breezeSum((sumOfSquares :/ n) :- breezeNorm(sum :/ n, 2.0))
+ }.sum
+ relativeError = math.abs(oldTotalStd - totalStd) / totalStd
+ oldTotalStd = totalStd
+ subIter += 1
+ }
+ stats.map { case (i, stat) => i -> new BisectingClusterStat(stat._2.toLong, stat._1, stat._3) }
+ }
+
+ /**
+ * Creates the map of cluster stats to the map of cluster nodes with their criterions
+ *
+ * @param data input data
+ * @param stats map of cluster stats which is described as a binary tree
+ */
+ def createClusterNodes(
+ data: RDD[(BigInt, BV[Double])],
+ stats: Map[BigInt, BisectingClusterStat]): Map[BigInt, BisectingClusterNode] = {
+
+ // TODO: support other criterion, such as entropy
+ createClusterNodesWithAverageCost(data, stats)
+ }
+
+ /**
+ * Creates the map of cluster stats to the map of cluster nodes with their average costs
+ */
+ private def createClusterNodesWithAverageCost(
+ data: RDD[(BigInt, BV[Double])],
+ stats: Map[BigInt, BisectingClusterStat]): Map[BigInt, BisectingClusterNode] = {
+
+ // calculate average costs of all clusters
+ val bcCenters = data.sparkContext.broadcast(stats.map { case (i, stat) => i -> stat.center })
+ val costs = data.mapPartitions { iter =>
+ val counters = mutable.Map.empty[BigInt, (Long, Double)]
+ bcCenters.value.foreach {case (i, center) => counters(i) = (0L, 0.0)}
+ iter.foreach { case (i, point) =>
+ val cost = breezeNorm(bcCenters.value.apply(i) - point, 2.0)
+ counters(i) = (counters(i)._1 + 1, counters(i)._2 + cost)
+ }
+ counters.toIterator
+ }.reduceByKey { case((n1, cost1), (n2, cost2)) =>
+ (n1 + n2, cost1 + cost2)
+ }.collectAsMap()
+
+ stats.map { case (i, stat) =>
+ val avgCost = costs(i)._1 match {
+ case x if x == 0.0 => 0.0
+ case _ => costs(i)._2 / costs(i)._1
+ }
+ i -> new BisectingClusterNode(Vectors.fromBreeze(stat.center), stat.rows, avgCost)
+ }
+ }
+
+ /**
+ * Builds a cluster tree from a Map of clusters
+ *
+ * @param treeMap divided clusters as a Map class
+ * @param rootIndex index you want to start
+ * @param numClusters the number of clusters you want
+ * @return a built cluster tree
+ */
+ private def buildTree(
+ treeMap: Map[BigInt, BisectingClusterNode],
+ rootIndex: BigInt,
+ numClusters: Int): Option[BisectingClusterNode] = {
+
+ // if there is no index in the Map
+ if (!treeMap.contains(rootIndex)) return None
+
+ // build a cluster tree if the queue is empty or until the number of leaf clusters is enough
+ var numLeavesClusters = 1
+ val root = treeMap(rootIndex)
+ var leavesQueue = Map(rootIndex -> root)
+ while (leavesQueue.nonEmpty && numLeavesClusters < numClusters) {
+ // pick up the largest cluster by the maximum criterion of all the clusters
+ val mostScattered = leavesQueue.maxBy(_._2.criterion)
+ val mostScatteredKey = mostScattered._1
+ val mostScatteredCluster = mostScattered._2
+
+ // relate the most scattered cluster to its children clusters
+ val childrenIndexes = Array(2 * mostScatteredKey, 2 * mostScatteredKey + 1)
+ if (childrenIndexes.forall(i => treeMap.contains(i))) {
+ // insert children to the most scattered cluster
+ val children = childrenIndexes.map(i => treeMap(i))
+ mostScatteredCluster.insert(children)
+
+ // calculate the local dendrogram height
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val localHeight = children
+ .map(child => metric(child.center.toBreeze, mostScatteredCluster.center.toBreeze)).max
+ mostScatteredCluster.setLocalHeight(localHeight)
+
+ // update the queue
+ leavesQueue = leavesQueue ++ childrenIndexes.map(i => i -> treeMap(i)).toMap
+ numLeavesClusters += 1
+ }
+
+ // remove the cluster which is involved to the cluster tree
+ leavesQueue = leavesQueue.filterNot(_ == mostScattered)
+ }
+ Some(root)
+ }
+}
+
+/**
+ * A cluster as a tree node which can have its sub nodes
+ *
+ * @param center the center of the cluster
+ * @param rows the number of rows in the cluster
+ * @param criterion how large a cluster is
+ * @param localHeight the maximal distance between this node and its children
+ * @param parent the parent cluster of the cluster
+ * @param children the children nodes of the cluster
+ */
+@Since("1.6.0")
+class BisectingClusterNode private (
+ @Since("1.6.0") val center: Vector,
+ @Since("1.6.0") val rows: Long,
+ @Since("1.6.0") val criterion: Double,
+ private var localHeight: Double,
+ private var parent: Option[BisectingClusterNode],
+ private var children: Seq[BisectingClusterNode]) extends Serializable {
+
+ require(!criterion.isNaN)
+
+ @Since("1.6.0")
+ def this(center: Vector, rows: Long, criterion: Double) =
+ this(center, rows, criterion, 0.0, None, Array.empty[BisectingClusterNode])
+
+ /**
+ * Inserts a sub node as its child
+ *
+ * @param child inserted sub node
+ */
+ @Since("1.6.0")
+ def insert(child: BisectingClusterNode) {
+ insert(Array(child))
+ }
+
+ /**
+ * Inserts sub nodes as its children
+ *
+ * @param children inserted sub nodes
+ */
+ @Since("1.6.0")
+ def insert(children: Array[BisectingClusterNode]) {
+ this.children = this.children ++ children
+ children.foreach(child => child.parent = Some(this))
+ }
+
+ /**
+ * Converts the tree into Array class
+ * the sub nodes are recursively expanded
+ *
+ * @return an Array class which the cluster tree is expanded
+ */
+ @Since("1.6.0")
+ def toArray: Array[BisectingClusterNode] = {
+ val array = this.children.size match {
+ case 0 => Array(this)
+ case _ => Array(this) ++ this.children.flatMap(child => child.toArray.toIterator)
+ }
+ array.sortWith { case (a, b) =>
+ a.getDepth < b.getDepth && a.criterion < b.criterion && a.rows < b.rows
+ }
+ }
+
+ /**
+ * Gets the depth of the cluster in the tree
+ *
+ * @return the depth from the root
+ */
+ @Since("1.6.0")
+ def getDepth: Int = {
+ this.parent match {
+ case None => 0
+ case _ => 1 + this.parent.get.getDepth
+ }
+ }
+
+ /**
+ * Gets the leaves nodes in the cluster tree
+ */
+ @Since("1.6.0")
+ def getLeavesNodes: Array[BisectingClusterNode] = {
+ this.toArray.filter(_.isLeaf).sortBy(_.center.toArray.sum)
+ }
+
+ @Since("1.6.0")
+ def isLeaf: Boolean = this.children.isEmpty
+
+ @Since("1.6.0")
+ def getParent: Option[BisectingClusterNode] = this.parent
+
+ @Since("1.6.0")
+ def getChildren: Seq[BisectingClusterNode] = this.children
+
+ /**
+ * Gets the dendrogram height of the cluster at the cluster tree.
+ * A dendrogram height is different from a local height.
+ * A dendrogram height means a total height of a node in a tree.
+ * A local height means a maximum distance between a node and its children.
+ *
+ * @return the dendrogram height
+ */
+ @Since("1.6.0")
+ def getHeight: Double = {
+ this.children.size match {
+ case 0 => 0.0
+ case _ => this.localHeight + this.children.map(_.getHeight).max
+ }
+ }
+
+ @Since("1.6.0")
+ def setLocalHeight(height: Double): Unit = this.localHeight = height
+
+ /**
+ * Converts to an adjacency list
+ *
+ * @return List[(fromNodeId, toNodeId, distance)]
+ */
+ @Since("1.6.0")
+ def toAdjacencyList: Array[(Int, Int, Double)] = {
--- End diff --
Maybe this is too much for the first PR. We can remove it and address it in the future.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by freeman-lab <gi...@git.apache.org>.
Github user freeman-lab commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r29121407
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/HierarchicalClusteringModel.scala ---
@@ -0,0 +1,116 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseVector => BDV, Vector => BV, norm => breezeNorm}
+import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.mllib.util.{Loader, Saveable}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.{Logging, SparkContext}
+
+/**
+ * This class is used for the model of the hierarchical clustering
+ *
+ * @param tree a cluster as a tree node
+ */
+class HierarchicalClusteringModel(val tree: ClusterTree)
+ extends Serializable with Saveable with Logging {
+
+ /** Current version of model save/load format. */
+ override protected def formatVersion: String = "1.0"
+
+ override def save(sc: SparkContext, path: String) {
+ val oos = new java.io.ObjectOutputStream(new java.io.FileOutputStream(path))
+ try {
+ oos.writeObject(this)
+ } finally {
+ oos.close()
+ }
+ }
+
+ def getClusters(): Array[ClusterTree] = this.tree.getLeavesNodes()
--- End diff --
Remove parentheses after `getClusters`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151334250
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44387/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152362401
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44646/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r43072422
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,744 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg.{SparseVector => BSV, Vector => BV, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the costs, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The costs means how large the cluster is. That is, the cluster
+ * whose cost is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ import BisectingKMeans._
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[Long, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var updatedDataHistory = Array.empty[RDD[(Long, BV[Double])]]
+ // the minimum number of nodes of a binary tree by given parameter
+ val numNodeLimit = getMinimumNumNodesInTree(this.k)
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = initData(input).cache()
+ while (clusterStats.size < numNodeLimit && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ // TODO Remove non-leaf cluster stats from `leafClusterStats`
+ val leafClusterStats = summarizeClusters(data)
+ val dividableLeafClusters = leafClusterStats.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusterStats
+
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided = divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = updateClusterIndex(data, divided).cache()
+ updatedDataHistory = updatedDataHistory ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (updatedDataHistory.length > 1) {
+ val head = updatedDataHistory.head
+ updatedDataHistory = updatedDataHistory.tail
+ head.unpersist()
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ }
+ // create a map of cluster node with their costs
+ val nodes = createClusterNodes(data, clusterStats)
+ // unpersist RDDs
+ data.unpersist()
+ updatedDataHistory.foreach(_.unpersist())
+
+ // build a cluster tree by Map class which is expressed
+ logInfo(s"Building the cluster tree is started in ${sc.appName}")
+ val root = buildTree(nodes, ROOT_INDEX_KEY, this.k)
+ if (root.isEmpty) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusterStats")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ logInfo(s"Elapsed Time for ${this.getClass.getSimpleName} Training: ${finishTime} [sec]")
+
+ // make a bisecting kmeans model
+ val model = new BisectingKMeansModel(root.get)
+ val leavesNodes = model.getClusters
+ if (leavesNodes.length < this.k) {
+ logWarning(s"# clusters is less than you want: ${leavesNodes.length} / ${k}")
+ }
+ model
+ }
+}
+
+
+private[clustering] object BisectingKMeans {
+
+ import BisectingClusterStat._
+
+ val ROOT_INDEX_KEY: Long = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ def findClosestCenter(metric: (BV[Double], BV[Double]) => Double)
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ // get the closest index
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)._2
+ }
+
+ /**
+ * Gets the minimum number of nodes in a tree by the number of leaves
+ *
+ * @param k: the number of leaf nodes
+ */
+ def getMinimumNumNodesInTree(k: Int): Int = {
+ val multiplier = math.ceil(math.log(k) / math.log(2.0))
+ // the calculation is same as `math.pow(2, multiplier)`
+ var numNodes = 2
+ (1 to multiplier.toInt).foreach (i => numNodes = numNodes << 1)
+ numNodes
+ }
+
+ /**
+ * Summarizes data by each cluster as Map
+ *
+ * @param data pairs of point and its cluster index
+ */
+ def summarizeClusters(data: RDD[(Long, BV[Double])]): Map[Long, BisectingClusterStat] = {
+
+ data.mapPartitions { iter =>
+ // calculate the accumulation of the all point in a partition and count the rows
+ val map = mutable.Map.empty[Long, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx: Long, point: BV[Double]) =>
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(idx, (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size)))
+ map(idx) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ map.toIterator
+ }.reduceByKey { case ((sum1, n1, sumOfSquares1), (sum2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sum1 + sum2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.map { case (i, (sum, n, sumOfSquares)) =>
+ val mean = calcMean(n.toLong, sum)
+ val variance = getVariance(n.toLong, sum, sumOfSquares)
+ (i, new BisectingClusterStat(n.toLong, mean, variance))
+ }.collectAsMap()
+ }
+
+ /**
+ * Assigns the initial cluster index id to all data
+ */
+ def initData(data: RDD[Vector]): RDD[(Long, BV[Double])] = {
+ data.map { v: Vector => (ROOT_INDEX_KEY, v.toBreeze)}
+ }
+
+ /**
+ * Gets the initial centers for bisecting k-means
+ *
+ * @param data pairs of point and its cluster index
+ * @param stats pairs of cluster index and cluster statistics
+ */
+ def initNextCenters(
+ data: RDD[(Long, BV[Double])],
+ stats: Map[Long, BisectingClusterStat]): Map[Long, BV[Double]] = {
+
+ // Since the combination sampleByKey and groupByKey is more expensive,
+ // this as follows would be better.
+ val bcIndeces = data.sparkContext.broadcast(stats.keySet)
+ val samples = data.mapPartitions { iter =>
+ val map = mutable.Map.empty[Long, mutable.ArrayBuffer[BV[Double]]]
+
+ bcIndeces.value.foreach {i => map(i) = mutable.ArrayBuffer.empty[BV[Double]]}
+ val LOCAL_SAMPLE_SIZE = 100
+ iter.foreach { case (i, point) =>
+ map(i).append(point)
+ // to avoid to increase the memory usage on each map thread,
+ // the number of elements is cut off at the right time.
+ if (map(i).size > LOCAL_SAMPLE_SIZE) {
+ val elements = map(i).sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ map(i) = mutable.ArrayBuffer(elements.head, elements.last)
+ }
+ }
+
+ // in order to reduce the shuffle size, take only two elements
+ map.filterNot(_._2.isEmpty).map { case (i, points) =>
+ val elements = map(i).toSeq.sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ i -> mutable.ArrayBuffer(elements.head, elements.last)
+ }.toIterator
+ }.reduceByKey { case (points1, points2) =>
+ points1.union(points2)
+ }.collect()
+
+ val nextCenters = samples.flatMap { case (i, points) =>
+ val elements = points.toSeq.sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ Array((2 * i, elements.head), (2 * i + 1, elements.last))
+ }.toMap
+ if (!stats.keySet.flatMap(idx => Array(2 * idx, 2 * idx + 1)).forall(nextCenters.contains(_))) {
+ throw new SparkException("Failed to initialize centers for next step")
+ }
+ nextCenters
+ }
+
+ /**
+ * Updates the indexes of clusters which is divided to its children indexes
+ *
+ * @param data pairs of point and its cluster index
+ * @param dividedClusters pairs of cluster index and cluster statistics
+ */
+ def updateClusterIndex(
+ data: RDD[(Long, BV[Double])],
+ dividedClusters: Map[Long, BisectingClusterStat]): RDD[(Long, BV[Double])] = {
+
+ // If there is no divided clusters, return the original
+ if (dividedClusters.size == 0) {
+ return data
+ }
+
+ // extract the centers of the clusters
+ val sc = data.sparkContext
+ val centers = dividedClusters.map { case (idx, cluster) => (idx, cluster.mean)}
+ val bcCenters = sc.broadcast(centers)
+
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val bcMetric = sc.broadcast(metric)
+
+ // update the indexes to their children indexes
+ data.map { case (idx, point) =>
+ // TODO improve how to extract child indexes
+ val childIndexes = Array(2 * idx, 2 * idx + 1)
+ val extractedChildIndexes = childIndexes.filter(c => bcCenters.value.contains(c))
+ extractedChildIndexes.length match {
+ // update the indexes
+ case s if s == 2 => {
+ val nextCenters = extractedChildIndexes.map(bcCenters.value(_))
+ val closestIndex = BisectingKMeans
+ .findClosestCenter(bcMetric.value)(nextCenters)(point)
+ val nextIndex = 2 * idx + closestIndex
+ (nextIndex, point)
+ }
+ // stay the index if a cluster which a point belongs wasn't divided
+ case _ => (idx, point)
+ }
+ }
+ }
+
+ /**
+ * Divides clusters according to their statistics
+ *
+ * @param data pairs of point and its cluster index
+ * @param clusterStats target clusters to divide
+ * @param maxIterations the maximum iterations to calculate clusters statistics
+ */
+ def divideClusters(
+ data: RDD[(Long, BV[Double])],
+ clusterStats: Map[Long, BisectingClusterStat],
+ maxIterations: Int): Map[Long, BisectingClusterStat] = {
+ val sc = data.sparkContext
+ val appName = sc.appName
+
+ // get keys of dividable clusters
+ val dividableClusterStats = clusterStats.filter { case (idx, cluster) => cluster.isDividable }
+ if (dividableClusterStats.isEmpty) {
+ return Map.empty[Long, BisectingClusterStat]
+ }
+ // extract dividable input data
+ val dividableData = data.filter { case (idx, point) => dividableClusterStats.contains(idx)}
+
+ var newCenters = initNextCenters(dividableData, dividableClusterStats)
+ var bcNewCenters = sc.broadcast(newCenters)
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val bcMetric = sc.broadcast(metric)
+ // pairs of cluster index and (sums, #points, sumOfSquares)
+ var stats = Map.empty[Long, (BV[Double], Double, BV[Double])]
+
+ var subIter = 0
+ var totalStd = Double.MaxValue
+ var oldTotalStd = Double.MaxValue
+ var relativeError = Double.MaxValue
+ while (subIter < maxIterations && relativeError > 10E-4) {
--- End diff --
`1E-4`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152343011
**[Test build #44642 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44642/consoleFull)** for PR 5267 at commit [`5da05d3`](https://github.com/apache/spark/commit/5da05d3fd41d5b730d19f595ca9bf622aaf0a14d).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r27743987
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/HierarchicalClustering.scala ---
@@ -0,0 +1,610 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseVector => BDV, SparseVector => BSV, Vector => BV, norm => breezeNorm}
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.random.XORShiftRandom
+import org.apache.spark.{Logging, SparkException}
+
+import scala.collection.{Map, mutable}
+
+
+/**
+ * Top-level methods for calling the hierarchical clustering algorithm
+ */
+object HierarchicalClustering extends Logging {
--- End diff --
Alright. Thank you for letting me know.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r43072197
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,744 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg.{SparseVector => BSV, Vector => BV, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the costs, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The costs means how large the cluster is. That is, the cluster
+ * whose cost is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ import BisectingKMeans._
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[Long, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var updatedDataHistory = Array.empty[RDD[(Long, BV[Double])]]
+ // the minimum number of nodes of a binary tree by given parameter
+ val numNodeLimit = getMinimumNumNodesInTree(this.k)
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = initData(input).cache()
+ while (clusterStats.size < numNodeLimit && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ // TODO Remove non-leaf cluster stats from `leafClusterStats`
+ val leafClusterStats = summarizeClusters(data)
+ val dividableLeafClusters = leafClusterStats.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusterStats
+
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided = divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = updateClusterIndex(data, divided).cache()
+ updatedDataHistory = updatedDataHistory ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (updatedDataHistory.length > 1) {
+ val head = updatedDataHistory.head
+ updatedDataHistory = updatedDataHistory.tail
+ head.unpersist()
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ }
+ // create a map of cluster node with their costs
+ val nodes = createClusterNodes(data, clusterStats)
+ // unpersist RDDs
+ data.unpersist()
+ updatedDataHistory.foreach(_.unpersist())
+
+ // build a cluster tree by Map class which is expressed
+ logInfo(s"Building the cluster tree is started in ${sc.appName}")
+ val root = buildTree(nodes, ROOT_INDEX_KEY, this.k)
+ if (root.isEmpty) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusterStats")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ logInfo(s"Elapsed Time for ${this.getClass.getSimpleName} Training: ${finishTime} [sec]")
+
+ // make a bisecting kmeans model
+ val model = new BisectingKMeansModel(root.get)
+ val leavesNodes = model.getClusters
+ if (leavesNodes.length < this.k) {
+ logWarning(s"# clusters is less than you want: ${leavesNodes.length} / ${k}")
+ }
+ model
+ }
+}
+
+
+private[clustering] object BisectingKMeans {
+
+ import BisectingClusterStat._
+
+ val ROOT_INDEX_KEY: Long = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ def findClosestCenter(metric: (BV[Double], BV[Double]) => Double)
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ // get the closest index
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)._2
+ }
+
+ /**
+ * Gets the minimum number of nodes in a tree by the number of leaves
+ *
+ * @param k: the number of leaf nodes
+ */
+ def getMinimumNumNodesInTree(k: Int): Int = {
+ val multiplier = math.ceil(math.log(k) / math.log(2.0))
+ // the calculation is same as `math.pow(2, multiplier)`
+ var numNodes = 2
+ (1 to multiplier.toInt).foreach (i => numNodes = numNodes << 1)
+ numNodes
+ }
+
+ /**
+ * Summarizes data by each cluster as Map
+ *
+ * @param data pairs of point and its cluster index
+ */
+ def summarizeClusters(data: RDD[(Long, BV[Double])]): Map[Long, BisectingClusterStat] = {
+
+ data.mapPartitions { iter =>
--- End diff --
Replace `.mapPartitions { ... }.reduceByKey {...}` by `aggregateByKey`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151311507
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151313120
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r43072937
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,691 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg.{SparseVector => BSV, Vector => BV, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the costs, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The costs means how large the cluster is. That is, the cluster
+ * whose cost is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ import BisectingKMeans._
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(2, 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[Long, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var updatedDataHistory = Array.empty[RDD[(Long, BV[Double])]]
+ // the minimum number of nodes of a binary tree by given parameter
+ val numNodeLimit = getMinimumNumNodesInTree(this.k)
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = initData(input).cache()
+ while (clusterStats.size < numNodeLimit && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ // TODO Remove non-leaf cluster stats from `leafClusterStats`
+ val leafClusterStats = summarizeClusters(data)
+ val dividableLeafClusters = leafClusterStats.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusterStats
+
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided = divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = updateClusterIndex(data, divided).cache()
+ updatedDataHistory = updatedDataHistory ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (updatedDataHistory.length > 1) {
+ val head = updatedDataHistory.head
+ updatedDataHistory = updatedDataHistory.tail
+ head.unpersist()
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ }
+ // create a map of cluster node with their costs
+ val nodes = createClusterNodes(data, clusterStats)
+ // unpersist RDDs
+ data.unpersist()
+ updatedDataHistory.foreach(_.unpersist())
+
+ // build a cluster tree by Map class which is expressed
+ logInfo(s"Building the cluster tree is started in ${sc.appName}")
+ val root = buildTree(nodes, ROOT_INDEX_KEY, this.k)
+ if (root.isEmpty) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusterStats")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ logInfo(s"Elapsed Time for ${this.getClass.getSimpleName} Training: ${finishTime} [sec]")
+
+ // make a bisecting kmeans model
+ val model = new BisectingKMeansModel(root.get)
+ val leavesNodes = model.getClusters
+ if (leavesNodes.length < this.k) {
+ logWarning(s"# clusters is less than you want: ${leavesNodes.length} / ${k}")
+ }
+ model
+ }
+}
+
+
+private[clustering] object BisectingKMeans {
+
+ import BisectingClusterStat._
+
+ val ROOT_INDEX_KEY: Long = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ def findClosestCenter(metric: (BV[Double], BV[Double]) => Double)
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ // get the closest index
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)._2
+ }
+
+ /**
+ * Gets the minimum number of nodes in a tree by the number of leaves
+ *
+ * @param k: the number of leaf nodes
+ */
+ def getMinimumNumNodesInTree(k: Int): Int = {
+ val multiplier = math.ceil(math.log(k) / math.log(2.0))
+ // the calculation is same as `math.pow(2, multiplier)`
+ var numNodes = 2
+ (1 to multiplier.toInt).foreach (i => numNodes = numNodes << 1)
+ numNodes
+ }
+
+ /**
+ * Summarizes data by each cluster as Map
+ *
+ * @param data pairs of point and its cluster index
+ */
+ def summarizeClusters(data: RDD[(Long, BV[Double])]): Map[Long, BisectingClusterStat] = {
+
+ data.mapPartitions { iter =>
+ // calculate the accumulation of the all point in a partition and count the rows
+ val map = mutable.Map.empty[Long, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx: Long, point: BV[Double]) =>
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(idx, (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size)))
+ map(idx) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ map.toIterator
+ }.reduceByKey { case ((sum1, n1, sumOfSquares1), (sum2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sum1 + sum2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.map { case (i, (sum, n, sumOfSquares)) =>
+ val mean = calcMean(n.toLong, sum)
+ val variance = getVariance(n.toLong, sum, sumOfSquares)
+ (i, new BisectingClusterStat(n.toLong, mean, variance))
+ }.collectAsMap()
+ }
+
+ /**
+ * Assigns the initial cluster index id to all data
+ */
+ def initData(data: RDD[Vector]): RDD[(Long, BV[Double])] = {
+ data.map { v: Vector => (ROOT_INDEX_KEY, v.toBreeze)}
+ }
+
+ /**
+ * Gets the initial centers for bisecting k-means
+ *
+ * @param data pairs of point and its cluster index
+ * @param stats pairs of cluster index and cluster statistics
+ */
+ def initNextCenters(
+ data: RDD[(Long, BV[Double])],
+ stats: Map[Long, BisectingClusterStat]): Map[Long, BV[Double]] = {
+
+ // Since the combination sampleByKey and groupByKey is more expensive,
+ // this as follows would be better.
+ val bcIndeces = data.sparkContext.broadcast(stats.keySet)
+ val samples = data.mapPartitions { iter =>
+ val map = mutable.Map.empty[Long, mutable.ArrayBuffer[BV[Double]]]
+
+ bcIndeces.value.foreach {i => map(i) = mutable.ArrayBuffer.empty[BV[Double]]}
+ val LOCAL_SAMPLE_SIZE = 100
+ iter.foreach { case (i, point) =>
+ map(i).append(point)
+ // to avoid to increase the memory usage on each map thread,
+ // the number of elements is cut off at the right time.
+ if (map(i).size > LOCAL_SAMPLE_SIZE) {
+ val elements = map(i).sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ map(i) = mutable.ArrayBuffer(elements.head, elements.last)
+ }
+ }
+
+ // in order to reduce the shuffle size, take only two elements
+ map.filterNot(_._2.isEmpty).map { case (i, points) =>
+ val elements = map(i).toSeq.sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ i -> mutable.ArrayBuffer(elements.head, elements.last)
+ }.toIterator
+ }.reduceByKey { case (points1, points2) =>
+ points1.union(points2)
+ }.collect()
+
+ val nextCenters = samples.flatMap { case (i, points) =>
+ val elements = points.toSeq.sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ Array((2 * i, elements.head), (2 * i + 1, elements.last))
+ }.toMap
+ if (!stats.keySet.flatMap(idx => Array(2 * idx, 2 * idx + 1)).forall(nextCenters.contains(_))) {
+ throw new SparkException("Failed to initialize centers for next step")
+ }
+ nextCenters
+ }
+
+ /**
+ * Updates the indexes of clusters which is divided to its children indexes
+ *
+ * @param data pairs of point and its cluster index
+ * @param dividedClusters pairs of cluster index and cluster statistics
+ */
+ def updateClusterIndex(
+ data: RDD[(Long, BV[Double])],
+ dividedClusters: Map[Long, BisectingClusterStat]): RDD[(Long, BV[Double])] = {
+
+ // If there is no divided clusters, return the original
+ if (dividedClusters.size == 0) {
+ return data
+ }
+
+ // extract the centers of the clusters
+ val sc = data.sparkContext
+ val centers = dividedClusters.map { case (idx, cluster) => (idx, cluster.mean)}
+ val bcCenters = sc.broadcast(centers)
+
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val bcMetric = sc.broadcast(metric)
+
+ // update the indexes to their children indexes
+ data.map { case (idx, point) =>
+ // TODO improve how to extract child indexes
+ val childIndexes = Array(2 * idx, 2 * idx + 1)
+ val extractedChildIndexes = childIndexes.filter(c => bcCenters.value.contains(c))
+ extractedChildIndexes.length match {
+ // update the indexes
+ case s if s == 2 => {
+ val nextCenters = extractedChildIndexes.map(bcCenters.value(_))
+ val closestIndex = BisectingKMeans
+ .findClosestCenter(bcMetric.value)(nextCenters)(point)
+ val nextIndex = 2 * idx + closestIndex
+ (nextIndex, point)
+ }
+ // stay the index if a cluster which a point belongs wasn't divided
+ case _ => (idx, point)
+ }
+ }
+ }
+
+ /**
+ * Divides clusters according to their statistics
+ *
+ * @param data pairs of point and its cluster index
+ * @param clusterStats target clusters to divide
+ * @param maxIterations the maximum iterations to calculate clusters statistics
+ */
+ def divideClusters(
+ data: RDD[(Long, BV[Double])],
+ clusterStats: Map[Long, BisectingClusterStat],
+ maxIterations: Int): Map[Long, BisectingClusterStat] = {
+ val sc = data.sparkContext
+ val appName = sc.appName
+
+ // get keys of dividable clusters
+ val dividableClusterStats = clusterStats.filter { case (idx, cluster) => cluster.isDividable }
+ if (dividableClusterStats.isEmpty) {
+ return Map.empty[Long, BisectingClusterStat]
+ }
+ // extract dividable input data
+ val dividableData = data.filter { case (idx, point) => dividableClusterStats.contains(idx)}
+
+ var newCenters = initNextCenters(dividableData, dividableClusterStats)
+ var bcNewCenters = sc.broadcast(newCenters)
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val bcMetric = sc.broadcast(metric)
+ // pairs of cluster index and (sums, #points, sumOfSquares)
+ var stats = Map.empty[Long, (BV[Double], Double, BV[Double])]
+
+ var subIter = 0
+ var totalStd = Double.MaxValue
+ var oldTotalStd = Double.MaxValue
+ var relativeError = Double.MaxValue
+ while (subIter < maxIterations && relativeError > 10e-4) {
--- End diff --
`1e-4`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152350977
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44642/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-87726920
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29403/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42821012
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = BisectingKMeans.initData(input).cache()
+ while (clusterStats.size < maxAllNodesInTree && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ val leafClusters = BisectingKMeans.summarizeClusters(data)
+ val dividableLeafClusters = leafClusters.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusters
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ else {
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided =
+ BisectingKMeans.divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = BisectingKMeans.updateClusterIndex(data, divided).cache()
+ rddArray = rddArray ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (rddArray.length > 1) {
+ val head = rddArray.head
+ head.unpersist()
+ rddArray = rddArray.filterNot(_.hashCode() == head.hashCode())
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+ }
+ }
+ // unpersist kept RDDs
+ rddArray.foreach(_.unpersist())
--- End diff --
make sure it doesn't unpersist the input RDD
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152275050
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44615/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r27632107
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/HierarchicalClustering.scala ---
@@ -0,0 +1,610 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseVector => BDV, SparseVector => BSV, Vector => BV, norm => breezeNorm}
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.random.XORShiftRandom
+import org.apache.spark.{Logging, SparkException}
+
+import scala.collection.{Map, mutable}
+
+
+/**
+ * Top-level methods for calling the hierarchical clustering algorithm
+ */
+object HierarchicalClustering extends Logging {
--- End diff --
Should we remove static train() methods of others? For example, KMeans has static train() method. I'd like to understand the difference. I think we should keep the design concept consistent for users. It is not good that some algorithms support static train() method and others don't support it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-90884844
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29855/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155156703
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152362256
**[Test build #44646 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44646/consoleFull)** for PR 5267 at commit [`a50689a`](https://github.com/apache/spark/commit/a50689a144bf5801c831d0b2f33eb6435e87f929).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class BisectingKMeansModel @Since(\"1.6.0\") (`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by freeman-lab <gi...@git.apache.org>.
Github user freeman-lab commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r29120850
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/HierarchicalClustering.scala ---
@@ -0,0 +1,574 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseVector => BDV, SparseVector => BSV, Vector => BV, norm => breezeNorm}
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.random.XORShiftRandom
+import org.apache.spark.{Logging, SparkException}
+
+import scala.collection.{Map, mutable}
+
+
+object HierarchicalClustering extends Logging {
+
+ private[clustering] val ROOT_INDEX_KEY: Long = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ private[mllib]
+ def findClosestCenter(metric: Function2[BV[Double], BV[Double], Double])
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)
+ closestIndex
+ }
+}
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bi-sect k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * @param numClusters tne number of clusters you want
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations
+ * @param maxRetries the number of maximum retries
+ * @param seed a random seed
+ */
+class HierarchicalClustering private (
+ private var numClusters: Int,
+ private var clusterMap: Map[Long, ClusterTree],
+ private var maxIterations: Int,
+ private var maxRetries: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ def this() = this(20, mutable.ListMap.empty[Long, ClusterTree], 20, 10, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ def setNumClusters(numClusters: Int): this.type = {
+ this.numClusters = numClusters
+ this
+ }
+
+ def getNumClusters: Int = this.numClusters
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ def getSubIterations: Int = this.maxIterations
--- End diff --
Why the name swap? Shouldn't this be `getMaxIterations`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by freeman-lab <gi...@git.apache.org>.
Github user freeman-lab commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155231687
awesome, nice job all!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152259858
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152274585
**[Test build #44615 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44615/consoleFull)** for PR 5267 at commit [`a876ba2`](https://github.com/apache/spark/commit/a876ba233e197bb5228a05fe2946523af1b40ee6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class BisectingKMeansModel @Since(\"1.6.0\") (`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-150270427
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42820997
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
--- End diff --
* this is the minimum depth of the tree
* `math.log10` -> `math.log`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r43072115
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,744 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg.{SparseVector => BSV, Vector => BV, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the costs, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The costs means how large the cluster is. That is, the cluster
+ * whose cost is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ import BisectingKMeans._
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[Long, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var updatedDataHistory = Array.empty[RDD[(Long, BV[Double])]]
+ // the minimum number of nodes of a binary tree by given parameter
+ val numNodeLimit = getMinimumNumNodesInTree(this.k)
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = initData(input).cache()
+ while (clusterStats.size < numNodeLimit && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ // TODO Remove non-leaf cluster stats from `leafClusterStats`
+ val leafClusterStats = summarizeClusters(data)
+ val dividableLeafClusters = leafClusterStats.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusterStats
+
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided = divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = updateClusterIndex(data, divided).cache()
+ updatedDataHistory = updatedDataHistory ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (updatedDataHistory.length > 1) {
+ val head = updatedDataHistory.head
+ updatedDataHistory = updatedDataHistory.tail
+ head.unpersist()
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ }
+ // create a map of cluster node with their costs
+ val nodes = createClusterNodes(data, clusterStats)
+ // unpersist RDDs
+ data.unpersist()
+ updatedDataHistory.foreach(_.unpersist())
+
+ // build a cluster tree by Map class which is expressed
+ logInfo(s"Building the cluster tree is started in ${sc.appName}")
+ val root = buildTree(nodes, ROOT_INDEX_KEY, this.k)
+ if (root.isEmpty) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusterStats")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ logInfo(s"Elapsed Time for ${this.getClass.getSimpleName} Training: ${finishTime} [sec]")
+
+ // make a bisecting kmeans model
+ val model = new BisectingKMeansModel(root.get)
+ val leavesNodes = model.getClusters
+ if (leavesNodes.length < this.k) {
+ logWarning(s"# clusters is less than you want: ${leavesNodes.length} / ${k}")
+ }
+ model
+ }
+}
+
+
+private[clustering] object BisectingKMeans {
+
+ import BisectingClusterStat._
+
+ val ROOT_INDEX_KEY: Long = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ def findClosestCenter(metric: (BV[Double], BV[Double]) => Double)
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ // get the closest index
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)._2
+ }
+
+ /**
+ * Gets the minimum number of nodes in a tree by the number of leaves
+ *
+ * @param k: the number of leaf nodes
+ */
+ def getMinimumNumNodesInTree(k: Int): Int = {
+ val multiplier = math.ceil(math.log(k) / math.log(2.0))
+ // the calculation is same as `math.pow(2, multiplier)`
+ var numNodes = 2
--- End diff --
`1 << multiplier`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152356488
**[Test build #44646 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44646/consoleFull)** for PR 5267 at commit [`a50689a`](https://github.com/apache/spark/commit/a50689a144bf5801c831d0b2f33eb6435e87f929).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by freeman-lab <gi...@git.apache.org>.
Github user freeman-lab commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r29121200
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/HierarchicalClustering.scala ---
@@ -0,0 +1,574 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseVector => BDV, SparseVector => BSV, Vector => BV, norm => breezeNorm}
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.random.XORShiftRandom
+import org.apache.spark.{Logging, SparkException}
+
+import scala.collection.{Map, mutable}
+
+
+object HierarchicalClustering extends Logging {
+
+ private[clustering] val ROOT_INDEX_KEY: Long = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ private[mllib]
+ def findClosestCenter(metric: Function2[BV[Double], BV[Double], Double])
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)
+ closestIndex
+ }
+}
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bi-sect k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * @param numClusters tne number of clusters you want
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations
+ * @param maxRetries the number of maximum retries
+ * @param seed a random seed
+ */
+class HierarchicalClustering private (
+ private var numClusters: Int,
+ private var clusterMap: Map[Long, ClusterTree],
+ private var maxIterations: Int,
+ private var maxRetries: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ def this() = this(20, mutable.ListMap.empty[Long, ClusterTree], 20, 10, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ def setNumClusters(numClusters: Int): this.type = {
+ this.numClusters = numClusters
+ this
+ }
+
+ def getNumClusters: Int = this.numClusters
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ def getSubIterations: Int = this.maxIterations
+
+ /**
+ * Sets the number of maximum retries of each clustering step
+ */
+ def setMaxRetries(maxRetries: Int): this.type = {
+ this.maxRetries = maxRetries
+ this
+ }
+
+ def getMaxRetries: Int = this.maxRetries
+
+ /**
+ * Sets the random seed
+ */
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the hierarchical clustering algorithm
+ * @param input RDD of vectors
+ * @return model for the hierarchical clustering
+ */
+ def run(input: RDD[Vector]): HierarchicalClusteringModel = {
+ val sc = input.sparkContext
+ log.info(s"${sc.appName} starts a hierarchical clustering algorithm")
+
+ var data = initData(input).cache()
--- End diff --
This algorithm contains a lot of `cacheing` and `unpersisting`. Can we add a more detailed note in the docstrings as to how much of a data set will be cached? In other words, what is the expected memory footprint of this algorithm relative to the size of the data set? This will be useful for users in considering how much RAM to have available for this algorithm.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155239249
@yu-iskw Thanks for persevering and getting this merged!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r27613304
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/HierarchicalClustering.scala ---
@@ -0,0 +1,610 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseVector => BDV, SparseVector => BSV, Vector => BV, norm => breezeNorm}
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.random.XORShiftRandom
+import org.apache.spark.{Logging, SparkException}
+
+import scala.collection.{Map, mutable}
+
+
+/**
+ * Top-level methods for calling the hierarchical clustering algorithm
+ */
+object HierarchicalClustering extends Logging {
+
+ private[clustering] val ROOT_INDEX_KEY: Long = 1
+
+ /**
+ * Trains a hierarchical clustering model with the given data
+ *
+ * @param data trained data
+ * @param numClusters the maximum number of clusters you want
+ * @return a hierarchical clustering model
+ */
+ def train(data: RDD[Vector], numClusters: Int): HierarchicalClusteringModel = {
+ val algo = new HierarchicalClustering().setNumClusters(numClusters)
+ algo.run(data)
+ }
+
+ /**
+ * Trains a hierarchical clustering model with the given data
+ *
+ * @param data training data
+ * @param numClusters the maximum number of clusters you want
+ * @param maxIterations the number of maximal iterations
+ * @param maxRetries the number of maximum retries when the clustering can't be succeeded
+ * @param seed the randomseed to generate the initial vectors for each bisecting
+ * @return a hierarchical clustering model
+ */
+ def train(data: RDD[Vector],
+ numClusters: Int,
+ maxIterations: Int,
+ maxRetries: Int,
+ seed: Int): HierarchicalClusteringModel = {
+
+ val algo = new HierarchicalClustering().setNumClusters(numClusters)
+ .setMaxIterations(maxIterations)
+ .setMaxRetries(maxRetries)
+ .setSeed(seed)
+ algo.run(data)
+ }
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ private[mllib]
+ def findClosestCenter(metric: Function2[BV[Double], BV[Double], Double])
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)
+ closestIndex
+ }
+}
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bi-sect k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * @param numClusters tne number of clusters you want
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations
+ * @param maxRetries the number of maximum retries
+ * @param seed a random seed
+ */
+class HierarchicalClustering(
+ private var numClusters: Int,
+ private var clusterMap: Map[Long, ClusterTree],
+ private var maxIterations: Int,
+ private var maxRetries: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ def this() = this(20, mutable.ListMap.empty[Long, ClusterTree], 20, 10, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ def setNumClusters(numClusters: Int): this.type = {
+ this.numClusters = numClusters
+ this
+ }
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ def getSubIterations(): Int = this.maxIterations
--- End diff --
No parentheses for getters
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by freeman-lab <gi...@git.apache.org>.
Github user freeman-lab commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-88386004
@yu-iskw great putting this new version together, I'd be happy to do a review (especially re: the algorithm), should be able to get to it in the next few days!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42821021
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = BisectingKMeans.initData(input).cache()
+ while (clusterStats.size < maxAllNodesInTree && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ val leafClusters = BisectingKMeans.summarizeClusters(data)
+ val dividableLeafClusters = leafClusters.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusters
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ else {
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided =
+ BisectingKMeans.divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = BisectingKMeans.updateClusterIndex(data, divided).cache()
+ rddArray = rddArray ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (rddArray.length > 1) {
+ val head = rddArray.head
+ head.unpersist()
+ rddArray = rddArray.filterNot(_.hashCode() == head.hashCode())
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+ }
+ }
+ // unpersist kept RDDs
+ rddArray.foreach(_.unpersist())
+ // create a map of cluster node with their criterions
+ val nodes = BisectingKMeans.createClusterNodes(data, clusterStats)
+
+ // build a cluster tree by Map class which is expressed
+ logInfo(s"Building the cluster tree is started in ${sc.appName}")
+ val root = BisectingKMeans.buildTree(nodes, BisectingKMeans.ROOT_INDEX_KEY, this.k)
+ if (root.isEmpty) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusterStats")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ logInfo(s"Elapsed Time for ${this.getClass.getSimpleName} Training: ${finishTime} [sec]")
+
+ // make a bisecting kmeans model
+ val model = new BisectingKMeansModel(root.get)
+ val leavesNodes = model.getClusters
+ if (leavesNodes.length < this.k) {
+ logWarning(s"# clusters is less than you want: ${leavesNodes.length} / ${k}")
+ }
+ model
+ }
+}
+
+
+private[clustering] object BisectingKMeans {
+
+ val ROOT_INDEX_KEY: BigInt = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ def findClosestCenter(metric: (BV[Double], BV[Double]) => Double)
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)
+ closestIndex
+ }
+
+ /**
+ * Summarizes data by each cluster as Map
+ *
+ * @param data pairs of point and its cluster index
+ */
+ def summarizeClusters(data: RDD[(BigInt, BV[Double])]): Map[BigInt, BisectingClusterStat] = {
+
+ val stats = data.mapPartitions { iter =>
+ // calculate the accumulation of the all point in a partition and count the rows
+ val map = mutable.Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx: BigInt, point: BV[Double]) =>
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(idx, (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size)))
+ map(idx) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ map.toIterator
+ }.reduceByKey { case ((sum1, n1, sumOfSquares1), (sum2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sum1 + sum2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.collect().toMap
+
+ stats.map {case (i, stat) => i -> new BisectingClusterStat(stat._2.toLong, stat._1, stat._3)}
--- End diff --
`{ case (i, (sum, n, sumOfSquares)) => ... }.toMap`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155174143
**[Test build #45396 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45396/consoleFull)** for PR 5267 at commit [`75ca2a0`](https://github.com/apache/spark/commit/75ca2a000936fe62fb53293d805b2888c67cb563).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * ` * Sets the random seed (default: hash value of the class name).`\n * `class BisectingKMeansModel @Since(\"1.6.0\") (`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-97625133
@freeman-lab thank you for reviewing. I will modify it soon.
The reason removing `toMergeList` at the old version is that I think it depends on scipy much. However, as you're suggesting, we should support any function to get a dendrogram. How is adjacency list?
Or if there is any general data structure for a linkage matrix, we should support it. Do you have any idea?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42821750
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = BisectingKMeans.initData(input).cache()
+ while (clusterStats.size < maxAllNodesInTree && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ val leafClusters = BisectingKMeans.summarizeClusters(data)
+ val dividableLeafClusters = leafClusters.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusters
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ else {
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided =
+ BisectingKMeans.divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = BisectingKMeans.updateClusterIndex(data, divided).cache()
+ rddArray = rddArray ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (rddArray.length > 1) {
+ val head = rddArray.head
+ head.unpersist()
+ rddArray = rddArray.filterNot(_.hashCode() == head.hashCode())
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+ }
+ }
+ // unpersist kept RDDs
+ rddArray.foreach(_.unpersist())
+ // create a map of cluster node with their criterions
+ val nodes = BisectingKMeans.createClusterNodes(data, clusterStats)
+
+ // build a cluster tree by Map class which is expressed
+ logInfo(s"Building the cluster tree is started in ${sc.appName}")
+ val root = BisectingKMeans.buildTree(nodes, BisectingKMeans.ROOT_INDEX_KEY, this.k)
+ if (root.isEmpty) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusterStats")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ logInfo(s"Elapsed Time for ${this.getClass.getSimpleName} Training: ${finishTime} [sec]")
+
+ // make a bisecting kmeans model
+ val model = new BisectingKMeansModel(root.get)
+ val leavesNodes = model.getClusters
+ if (leavesNodes.length < this.k) {
+ logWarning(s"# clusters is less than you want: ${leavesNodes.length} / ${k}")
+ }
+ model
+ }
+}
+
+
+private[clustering] object BisectingKMeans {
+
+ val ROOT_INDEX_KEY: BigInt = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ def findClosestCenter(metric: (BV[Double], BV[Double]) => Double)
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)
+ closestIndex
+ }
+
+ /**
+ * Summarizes data by each cluster as Map
+ *
+ * @param data pairs of point and its cluster index
+ */
+ def summarizeClusters(data: RDD[(BigInt, BV[Double])]): Map[BigInt, BisectingClusterStat] = {
+
+ val stats = data.mapPartitions { iter =>
+ // calculate the accumulation of the all point in a partition and count the rows
+ val map = mutable.Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx: BigInt, point: BV[Double]) =>
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(idx, (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size)))
+ map(idx) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ map.toIterator
+ }.reduceByKey { case ((sum1, n1, sumOfSquares1), (sum2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sum1 + sum2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.collect().toMap
+
+ stats.map {case (i, stat) => i -> new BisectingClusterStat(stat._2.toLong, stat._1, stat._3)}
+ }
+
+ /**
+ * Assigns the initial cluster index id to all data
+ */
+ def initData(data: RDD[Vector]): RDD[(BigInt, BV[Double])] = {
+ data.map { v: Vector => (BisectingKMeans.ROOT_INDEX_KEY, v.toBreeze)}
+ }
+
+ /**
+ * Gets the initial centers for bisect k-means
+ *
+ * @param data pairs of point and its cluster index
+ * @param stats pairs of cluster index and cluster statistics
+ */
+ def initNextCenters(
+ data: RDD[(BigInt, BV[Double])],
+ stats: Map[BigInt, BisectingClusterStat]): Map[BigInt, BV[Double]] = {
+
+ // Since the combination sampleByKey and groupByKey is more expensive,
+ // this as follows would be better.
+ val bcIndeces = data.sparkContext.broadcast(stats.keySet)
+ val samples = data.mapPartitions { iter =>
+ val map = mutable.Map.empty[BigInt, mutable.ArrayBuffer[BV[Double]]]
+
+ bcIndeces.value.foreach {i => map(i) = mutable.ArrayBuffer.empty[BV[Double]]}
+ val LOCAL_SAMPLE_SIZE = 100
+ iter.foreach { case (i, point) =>
+ map(i).append(point)
+ // to avoid to increase the memory usage on each map thread,
+ // the number of elements is cut off at the right time.
+ if (map(i).size > LOCAL_SAMPLE_SIZE) {
+ val elements = map(i).sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ map(i) = mutable.ArrayBuffer(elements.head, elements.last)
+ }
+ }
+
+ // in order to reduce the shuffle size, take only two elements
+ map.filterNot(_._2.isEmpty).map { case (i, points) =>
+ val elements = map(i).toSeq.sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ i -> mutable.ArrayBuffer(elements.head, elements.last)
+ }.toIterator
+ }.reduceByKey { case (points1, points2) =>
+ points1.union(points2)
+ }.collect()
+
+ val nextCenters = samples.flatMap { case (i, points) =>
+ val elements = points.toSeq.sortWith((a, b) => breezeNorm(a, 2.0) < breezeNorm(b, 2.0))
+ Array((2 * i, elements.head), (2 * i + 1, elements.last))
+ }.toMap
+ if (!stats.keySet.flatMap(idx => Array(2 * idx, 2 * idx + 1)).forall(nextCenters.contains(_))) {
+ throw new SparkException("Failed to initialize centers for next step")
+ }
+ nextCenters
+ }
+
+ /**
+ * Updates the indexes of clusters which is divided to its children indexes
+ *
+ * @param data pairs of point and its cluster index
+ * @param dividedClusters pairs of cluster index and cluster statistics
+ */
+ def updateClusterIndex(
+ data: RDD[(BigInt, BV[Double])],
+ dividedClusters: Map[BigInt, BisectingClusterStat]): RDD[(BigInt, BV[Double])] = {
+
+ // extract the centers of the clusters
+ val sc = data.sparkContext
+ var centers = dividedClusters.map { case (idx, cluster) => (idx, cluster.center)}
+ val bcCenters = sc.broadcast(centers)
+
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val bcMetric = sc.broadcast(metric)
+
+ // update the indexes to their children indexes
+ data.map { case (idx, point) =>
+ val childrenIndexes = Array(2 * idx, 2 * idx + 1).filter(c => bcCenters.value.contains(c))
+ childrenIndexes.length match {
+ // stay the index if the number of children is not enough
+ case s if s < 2 => (idx, point)
+ // update the indexes
+ case _ => {
+ val nextCenters = childrenIndexes.map(bcCenters.value(_))
+ val closestIndex = BisectingKMeans
+ .findClosestCenter(bcMetric.value)(nextCenters)(point)
+ val nextIndex = 2 * idx + closestIndex
+ (nextIndex, point)
+ }
+ }
+ }
+ }
+
+ /**
+ * Divides clusters according to their statistics
+ *
+ * @param data pairs of point and its cluster index
+ * @param targetClusters target clusters to divide
+ * @param maxIterations the maximum iterations to calculate clusters statistics
+ */
+ def divideClusters(
+ data: RDD[(BigInt, BV[Double])],
+ targetClusters: Map[BigInt, BisectingClusterStat],
+ maxIterations: Int): Map[BigInt, BisectingClusterStat] = {
+ val sc = data.sparkContext
+ val appName = sc.appName
+
+ // get keys of dividable clusters
+ val dividableClusters = targetClusters.filter { case (idx, cluster) => cluster.isDividable }
+ if (dividableClusters.isEmpty) {
+ return Map.empty[BigInt, BisectingClusterStat]
+ }
+ // extract dividable input data
+ val dividableData = data.filter { case (idx, point) => dividableClusters.contains(idx)}
+
+ var newCenters = BisectingKMeans.initNextCenters(dividableData, dividableClusters)
+ var bcNewCenters = sc.broadcast(newCenters)
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val bcMetric = sc.broadcast(metric)
+ var stats = Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+
+ var subIter = 0
+ var totalStd = Double.MaxValue
+ var oldTotalStd = Double.MaxValue
+ var relativeError = Double.MaxValue
+ while (subIter < maxIterations && relativeError > 10E-4) {
+ // calculate summary of each cluster
+ val eachStats = dividableData.mapPartitions { iter =>
+ val map = mutable.Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx, point) =>
+ // calculate next index number
+ val childrenCenters = Array(2 * idx, 2 * idx + 1)
+ .filter(x => bcNewCenters.value.contains(x)).map(bcNewCenters.value(_))
+ if (childrenCenters.length == 2) {
+ val closestIndex =
+ BisectingKMeans.findClosestCenter(bcMetric.value)(childrenCenters)(point)
+ val nextIndex = 2 * idx + closestIndex
+
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(
+ nextIndex,
+ (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size))
+ )
+ map(nextIndex) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ }
+ map.toIterator
+ }.reduceByKey { case ((sv1, n1, sumOfSquares1), (sv2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sv1 + sv2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.collect().toMap
+
+ // calculate the center of each cluster
+ newCenters = eachStats.map { case (idx, (sum, n, sumOfSquares)) => (idx, sum :/ n)}
+ bcNewCenters = sc.broadcast(newCenters)
+
+ // update summary of each cluster
+ stats = eachStats.toMap
+
+ totalStd = stats.map { case (idx, (sum, n, sumOfSquares)) =>
+ breezeSum((sumOfSquares :/ n) :- breezeNorm(sum :/ n, 2.0))
+ }.sum
+ relativeError = math.abs(oldTotalStd - totalStd) / totalStd
+ oldTotalStd = totalStd
+ subIter += 1
+ }
+ stats.map { case (i, stat) => i -> new BisectingClusterStat(stat._2.toLong, stat._1, stat._3) }
+ }
+
+ /**
+ * Creates the map of cluster stats to the map of cluster nodes with their criterions
+ *
+ * @param data input data
+ * @param stats map of cluster stats which is described as a binary tree
+ */
+ def createClusterNodes(
+ data: RDD[(BigInt, BV[Double])],
+ stats: Map[BigInt, BisectingClusterStat]): Map[BigInt, BisectingClusterNode] = {
+
+ // TODO: support other criterion, such as entropy
+ createClusterNodesWithAverageCost(data, stats)
+ }
+
+ /**
+ * Creates the map of cluster stats to the map of cluster nodes with their average costs
+ */
+ private def createClusterNodesWithAverageCost(
+ data: RDD[(BigInt, BV[Double])],
+ stats: Map[BigInt, BisectingClusterStat]): Map[BigInt, BisectingClusterNode] = {
+
+ // calculate average costs of all clusters
+ val bcCenters = data.sparkContext.broadcast(stats.map { case (i, stat) => i -> stat.center })
+ val costs = data.mapPartitions { iter =>
+ val counters = mutable.Map.empty[BigInt, (Long, Double)]
+ bcCenters.value.foreach {case (i, center) => counters(i) = (0L, 0.0)}
+ iter.foreach { case (i, point) =>
+ val cost = breezeNorm(bcCenters.value.apply(i) - point, 2.0)
+ counters(i) = (counters(i)._1 + 1, counters(i)._2 + cost)
+ }
+ counters.toIterator
+ }.reduceByKey { case((n1, cost1), (n2, cost2)) =>
+ (n1 + n2, cost1 + cost2)
+ }.collectAsMap()
+
+ stats.map { case (i, stat) =>
+ val avgCost = costs(i)._1 match {
+ case x if x == 0.0 => 0.0
+ case _ => costs(i)._2 / costs(i)._1
+ }
+ i -> new BisectingClusterNode(Vectors.fromBreeze(stat.center), stat.rows, avgCost)
+ }
+ }
+
+ /**
+ * Builds a cluster tree from a Map of clusters
+ *
+ * @param treeMap divided clusters as a Map class
+ * @param rootIndex index you want to start
+ * @param numClusters the number of clusters you want
+ * @return a built cluster tree
+ */
+ private def buildTree(
+ treeMap: Map[BigInt, BisectingClusterNode],
+ rootIndex: BigInt,
+ numClusters: Int): Option[BisectingClusterNode] = {
+
+ // if there is no index in the Map
+ if (!treeMap.contains(rootIndex)) return None
+
+ // build a cluster tree if the queue is empty or until the number of leaf clusters is enough
+ var numLeavesClusters = 1
+ val root = treeMap(rootIndex)
+ var leavesQueue = Map(rootIndex -> root)
+ while (leavesQueue.nonEmpty && numLeavesClusters < numClusters) {
+ // pick up the largest cluster by the maximum criterion of all the clusters
+ val mostScattered = leavesQueue.maxBy(_._2.criterion)
+ val mostScatteredKey = mostScattered._1
+ val mostScatteredCluster = mostScattered._2
+
+ // relate the most scattered cluster to its children clusters
+ val childrenIndexes = Array(2 * mostScatteredKey, 2 * mostScatteredKey + 1)
+ if (childrenIndexes.forall(i => treeMap.contains(i))) {
+ // insert children to the most scattered cluster
+ val children = childrenIndexes.map(i => treeMap(i))
+ mostScatteredCluster.insert(children)
+
+ // calculate the local dendrogram height
+ // TODO Supports distance metrics other Euclidean distance metric
+ val metric = (bv1: BV[Double], bv2: BV[Double]) => breezeNorm(bv1 - bv2, 2.0)
+ val localHeight = children
+ .map(child => metric(child.center.toBreeze, mostScatteredCluster.center.toBreeze)).max
+ mostScatteredCluster.setLocalHeight(localHeight)
+
+ // update the queue
+ leavesQueue = leavesQueue ++ childrenIndexes.map(i => i -> treeMap(i)).toMap
+ numLeavesClusters += 1
+ }
+
+ // remove the cluster which is involved to the cluster tree
+ leavesQueue = leavesQueue.filterNot(_ == mostScattered)
+ }
+ Some(root)
+ }
+}
+
+/**
+ * A cluster as a tree node which can have its sub nodes
+ *
+ * @param center the center of the cluster
+ * @param rows the number of rows in the cluster
+ * @param criterion how large a cluster is
+ * @param localHeight the maximal distance between this node and its children
+ * @param parent the parent cluster of the cluster
+ * @param children the children nodes of the cluster
+ */
+@Since("1.6.0")
+class BisectingClusterNode private (
+ @Since("1.6.0") val center: Vector,
+ @Since("1.6.0") val rows: Long,
+ @Since("1.6.0") val criterion: Double,
+ private var localHeight: Double,
+ private var parent: Option[BisectingClusterNode],
+ private var children: Seq[BisectingClusterNode]) extends Serializable {
+
+ require(!criterion.isNaN)
+
+ @Since("1.6.0")
+ def this(center: Vector, rows: Long, criterion: Double) =
+ this(center, rows, criterion, 0.0, None, Array.empty[BisectingClusterNode])
+
+ /**
+ * Inserts a sub node as its child
+ *
+ * @param child inserted sub node
+ */
+ @Since("1.6.0")
+ def insert(child: BisectingClusterNode) {
+ insert(Array(child))
+ }
+
+ /**
+ * Inserts sub nodes as its children
+ *
+ * @param children inserted sub nodes
+ */
+ @Since("1.6.0")
+ def insert(children: Array[BisectingClusterNode]) {
+ this.children = this.children ++ children
+ children.foreach(child => child.parent = Some(this))
+ }
+
+ /**
+ * Converts the tree into Array class
+ * the sub nodes are recursively expanded
+ *
+ * @return an Array class which the cluster tree is expanded
+ */
+ @Since("1.6.0")
+ def toArray: Array[BisectingClusterNode] = {
+ val array = this.children.size match {
+ case 0 => Array(this)
+ case _ => Array(this) ++ this.children.flatMap(child => child.toArray.toIterator)
+ }
+ array.sortWith { case (a, b) =>
+ a.getDepth < b.getDepth && a.criterion < b.criterion && a.rows < b.rows
+ }
+ }
+
+ /**
+ * Gets the depth of the cluster in the tree
+ *
+ * @return the depth from the root
+ */
+ @Since("1.6.0")
+ def getDepth: Int = {
+ this.parent match {
+ case None => 0
+ case _ => 1 + this.parent.get.getDepth
+ }
+ }
+
+ /**
+ * Gets the leaves nodes in the cluster tree
+ */
+ @Since("1.6.0")
+ def getLeavesNodes: Array[BisectingClusterNode] = {
+ this.toArray.filter(_.isLeaf).sortBy(_.center.toArray.sum)
+ }
+
+ @Since("1.6.0")
+ def isLeaf: Boolean = this.children.isEmpty
+
+ @Since("1.6.0")
+ def getParent: Option[BisectingClusterNode] = this.parent
+
+ @Since("1.6.0")
+ def getChildren: Seq[BisectingClusterNode] = this.children
+
+ /**
+ * Gets the dendrogram height of the cluster at the cluster tree.
+ * A dendrogram height is different from a local height.
+ * A dendrogram height means a total height of a node in a tree.
+ * A local height means a maximum distance between a node and its children.
+ *
+ * @return the dendrogram height
+ */
+ @Since("1.6.0")
+ def getHeight: Double = {
+ this.children.size match {
+ case 0 => 0.0
+ case _ => this.localHeight + this.children.map(_.getHeight).max
+ }
+ }
+
+ @Since("1.6.0")
+ def setLocalHeight(height: Double): Unit = this.localHeight = height
+
+ /**
+ * Converts to an adjacency list
+ *
+ * @return List[(fromNodeId, toNodeId, distance)]
+ */
+ @Since("1.6.0")
+ def toAdjacencyList: Array[(Int, Int, Double)] = {
+ val nodes = toArray
+
+ var adjacencyList = Array.empty[(Int, Int, Double)]
+ nodes.foreach { parent =>
+ if (parent.children.size > 1) {
+ val parentIndex = nodes.indexOf(parent)
+ parent.children.foreach { child =>
+ val childIndex = nodes.indexOf(child)
+ adjacencyList = adjacencyList :+(parentIndex, childIndex, parent.localHeight)
+ }
+ }
+ }
+ adjacencyList
+ }
+
+ /**
+ * Converts to a linkage matrix
+ * Returned data format is fit for scipy's dendrogram function
+ *
+ * @return List[(node1, node2, distance, tree size)]
+ */
+ @Since("1.6.0")
+ def toLinkageMatrix: Array[(Int, Int, Double, Int)] = {
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by freeman-lab <gi...@git.apache.org>.
Github user freeman-lab commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r29121153
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/HierarchicalClustering.scala ---
@@ -0,0 +1,574 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseVector => BDV, SparseVector => BSV, Vector => BV, norm => breezeNorm}
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.random.XORShiftRandom
+import org.apache.spark.{Logging, SparkException}
+
+import scala.collection.{Map, mutable}
+
+
+object HierarchicalClustering extends Logging {
+
+ private[clustering] val ROOT_INDEX_KEY: Long = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ private[mllib]
+ def findClosestCenter(metric: Function2[BV[Double], BV[Double], Double])
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)
+ closestIndex
+ }
+}
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bi-sect k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * @param numClusters tne number of clusters you want
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations
+ * @param maxRetries the number of maximum retries
+ * @param seed a random seed
+ */
+class HierarchicalClustering private (
+ private var numClusters: Int,
+ private var clusterMap: Map[Long, ClusterTree],
+ private var maxIterations: Int,
+ private var maxRetries: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ def this() = this(20, mutable.ListMap.empty[Long, ClusterTree], 20, 10, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ def setNumClusters(numClusters: Int): this.type = {
+ this.numClusters = numClusters
+ this
+ }
+
+ def getNumClusters: Int = this.numClusters
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ def getSubIterations: Int = this.maxIterations
+
+ /**
+ * Sets the number of maximum retries of each clustering step
+ */
+ def setMaxRetries(maxRetries: Int): this.type = {
+ this.maxRetries = maxRetries
+ this
+ }
+
+ def getMaxRetries: Int = this.maxRetries
+
+ /**
+ * Sets the random seed
+ */
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the hierarchical clustering algorithm
+ * @param input RDD of vectors
+ * @return model for the hierarchical clustering
+ */
+ def run(input: RDD[Vector]): HierarchicalClusteringModel = {
+ val sc = input.sparkContext
+ log.info(s"${sc.appName} starts a hierarchical clustering algorithm")
+
+ var data = initData(input).cache()
+ val startTime = System.currentTimeMillis()
+
+ // `clusters` is described as binary tree structure
+ // `clusters(1)` means the root of a binary tree
+ var clusters = summarizeAsClusters(data)
+ var leafClusters = clusters
+ var step = 1
+ var numDividedClusters = 0
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(Long, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.numClusters) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ while (clusters.size < maxAllNodesInTree && noMoreDividable == false) {
+ log.info(s"${sc.appName} starts step ${step}")
+
+ // enough to be clustered if the number of divided clusters is equal to 0
+ val divided = getDividedClusters(data, leafClusters)
+ if (divided.size == 0) {
+ noMoreDividable = true
+ }
+ else {
+ // update each index
+ val newData = updateClusterIndex(data, divided).cache()
+ rddArray = rddArray ++ Array(data)
+ data = newData
+
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (rddArray.size > 1) {
+ val head = rddArray.head
+ head.unpersist()
+ rddArray = rddArray.filterNot(_.hashCode() == head.hashCode())
+ }
+
+ // merge the divided clusters with the map as the cluster tree
+ clusters = clusters ++ divided
+ numDividedClusters = data.map(_._1).distinct().count().toInt
+ leafClusters = divided
+ step += 1
+
+ log.info(s"${sc.appName} adding ${divided.size} new clusters at step:${step}")
+ }
+ }
+ // unpersist kept RDDs
+ rddArray.foreach(_.unpersist())
+
+ // build a cluster tree by Map class which is expressed
+ log.info(s"Building the cluster tree is started in ${sc.appName}")
+ val root = buildTree(clusters, HierarchicalClustering.ROOT_INDEX_KEY, this.numClusters)
+ if (root == None) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusters")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ log.info(s"Elapsed Time for Hierarchical Clustering Training: ${finishTime} [sec]")
+
+ // make a hierarchical clustering model
+ val model = new HierarchicalClusteringModel(root.get)
+ val leavesNodes = model.getClusters()
+ if (leavesNodes.size < this.numClusters) {
+ log.warn(s"# clusters is less than you have expected: ${leavesNodes.size} / ${numClusters}. ")
+ }
+ model
+ }
+
+ /**
+ * Assigns the initial cluster index id to all data
+ */
+ private[clustering]
+ def initData(data: RDD[Vector]): RDD[(Long, BV[Double])] = {
+ data.map { v: Vector => (HierarchicalClustering.ROOT_INDEX_KEY, v.toBreeze)}.cache
+ }
+
+ /**
+ * Summarizes data by each cluster as ClusterTree2 classes
--- End diff --
What is "ClusterTree2"? Did you mean "ClusterTree"?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-88387155
@freeman-lab, Thank you for your attention to this matter.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152350975
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-87696607
[Test build #29403 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29403/consoleFull) for PR 5267 at commit [`3df7f11`](https://github.com/apache/spark/commit/3df7f1157c67135b5dde451b540fe30deb730c99).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-150136686
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44138/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155190761
**[Test build #45406 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45406/consoleFull)** for PR 5267 at commit [`29ccdf9`](https://github.com/apache/spark/commit/29ccdf9eaa987530435782d2051acbeda3d3ac36).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * ` * Sets the random seed (default: hash value of the class name).`\n * `class BisectingKMeansModel @Since(\"1.6.0\") (`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155177514
**[Test build #45406 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/45406/consoleFull)** for PR 5267 at commit [`29ccdf9`](https://github.com/apache/spark/commit/29ccdf9eaa987530435782d2051acbeda3d3ac36).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155225558
LGTM. Merged into master. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r27613298
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/HierarchicalClustering.scala ---
@@ -0,0 +1,610 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseVector => BDV, SparseVector => BSV, Vector => BV, norm => breezeNorm}
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.random.XORShiftRandom
+import org.apache.spark.{Logging, SparkException}
+
+import scala.collection.{Map, mutable}
+
+
+/**
+ * Top-level methods for calling the hierarchical clustering algorithm
+ */
+object HierarchicalClustering extends Logging {
--- End diff --
I don't think we need static train() methods since users can use the builder pattern from the HierarchicalClustering class.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-87715753
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29402/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155190941
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155226627
Thank you for merging it!!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151320160
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r27743803
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/HierarchicalClustering.scala ---
@@ -0,0 +1,610 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseVector => BDV, SparseVector => BSV, Vector => BV, norm => breezeNorm}
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.random.XORShiftRandom
+import org.apache.spark.{Logging, SparkException}
+
+import scala.collection.{Map, mutable}
+
+
+/**
+ * Top-level methods for calling the hierarchical clustering algorithm
+ */
+object HierarchicalClustering extends Logging {
--- End diff --
Yes, that's an issue but one where there isn't a great solution. Let's discuss it on the JIRA: [https://issues.apache.org/jira/browse/SPARK-6682]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-96768993
[Test build #31035 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31035/consoleFull) for PR 5267 at commit [`9d32110`](https://github.com/apache/spark/commit/9d32110f9828225bf65cca6e93ff88457ded5ce8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-87715737
[Test build #29402 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29402/consoleFull) for PR 5267 at commit [`af0f65b`](https://github.com/apache/spark/commit/af0f65bb4726315c076b827d37276616c5218010).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class HierarchicalClustering(`
* `class ClusterTree(`
* `class HierarchicalClusteringModel(val tree: ClusterTree)`
* `class HierarchicalClusteringModel(JavaModelWrapper, JavaSaveable, JavaLoader):`
* `class HierarchicalClustering(object):`
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151321462
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44384/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-150272472
**[Test build #44157 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44157/consoleFull)** for PR 5267 at commit [`31623ea`](https://github.com/apache/spark/commit/31623ead10d4664c7c5d98b5c72731bcc804bcba).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-150136544
**[Test build #44138 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44138/consoleFull)** for PR 5267 at commit [`3156dd7`](https://github.com/apache/spark/commit/3156dd7e0c4fc4f5d1b1116bb689592eee76ec25).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class BisectingKMeansModel @Since(\"1.6.0\") (`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42821027
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = BisectingKMeans.initData(input).cache()
+ while (clusterStats.size < maxAllNodesInTree && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ val leafClusters = BisectingKMeans.summarizeClusters(data)
+ val dividableLeafClusters = leafClusters.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusters
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ else {
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided =
+ BisectingKMeans.divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = BisectingKMeans.updateClusterIndex(data, divided).cache()
+ rddArray = rddArray ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (rddArray.length > 1) {
+ val head = rddArray.head
+ head.unpersist()
+ rddArray = rddArray.filterNot(_.hashCode() == head.hashCode())
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+ }
+ }
+ // unpersist kept RDDs
+ rddArray.foreach(_.unpersist())
+ // create a map of cluster node with their criterions
+ val nodes = BisectingKMeans.createClusterNodes(data, clusterStats)
+
+ // build a cluster tree by Map class which is expressed
+ logInfo(s"Building the cluster tree is started in ${sc.appName}")
+ val root = BisectingKMeans.buildTree(nodes, BisectingKMeans.ROOT_INDEX_KEY, this.k)
+ if (root.isEmpty) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusterStats")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ logInfo(s"Elapsed Time for ${this.getClass.getSimpleName} Training: ${finishTime} [sec]")
+
+ // make a bisecting kmeans model
+ val model = new BisectingKMeansModel(root.get)
+ val leavesNodes = model.getClusters
+ if (leavesNodes.length < this.k) {
+ logWarning(s"# clusters is less than you want: ${leavesNodes.length} / ${k}")
+ }
+ model
+ }
+}
+
+
+private[clustering] object BisectingKMeans {
+
+ val ROOT_INDEX_KEY: BigInt = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ def findClosestCenter(metric: (BV[Double], BV[Double]) => Double)
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)
+ closestIndex
+ }
+
+ /**
+ * Summarizes data by each cluster as Map
+ *
+ * @param data pairs of point and its cluster index
+ */
+ def summarizeClusters(data: RDD[(BigInt, BV[Double])]): Map[BigInt, BisectingClusterStat] = {
+
+ val stats = data.mapPartitions { iter =>
+ // calculate the accumulation of the all point in a partition and count the rows
+ val map = mutable.Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx: BigInt, point: BV[Double]) =>
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(idx, (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size)))
+ map(idx) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ map.toIterator
+ }.reduceByKey { case ((sum1, n1, sumOfSquares1), (sum2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sum1 + sum2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.collect().toMap
+
+ stats.map {case (i, stat) => i -> new BisectingClusterStat(stat._2.toLong, stat._1, stat._3)}
+ }
+
+ /**
+ * Assigns the initial cluster index id to all data
+ */
+ def initData(data: RDD[Vector]): RDD[(BigInt, BV[Double])] = {
+ data.map { v: Vector => (BisectingKMeans.ROOT_INDEX_KEY, v.toBreeze)}
+ }
+
+ /**
+ * Gets the initial centers for bisect k-means
--- End diff --
`bisect` -> `bisecting`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-92547494
@freeman-lab, do you know any good evaluations of a hierarchical clustering algorithm except Within Set Sumb of Squared Error(WSSSE)? For example, I know Silhouette Coefficient is a evaluation for it. However, I thinks it is hard to implement as a distributed processing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42821019
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = BisectingKMeans.initData(input).cache()
+ while (clusterStats.size < maxAllNodesInTree && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ val leafClusters = BisectingKMeans.summarizeClusters(data)
+ val dividableLeafClusters = leafClusters.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusters
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ else {
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided =
+ BisectingKMeans.divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = BisectingKMeans.updateClusterIndex(data, divided).cache()
+ rddArray = rddArray ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (rddArray.length > 1) {
+ val head = rddArray.head
+ head.unpersist()
+ rddArray = rddArray.filterNot(_.hashCode() == head.hashCode())
+ }
+ clusterStats = clusterStats ++ divided
+ step += 1
+ logInfo(s"${sc.appName} adding ${divided.size} new clusterStats at step:${step}")
+ }
+ }
+ // unpersist kept RDDs
+ rddArray.foreach(_.unpersist())
+ // create a map of cluster node with their criterions
+ val nodes = BisectingKMeans.createClusterNodes(data, clusterStats)
+
+ // build a cluster tree by Map class which is expressed
+ logInfo(s"Building the cluster tree is started in ${sc.appName}")
+ val root = BisectingKMeans.buildTree(nodes, BisectingKMeans.ROOT_INDEX_KEY, this.k)
+ if (root.isEmpty) {
+ new SparkException("Failed to build a cluster tree from a Map type of clusterStats")
+ }
+
+ // set the elapsed time for training
+ val finishTime = (System.currentTimeMillis() - startTime) / 1000.0
+ logInfo(s"Elapsed Time for ${this.getClass.getSimpleName} Training: ${finishTime} [sec]")
+
+ // make a bisecting kmeans model
+ val model = new BisectingKMeansModel(root.get)
+ val leavesNodes = model.getClusters
+ if (leavesNodes.length < this.k) {
+ logWarning(s"# clusters is less than you want: ${leavesNodes.length} / ${k}")
+ }
+ model
+ }
+}
+
+
+private[clustering] object BisectingKMeans {
+
+ val ROOT_INDEX_KEY: BigInt = 1
+
+ /**
+ * Finds the closes cluster's center
+ *
+ * @param metric a distance metric
+ * @param centers centers of the clusters
+ * @param point a target point
+ * @return an index of the array of clusters
+ */
+ def findClosestCenter(metric: (BV[Double], BV[Double]) => Double)
+ (centers: Seq[BV[Double]])(point: BV[Double]): Int = {
+ val (closestCenter, closestIndex) =
+ centers.zipWithIndex.map { case (center, idx) => (metric(center, point), idx)}.minBy(_._1)
+ closestIndex
+ }
+
+ /**
+ * Summarizes data by each cluster as Map
+ *
+ * @param data pairs of point and its cluster index
+ */
+ def summarizeClusters(data: RDD[(BigInt, BV[Double])]): Map[BigInt, BisectingClusterStat] = {
+
+ val stats = data.mapPartitions { iter =>
+ // calculate the accumulation of the all point in a partition and count the rows
+ val map = mutable.Map.empty[BigInt, (BV[Double], Double, BV[Double])]
+ iter.foreach { case (idx: BigInt, point: BV[Double]) =>
+ // get a map value or else get a sparse vector
+ val (sumBV, n, sumOfSquares) = map
+ .getOrElse(idx, (BSV.zeros[Double](point.size), 0.0, BSV.zeros[Double](point.size)))
+ map(idx) = (sumBV + point, n + 1.0, sumOfSquares + (point :* point))
+ }
+ map.toIterator
+ }.reduceByKey { case ((sum1, n1, sumOfSquares1), (sum2, n2, sumOfSquares2)) =>
+ // sum the accumulation and the count in the all partition
+ (sum1 + sum2, n1 + n2, sumOfSquares1 + sumOfSquares2)
+ }.collect().toMap
--- End diff --
remove `.toMap`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by freeman-lab <gi...@git.apache.org>.
Github user freeman-lab commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-96511859
@yu-iskw I'm not familiar with any other self-contained metrics (there are a bunch of metrics for relating estimated clusters to some known ground-truth clustering, but I don't think that's what you mean). Are you wanting to provide other outputs to the user to assess clustering quality?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-150126127
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152342783
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151313092
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-87726902
[Test build #29403 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29403/consoleFull) for PR 5267 at commit [`3df7f11`](https://github.com/apache/spark/commit/3df7f1157c67135b5dde451b540fe30deb730c99).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class HierarchicalClustering(`
* `class ClusterTree(`
* `class HierarchicalClusteringModel(val tree: ClusterTree)`
* `class HierarchicalClusteringModel(JavaModelWrapper, JavaSaveable, JavaLoader):`
* `class HierarchicalClustering(object):`
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-155314648
@jkbradley thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by yu-iskw <gi...@git.apache.org>.
Github user yu-iskw commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-88730450
@jkbradley, thank you for your quick replying. I understand you are waiting for @freeman-lab 's review. I am making an example and writing the documentation in parallel now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r27682172
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/HierarchicalClustering.scala ---
@@ -0,0 +1,610 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{DenseVector => BDV, SparseVector => BSV, Vector => BV, norm => breezeNorm}
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+import org.apache.spark.util.random.XORShiftRandom
+import org.apache.spark.{Logging, SparkException}
+
+import scala.collection.{Map, mutable}
+
+
+/**
+ * Top-level methods for calling the hierarchical clustering algorithm
+ */
+object HierarchicalClustering extends Logging {
--- End diff --
Those static train() methods used to be added everywhere, but at some point, we realized that they require a lot of duplicated code (since Java does not recognize default arguments). We're trying to just add builder methods from now on, but we have to keep the old static train() methods for API stability.
You're right about consistency. I've been thinking about whether we should start deprecating the old static train() methods.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151321461
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42821931
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala ---
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import breeze.linalg.{Vector => BV, norm => breezeNorm}
+
+import org.apache.spark.Logging
+import org.apache.spark.annotation.Since
+import org.apache.spark.api.java.JavaRDD
+import org.apache.spark.mllib.linalg.Vector
+import org.apache.spark.rdd.RDD
+
+/**
+ * This class is used for the model of the bisecting kmeans
+ *
+ * @param node a cluster as a tree node
+ */
+@Since("1.6.0")
+class BisectingKMeansModel @Since("1.6.0") (
+ @Since("1.6.0") val node: BisectingClusterNode
+ ) extends Serializable with Logging {
+
+ @Since("1.6.0")
+ def getClusters: Array[BisectingClusterNode] = this.node.getLeavesNodes
+
+ @Since("1.6.0")
+ def getCenters: Array[Vector] = this.getClusters.map(_.center)
+
+ /**
+ * Predicts the closest cluster by one point
+ */
+ @Since("1.6.0")
+ def predict(vector: Vector): Int = {
--- End diff --
* We need to do this from root to leaf so we don't need to compare the point with all leaf clusters.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152259884
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151320164
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44382/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-150126146
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-150285292
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44157/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151311530
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/5267#discussion_r42821008
--- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeans.scala ---
@@ -0,0 +1,690 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.mllib.clustering
+
+import scala.collection.{Map, mutable}
+
+import breeze.linalg
+ .{SparseVector => BSV, Vector => BV, any => breezeAny, norm => breezeNorm, sum => breezeSum}
+
+import org.apache.spark.{Logging, SparkException}
+import org.apache.spark.annotation.Since
+import org.apache.spark.mllib.linalg.{Vector, Vectors}
+import org.apache.spark.rdd.RDD
+
+
+/**
+ * This is a divisive hierarchical clustering algorithm based on bisecting k-means algorithm.
+ *
+ * The main idea of this algorithm is based on "A comparison of document clustering techniques",
+ * M. Steinbach, G. Karypis and V. Kumar. Workshop on Text Mining, KDD, 2000.
+ * http://cs.fit.edu/~pkc/classes/ml-internet/papers/steinbach00tr.pdf
+ *
+ * However, we modified it to fit for Spark. This algorithm consists of the two main parts.
+ *
+ * 1. Split clusters until the number of clusters will be enough to build a cluster tree
+ * 2. Build a cluster tree as a binary tree by the splitted clusters
+ *
+ * First, it splits clusters to their children clusters step by step, not considering a cluster
+ * will be included in the final cluster tree or not. That's because it makes the algorithm more
+ * efficient on Spark and splitting a cluster one by one is very slow. It will keep splitting until
+ * the number of clusters will be enough to build a cluster tree. Otherwise, it will stop splitting
+ * when there are no dividable clusters before the number of clusters will be sufficient. And
+ * it calculates the criterions, such as average cost, entropy and so on, for building a cluster
+ * tree in the first part. The criterion means how large the cluster is. That is, the cluster
+ * whose criterion is maximum of all the clusters is the largest cluster.
+ *
+ * Second, it builds a cluster tree as a binary tree by the result of the first part.
+ * First of all, the cluster tree starts with only the root cluster which includes all points.
+ * So, there are two candidates which can be merged to the cluster tree. Those are the children of
+ * the root. Then, it picks up the larger child of the two and merge it to the cluster tree.
+ * After that, there are tree candidates to merge. Those are the smaller child of the root and
+ * the two children of the larger cluster of the root. It picks up the largest cluster of the tree
+ * and merge it to the * cluster tree. Like this, it continues to pick up the largest one of the
+ * candidates and merge it to the cluster tree until the desired number of clusters is reached.
+ *
+ * @param k tne desired number of clusters
+ * @param clusterMap the pairs of cluster and its index as Map
+ * @param maxIterations the number of maximal iterations to split clusters
+ * @param seed a random seed
+ */
+@Since("1.6.0")
+class BisectingKMeans private (
+ private var k: Int,
+ private var clusterMap: Map[BigInt, BisectingClusterNode],
+ private var maxIterations: Int,
+ private var seed: Long) extends Logging {
+
+ /**
+ * Constructs with the default configuration
+ */
+ @Since("1.6.0")
+ def this() = this(20, mutable.ListMap.empty[BigInt, BisectingClusterNode], 20, 1)
+
+ /**
+ * Sets the number of clusters you want
+ */
+ @Since("1.6.0")
+ def setK(k: Int): this.type = {
+ this.k = k
+ this
+ }
+
+ @Since("1.6.0")
+ def getK: Int = this.k
+
+ /**
+ * Sets the number of maximal iterations in each clustering step
+ */
+ @Since("1.6.0")
+ def setMaxIterations(maxIterations: Int): this.type = {
+ this.maxIterations = maxIterations
+ this
+ }
+
+ @Since("1.6.0")
+ def getMaxIterations: Int = this.maxIterations
+
+ /**
+ * Sets the random seed
+ */
+ @Since("1.6.0")
+ def setSeed(seed: Long): this.type = {
+ this.seed = seed
+ this
+ }
+
+ @Since("1.6.0")
+ def getSeed: Long = this.seed
+
+ /**
+ * Runs the bisecting k-means algorithm
+ * @param input RDD of vectors
+ * @return model for the bisecting kmeans
+ */
+ @Since("1.6.0")
+ def run(input: RDD[Vector]): BisectingKMeansModel = {
+ val sc = input.sparkContext
+
+ // `clusterStats` is described as binary tree structure
+ // `clusterStats(1)` means the root of a binary tree
+ var clusterStats = mutable.Map.empty[BigInt, BisectingClusterStat]
+ var step = 1
+ var noMoreDividable = false
+ var rddArray = Array.empty[RDD[(BigInt, BV[Double])]]
+ // the number of maximum nodes of a binary tree by given parameter
+ val multiplier = math.ceil(math.log10(this.k) / math.log10(2.0)) + 1
+ val maxAllNodesInTree = math.pow(2, multiplier).toInt
+
+ // divide clusters until the number of clusters reachs the condition
+ // or there is no dividable cluster
+ val startTime = System.currentTimeMillis()
+ var data = BisectingKMeans.initData(input).cache()
+ while (clusterStats.size < maxAllNodesInTree && noMoreDividable == false) {
+ logInfo(s"${sc.appName} starts step ${step}")
+ val leafClusters = BisectingKMeans.summarizeClusters(data)
+ val dividableLeafClusters = leafClusters.filter(_._2.isDividable)
+ clusterStats = clusterStats ++ leafClusters
+
+ if (dividableLeafClusters.isEmpty) {
+ noMoreDividable = true
+ }
+ else {
+ // can be clustered if the number of divided clusterStats is equal to 0
+ val divided =
+ BisectingKMeans.divideClusters(data, dividableLeafClusters, maxIterations)
+ // update each index
+ val newData = BisectingKMeans.updateClusterIndex(data, divided).cache()
+ rddArray = rddArray ++ Array(data)
+ data = newData
+ // keep recent 2 cached RDDs in order to run more quickly
+ if (rddArray.length > 1) {
+ val head = rddArray.head
+ head.unpersist()
+ rddArray = rddArray.filterNot(_.hashCode() == head.hashCode())
--- End diff --
`reedArray = rddArray.tail`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-152350863
**[Test build #44642 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44642/consoleFull)** for PR 5267 at commit [`5da05d3`](https://github.com/apache/spark/commit/5da05d3fd41d5b730d19f595ca9bf622aaf0a14d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class BisectingKMeansModel @Since(\"1.6.0\") (`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-88635522
Thanks @freeman-lab in advance for reviewing! I'd be happy to make a pass too, but will wait to avoid duplicate passes over the code.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151312766
**[Test build #44382 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44382/consoleFull)** for PR 5267 at commit [`2e8226d`](https://github.com/apache/spark/commit/2e8226d509ead4c14940335bc44c110d4c02c52f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-151321365
**[Test build #44384 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44384/consoleFull)** for PR 5267 at commit [`622499e`](https://github.com/apache/spark/commit/622499e95e2717625eecd11f5c43e0c9eddc820e).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:\n * `class BisectingKMeansModel @Since(\"1.6.0\") (`\n
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-90884821
[Test build #29855 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29855/consoleFull) for PR 5267 at commit [`9d32110`](https://github.com/apache/spark/commit/9d32110f9828225bf65cca6e93ff88457ded5ce8).
* This patch **passes all tests**.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class HierarchicalClusteringModel(val tree: ClusterTree)`
* `class HierarchicalClusteringModel(JavaModelWrapper, JavaSaveable, JavaLoader):`
* `class HierarchicalClustering(object):`
* This patch does not change any dependencies.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-150127788
**[Test build #44138 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44138/consoleFull)** for PR 5267 at commit [`3156dd7`](https://github.com/apache/spark/commit/3156dd7e0c4fc4f5d1b1116bb689592eee76ec25).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-6517][mllib] Implement the Algorithm of...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/5267#issuecomment-90865175
[Test build #29855 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29855/consoleFull) for PR 5267 at commit [`9d32110`](https://github.com/apache/spark/commit/9d32110f9828225bf65cca6e93ff88457ded5ce8).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org