You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by rx...@apache.org on 2013/12/18 07:26:30 UTC
[1/2] git commit: Fixed a performance problem in RDD.top and
BoundedPriorityQueue (size in BoundedPriority was actually traversing the
entire queue to calculate the size,
resulting in bad performance in insertion).
Updated Branches:
refs/heads/master 7a8169be9 -> f4effb375
Fixed a performance problem in RDD.top and BoundedPriorityQueue (size in BoundedPriority was actually traversing the entire queue to calculate the size, resulting in bad performance in insertion).
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/9a6864d0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/9a6864d0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/9a6864d0
Branch: refs/heads/master
Commit: 9a6864d016f0f923c885ca2a4977104cd19ded4f
Parents: 7a8169b
Author: Reynold Xin <rx...@apache.org>
Authored: Tue Dec 17 18:44:39 2013 -0800
Committer: Reynold Xin <rx...@apache.org>
Committed: Tue Dec 17 18:44:39 2013 -0800
----------------------------------------------------------------------
.../main/scala/org/apache/spark/util/BoundedPriorityQueue.scala | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-spark/blob/9a6864d0/core/src/main/scala/org/apache/spark/util/BoundedPriorityQueue.scala
----------------------------------------------------------------------
diff --git a/core/src/main/scala/org/apache/spark/util/BoundedPriorityQueue.scala b/core/src/main/scala/org/apache/spark/util/BoundedPriorityQueue.scala
index 0b51c23..a38329d 100644
--- a/core/src/main/scala/org/apache/spark/util/BoundedPriorityQueue.scala
+++ b/core/src/main/scala/org/apache/spark/util/BoundedPriorityQueue.scala
@@ -34,6 +34,8 @@ class BoundedPriorityQueue[A](maxSize: Int)(implicit ord: Ordering[A])
override def iterator: Iterator[A] = underlying.iterator.asScala
+ override def size: Int = underlying.size
+
override def ++=(xs: TraversableOnce[A]): this.type = {
xs.foreach { this += _ }
this
[2/2] git commit: Merge pull request #273 from rxin/top
Posted by rx...@apache.org.
Merge pull request #273 from rxin/top
Fixed a performance problem in RDD.top and BoundedPriorityQueue
BoundedPriority was actually traversing the entire queue to calculate the size, resulting in bad performance in insertion.
This should also cherry pick cleanly into branch-0.8.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/f4effb37
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/f4effb37
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/f4effb37
Branch: refs/heads/master
Commit: f4effb375e93993be1777ebb423c100ea8422f24
Parents: 7a8169b 9a6864d
Author: Reynold Xin <rx...@apache.org>
Authored: Tue Dec 17 22:26:21 2013 -0800
Committer: Reynold Xin <rx...@apache.org>
Committed: Tue Dec 17 22:26:21 2013 -0800
----------------------------------------------------------------------
.../main/scala/org/apache/spark/util/BoundedPriorityQueue.scala | 2 ++
1 file changed, 2 insertions(+)
----------------------------------------------------------------------