You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by SereneAnt <gi...@git.apache.org> on 2017/07/20 21:11:12 UTC
[GitHub] spark pull request #18693: [SPARK-21491][GraphX] Enhance GraphX performance:...
GitHub user SereneAnt opened a pull request:
https://github.com/apache/spark/pull/18693
[SPARK-21491][GraphX] Enhance GraphX performance: breakOut instead of .toMap
## What changes were proposed in this pull request?
`Traversable.toMap` changed to 'collections.breakOut', that eliminates intermediate tuple collection creation, see [Stack Overflow article](https://stackoverflow.com/questions/1715681/scala-2-8-breakout).
## How was this patch tested?
Unit tests run.
No performance tests performed yet.
Please review http://spark.apache.org/contributing.html before opening a pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/SereneAnt/spark performance_toMap-breakOut
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18693.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18693
----
commit d7b73b544116314439a536ac944d66a8d7ee1113
Author: iurii.ant <se...@gmail.com>
Date: 2017-07-20T21:03:43Z
[SPARK-21491][GraphX] Enhance GraphX performance: eliminate intermediate collections creation with breakOut
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18693: [SPARK-21491][GraphX] Enhance GraphX performance: breakO...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/18693
Merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18693: [SPARK-21491][GraphX] Enhance GraphX performance: breakO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18693
**[Test build #3850 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3850/testReport)** for PR 18693 at commit [`0ae9cc5`](https://github.com/apache/spark/commit/0ae9cc52737c6864d8db95ac2cbfe7b2334c5e5c).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18693: [SPARK-21491][GraphX] Enhance GraphX performance: breakO...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18693
**[Test build #3850 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3850/testReport)** for PR 18693 at commit [`0ae9cc5`](https://github.com/apache/spark/commit/0ae9cc52737c6864d8db95ac2cbfe7b2334c5e5c).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18693: [SPARK-21491][GraphX] Enhance GraphX performance:...
Posted by SereneAnt <gi...@git.apache.org>.
Github user SereneAnt commented on a diff in the pull request:
https://github.com/apache/spark/pull/18693#discussion_r128637486
--- Diff: graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala ---
@@ -199,10 +199,10 @@ object PageRank extends Logging {
require(sources.max <= Int.MaxValue.toLong,
s"This implementation currently only works for source vertex ids at most ${Int.MaxValue}")
val zero = Vectors.sparse(sources.size, List()).asBreeze
- val sourcesInitMap = sources.zipWithIndex.map { case (vid, i) =>
+ val sourcesInitMap: Map[VertexId, BV[Double]] = sources.zipWithIndex.map { case (vid, i) =>
val v = Vectors.sparse(sources.size, Array(i), Array(1.0)).asBreeze
(vid, v)
- }.toMap
+ }(collection.breakOut)
--- End diff --
Optimization nerds have already used it in the spark code:
https://github.com/apache/spark/blob/c4008480b781379ac0451b9220300d83c054c60d/sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala#L517
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18693: [SPARK-21491][GraphX] Enhance GraphX performance: breakO...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18693
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18693: [SPARK-21491][GraphX] Enhance GraphX performance:...
Posted by SereneAnt <gi...@git.apache.org>.
Github user SereneAnt commented on a diff in the pull request:
https://github.com/apache/spark/pull/18693#discussion_r128635631
--- Diff: graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala ---
@@ -199,10 +199,10 @@ object PageRank extends Logging {
require(sources.max <= Int.MaxValue.toLong,
s"This implementation currently only works for source vertex ids at most ${Int.MaxValue}")
val zero = Vectors.sparse(sources.size, List()).asBreeze
- val sourcesInitMap = sources.zipWithIndex.map { case (vid, i) =>
+ val sourcesInitMap: Map[VertexId, BV[Double]] = sources.zipWithIndex.map { case (vid, i) =>
val v = Vectors.sparse(sources.size, Array(i), Array(1.0)).asBreeze
(vid, v)
- }.toMap
+ }(collection.breakOut)
--- End diff --
The principles are the same,
`sources.zipWithIndex.map {...}' allocates a collection of tuples, `.toMap` then uterates and converts them into the map (hehe).
breakOut is the implementation of CanBuildFrom, the implicit parameter passed to `Traversable.map` method. Once used, it allows populating newborn map directly, without intermediate collection of tuples.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18693: [SPARK-21491][GraphX] Enhance GraphX performance:...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/18693
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18693: [SPARK-21491][GraphX] Enhance GraphX performance:...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/18693#discussion_r128632856
--- Diff: graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala ---
@@ -199,10 +199,10 @@ object PageRank extends Logging {
require(sources.max <= Int.MaxValue.toLong,
s"This implementation currently only works for source vertex ids at most ${Int.MaxValue}")
val zero = Vectors.sparse(sources.size, List()).asBreeze
- val sourcesInitMap = sources.zipWithIndex.map { case (vid, i) =>
+ val sourcesInitMap: Map[VertexId, BV[Double]] = sources.zipWithIndex.map { case (vid, i) =>
val v = Vectors.sparse(sources.size, Array(i), Array(1.0)).asBreeze
(vid, v)
- }.toMap
+ }(collection.breakOut)
--- End diff --
Ah OK that's what it is. Why is this different? the article you cited isn't comparing to toMap, and toMap doesn't make an intermediate collection.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org