You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by jk...@apache.org on 2018/09/06 16:53:02 UTC
spark git commit: [SPARK-25268][GRAPHX] run Parallel Personalized
PageRank throws serialization Exception
Repository: spark
Updated Branches:
refs/heads/master 7ef6d1daf -> 3b6591b0b
[SPARK-25268][GRAPHX] run Parallel Personalized PageRank throws serialization Exception
## What changes were proposed in this pull request?
mapValues in scala is currently not serializable. To avoid the serialization issue while running pageRank, we need to use map instead of mapValues.
Please review http://spark.apache.org/contributing.html before opening a pull request.
Closes #22271 from shahidki31/master_latest.
Authored-by: Shahid <sh...@gmail.com>
Signed-off-by: Joseph K. Bradley <jo...@databricks.com>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/3b6591b0
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/3b6591b0
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/3b6591b0
Branch: refs/heads/master
Commit: 3b6591b0b064b13a411e5b8f8ee4883a69c39e2d
Parents: 7ef6d1d
Author: Shahid <sh...@gmail.com>
Authored: Thu Sep 6 09:52:58 2018 -0700
Committer: Joseph K. Bradley <jo...@databricks.com>
Committed: Thu Sep 6 09:52:58 2018 -0700
----------------------------------------------------------------------
.../main/scala/org/apache/spark/graphx/lib/PageRank.scala | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/3b6591b0/graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala
----------------------------------------------------------------------
diff --git a/graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala b/graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala
index 96b635f..1305c05 100644
--- a/graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala
+++ b/graphx/src/main/scala/org/apache/spark/graphx/lib/PageRank.scala
@@ -198,9 +198,11 @@ object PageRank extends Logging {
val zero = Vectors.sparse(sources.size, List()).asBreeze
// map of vid -> vector where for each vid, the _position of vid in source_ is set to 1.0
- val sourcesInitMap = sources.zipWithIndex.toMap.mapValues { i =>
- Vectors.sparse(sources.size, Array(i), Array(1.0)).asBreeze
- }
+ val sourcesInitMap = sources.zipWithIndex.map { case (vid, i) =>
+ val v = Vectors.sparse(sources.size, Array(i), Array(1.0)).asBreeze
+ (vid, v)
+ }.toMap
+
val sc = graph.vertices.sparkContext
val sourcesInitMapBC = sc.broadcast(sourcesInitMap)
// Initialize the PageRank graph with each edge attribute having
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org