You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by ok...@apache.org on 2015/12/09 16:14:25 UTC

incubator-tinkerpop git commit: fixed a SparkGraphComputer sorting bug in MapReduce that occurred when there is more than one graph partition. This was exposed by ToyGraphInputRDD which is a 4 partition. We never saw this before cause all the InputFormat

Repository: incubator-tinkerpop
Updated Branches:
  refs/heads/master 67845f660 -> f8672a41f


fixed a SparkGraphComputer sorting bug in MapReduce that occurred when there is more than one graph partition. This was exposed by ToyGraphInputRDD which is a 4 partition. We never saw this before cause all the InputFormats are single partitions. Dah. Thank you @spmallette for helping me with this as my computer just happens to retrieve the partitions in order and thus, I never saw the problem locally. CTR.


Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/f8672a41
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/f8672a41
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/f8672a41

Branch: refs/heads/master
Commit: f8672a41face8935e3743d399d0bcf409fe1f9e4
Parents: 67845f6
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Wed Dec 9 08:14:21 2015 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Wed Dec 9 08:14:21 2015 -0700

----------------------------------------------------------------------
 CHANGELOG.asciidoc                                               | 1 +
 .../tinkerpop/gremlin/spark/process/computer/SparkExecutor.java  | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/f8672a41/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index 52dae62..d61d8db 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -26,6 +26,7 @@ image::https://raw.githubusercontent.com/apache/incubator-tinkerpop/master/docs/
 TinkerPop 3.1.1 (NOT OFFICIALLY RELEASED YET)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+* Fixed a `SparkGraphComputer` sorting bug in MapReduce that occurred when there was more than one partition.
 * Fixed a `NullPointerException` bug in `PeerPressureVertexProgram` that occurred when an adjacency traversal was not provided.
 * Fixed a long standing "view merge" issue requiring `reduceByKey()` on input data to Spark. It is no longer required.
 * Added `Spark` static object to allow "file system" control of persisted RDDs in Spark.

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/f8672a41/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkExecutor.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkExecutor.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkExecutor.java
index 1370f4a..0c6c795 100644
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkExecutor.java
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkExecutor.java
@@ -168,7 +168,7 @@ public final class SparkExecutor {
             });
         });
         if (mapReduce.getMapKeySort().isPresent())
-            mapRDD = mapRDD.sortByKey(mapReduce.getMapKeySort().get());
+            mapRDD = mapRDD.sortByKey(mapReduce.getMapKeySort().get(), true, 1);
         return mapRDD;
     }
 
@@ -188,7 +188,7 @@ public final class SparkExecutor {
             });
         });
         if (mapReduce.getReduceKeySort().isPresent())
-            reduceRDD = reduceRDD.sortByKey(mapReduce.getReduceKeySort().get());
+            reduceRDD = reduceRDD.sortByKey(mapReduce.getReduceKeySort().get(), true, 1);
         return reduceRDD;
     }