You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by ok...@apache.org on 2015/03/04 17:58:51 UTC
incubator-tinkerpop git commit: fixing up HadoopGremlinPlugin to work
for both Spark and Giraph. A few dangling oddities that I will fix up in a
few. Added SparkGraphComputer to the documentation.
Repository: incubator-tinkerpop
Updated Branches:
refs/heads/master 8017dae7a -> d9bbc3aa5
fixing up HadoopGremlinPlugin to work for both Spark and Giraph. A few dangling oddities that I will fix up in a few. Added SparkGraphComputer to the documentation.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/d9bbc3aa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/d9bbc3aa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/d9bbc3aa
Branch: refs/heads/master
Commit: d9bbc3aa583d07fc2c171cf98b80ddfb517535cf
Parents: 8017dae
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Wed Mar 4 09:58:42 2015 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Wed Mar 4 09:58:42 2015 -0700
----------------------------------------------------------------------
docs/src/implementations.asciidoc | 71 ++++++----
docs/static/images/tinkerpop3.graffle | 142 +++++++++----------
.../traversal/step/map/ComputerResultStep.java | 2 +-
.../groovy/plugin/HadoopGremlinPlugin.java | 2 +
.../gremlin/hadoop/structure/HadoopGraph.java | 4 +-
pom.xml | 2 +
6 files changed, 119 insertions(+), 104 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d9bbc3aa/docs/src/implementations.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/implementations.asciidoc b/docs/src/implementations.asciidoc
index cb07864..257f9b8 100644
--- a/docs/src/implementations.asciidoc
+++ b/docs/src/implementations.asciidoc
@@ -658,7 +658,7 @@ gremlin>
Properties Files
~~~~~~~~~~~~~~~~
-`HadoopGraph` makes heavy use of properties files which ultimately get turned into Apache configurations and Hadoop configurations. The properties file presented below is located at `conf/giraph-kryo.properties'.
+`HadoopGraph` makes heavy use of properties files which ultimately get turned into Apache configurations and Hadoop configurations. The properties file presented below is located at `conf/giraph-gryo.properties`.
[source,text]
gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
@@ -709,8 +709,8 @@ gremlin> hdfs.copyFromLocal('data/tinkerpop-modern-vertices.kryo', 'tinkerpop-mo
==>null
gremlin> hdfs.ls()
==>rw-r--r-- marko supergroup 1439 tinkerpop-modern-vertices.kryo
-gremlin> g = GraphFactory.open('../../../hadoop-gremlin/conf/hadoop-gryo.properties')
-==>hadoopgraph[gryoinputformat->gryooutputformat]
+gremlin> g = GraphFactory.open('../../../hadoop-gremlin/conf/giraph-gryo.properties')
+==>hadoopgraph[gryoinputformat->gryooutputformat[giraphgraphcomputer]]
gremlin> g.V().count()
==>6
gremlin> g.V().out().out().values('name')
@@ -729,18 +729,22 @@ image:hadoop-furnace.png[width=180,float=left] Hadoop-Gremlin was designed to ex
IMPORTANT: As of TinkerPop3 x.y.z, when using Hadoop-Gremlin OLAP from the Gremlin Console, the only Gremlin language subset supported is Gremlin-Groovy. Future versions will support other Gremlin language dialects.
-A `Graph` in TinkerPop3 can support any number of `GraphComputer` implementations. HadoopGraph supports two GraphComputer implementations.
+A `Graph` in TinkerPop3 can support any number of `GraphComputer` implementations. Out of the box, Hadoop-Gremlin supports three GraphComputer implementations.
* <<giraphgraphcomputer,`GiraphGraphComputer`>>: Leverages Giraph to execute TinkerPop3 OLAP computations.
** The graph must fit within the total RAM of the Hadoop cluster (graph size restriction). Messages passing is coordinated via ZooKeeper for the in-memory graph (speedy traversals).
+* <<sparkgraphcomputer,`SparkGraphComputer`>>: Leverages Spark to execute TinkerPop3 OLAP computations.
+** The graph must fit within the total disk space of the cluster (supports massive graphs). Message passing is coordinated via Spark map/reduce operations on potentially in-memory data (middle speed traversals).
* <<mapreducegraphcomputer,`MapReduceGraphComputer`>>: Leverages Hadoop's MapReduce to execute TinkerPop3 OLAP computations. (*coming soon*)
** The graph must fit within the total disk space of the Hadoop cluster (supports massive graphs). Message passing is coordinated via MapReduce jobs over the on-disk graph (slow traversals).
+TIP: image:gremlin-sugar.png[width=50,float=left] For those wanting to use the <<sugar-plugin,SugarPlugin>> with their submitted traversal, do `:remote config useSugar true` as well as `:plugin use tinkerpop.sugar` at the start of the Gremlin Console session if it is not already activated.
+
[[giraphgraphcomputer]]
GiraphGraphComputer
^^^^^^^^^^^^^^^^^^^
-image:giraph-logo.png[width=100,float=left] link:http://giraph.apache.org[Giraph] is an Apache Foundation project focused on OLAP-based graph processing. Giraph makes use of the distributed graph computing paradigm made popular by Google's Pregel. In Giraph, developers write "vertex programs" that get executed at each vertex in parallel. These programs communicate with one another in a bulk synchronous parallel (BSP) manner. This model aligns with TinkerPop3's `GraphComputer` API. TinkerPop3 provides an implementation of `GraphComputer` that works for Giraph called `GiraphGraphComputer`. Moreover, with TinkerPop3's <<mapreduce,MapReduce>>-framework, the standard Giraph/Pregel model is extended to support an arbitrary number of MapReduce phases to aggregate and yield results from the graph. Below are examples using `GiraphGraphComputer` from the <<gremlin-console,Gremlin-Console>>.
+image:giraph-logo.png[width=100,float=left] link:http://giraph.apache.org[Giraph] is an Apache Software Foundation project focused on OLAP-based graph processing. Giraph makes use of the distributed graph computing paradigm made popular by Google's Pregel. In Giraph, developers write "vertex programs" that get executed at each vertex in parallel. These programs communicate with one another in a bulk synchronous parallel (BSP) manner. This model aligns with TinkerPop3's `GraphComputer` API. TinkerPop3 provides an implementation of `GraphComputer` that works for Giraph called `GiraphGraphComputer`. Moreover, with TinkerPop3's <<mapreduce,MapReduce>>-framework, the standard Giraph/Pregel model is extended to support an arbitrary number of MapReduce phases to aggregate and yield results from the graph. Below are examples using `GiraphGraphComputer` from the <<gremlin-console,Gremlin-Console>>.
IMPORTANT: Be sure that the link:http://www.slf4j.org/[SLF4J] of Hadoop matches that of Giraph or else there will be conflicts. Simply copy the following jars to the `lib/` of the machines in the Hadoop cluster: `slf4j-api-a.b.c.jar` and `slf4j-log4j12-a.b.c.jar`.
@@ -748,6 +752,8 @@ WARNING: Giraph uses a large number of Hadoop counters. The default for Hadoop i
WARNING: The maximum number of workers can be no larger than the number of map-slots in the Hadoop cluster minus 1. For example, if the Hadoop cluster has 4 map slots, then `giraph.maxWorkers` can not be larger than 3. One map-slot is reserved for the master compute node and all other slots can be allocated as workers to execute the VertexPrograms on the vertices of the graph.
+IMPORTANT: `GiraphGraphComputer` is the default graph computer for `HadoopGraph`. If another graph computer is desired, then it can be set using `HadoopGraph.compute(...)` or via the properties file `gremlin.hadoop.defaultGraphComputer` setting.
+
[source,text]
gremlin> g.engine(computer)
==>null
@@ -776,53 +782,58 @@ INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
IMPORTANT: The examples above do not use lambdas (i.e. closures in Gremlin-Groovy). This makes the traversal serializable and thus, able to be distributed to all machines in the Hadoop cluster. If a lambda is required in a traversal, then the traversal must be sent as a `String` and compiled locally at each machine in the cluster. The following example demonstrates the `:remote` command which allows for submitting Gremlin traversals as a `String`.
[source,text]
-gremlin> :remote connect tinkerpop.hadoop ../../../hadoop-gremlin/conf/hadoop-kryo.properties
-==>hadoopgraph[kryoinputformat->kryooutputformat]
-gremlin> :> g.V().group('a').by{it.value('name')[1]}.by('name')
+gremlin> :remote connect tinkerpop.hadoop ../../../hadoop-gremlin/conf/giraph-gryo.properties
+==>hadoopgraph[gryoinputformat->gryooutputformat[giraphgraphcomputer]]
+gremlin> :> g.V().group().by{it.value('name')[1]}.by('name')
INFO org.apache.tinkerpop.gremlin.hadoop.process.computer.giraph.GiraphGraphComputer - HadoopGremlin(Giraph): TraversalVertexProgram[GraphStep(vertex), GroupStep(lambda,value(name)), ComputerResultStep]
INFO org.apache.hadoop.mapred.JobClient - Running job: job_201407281259_0039
INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
...
-INFO org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph - HadoopGremlin: GroupStepMapReduce
+INFO org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph - HadoopGremlin: GroupMapReduce[~reducing]
INFO org.apache.hadoop.mapred.JobClient - Running job: job_201407281259_0040
INFO org.apache.hadoop.mapred.JobClient - map 0% reduce 0%
...
==>[a:[marko, vadas], e:[peter], i:[ripple], o:[lop, josh]]
gremlin> result
-==>result[hadoopgraph[kryoinputformat->kryooutputformat],memory[size:1]]
+==>result[hadoopgraph[gryoinputformat->gryooutputformat[giraphgraphcomputer]],memory[size:1]]
gremlin> result.memory.runtime
==>20356
gremlin> result.memory.keys()
-==>a
+==>~reducing
gremlin> result.memory.get('~reducing')
==>a={marko=1, vadas=1}
==>e={peter=1}
==>i={ripple=1}
==>o={lop=1, josh=1}
-TIP: image:gremlin-sugar.png[width=50,float=left] For those wanting to use the <<sugar-plugin,SugarPlugin>> with their submitted traversal, do `:remote config useSugar true` as well as `:plugin use tinkerpop.sugar` at the start of the Gremlin Console session if it is not already activated.
-
-The `TraversalSupplier1` class mentioned in the `hadoop-kryo.properties` file is distributed with Hadoop-Gremlin. It declares what Gremlin-Java traversal to execute on the loaded `HadoopGraph`. This is the means by which traversals can be programmatically executed in Hadoop-Gremlin.
-
-[source,java]
-----
-// gremlin.traversalVertexProgram.traversalSupplier.type=CLASS
-// gremlin.traversalVertexProgram.traversalSupplier.object=org.apache.tinkerpop.gremlin.hadoop.process.computer.example.TraversalSupplier1
-
-public class TraversalSupplier1 implements Supplier<Traversal> {
- public Traversal get() {
- return HadoopGraph.open().V().out().out().values("name");
- }
-}
-----
-
[[sparkgraphcomputer]]
SparkGraphComputer
^^^^^^^^^^^^^^^^^^
-image:spark-logo.png[width=100,float=left] link:http://spark.apache.org[Spark] is an Apache Foundation project that provides a in-memory and disk-based distributed computing via a paradigm similar to Hadoop's MapReduce though more expressive and easier to use.
+image:spark-logo.png[width=175,float=left] link:http://spark.apache.org[Spark] is an Apache Software Foundation project focused on general-purpose OLAP data processing. Spark provides a hybrid in-memory/disk-based distributed computing model that is similar to Hadoop's MapReduce model. Spark provides a fluent function chaining DSL that is arguably easier for developers to work with than native Hadoop MapReduce. While Spark has a shorter startup time between "jobs" (a scatter/gather-step), the actual message passing algorithm (as designed by TinkerPop) is less efficient than that of Giraph. For small graphs, Spark will typically be much faster than Giraph, but as the graph becomes larger, the Hadoop MapReduce startup time incurred by Giraph will amortize as more time is spent passing messages (i.e. traversers) between the vertices of the graph.
+[source,text]
+gremlin> :remote connect tinkerpop.hadoop ../../../hadoop-gremlin/conf/spark-gryo.properties
+==>hadoopgraph[gryoinputformat->gryooutputformat[sparkgraphcomputer]]
+gremlin> g.engine(computer)
+==>null
+gremlin> g.V().count()
+==>6
+gremlin> g.V().out().out().values('name')
+==>lop
+==>ripple
+gremlin> :> g.V().group().by{it.value('name')[1]}.by('name') <1>
+==>[a:[marko, vadas], e:[peter], i:[ripple], o:[josh, lop]]
+gremlin> result.memory.runtime
+==>2807 <2>
+gremlin> result.memory.get('~reducing')
+==>a={marko=1, vadas=1}
+==>e={peter=1}
+==>i={ripple=1}
+==>o={josh=1, lop=1}
+<1> If lambdas are leverages in the traversal, then use `:>` to submit the string representation of the traversal to the cluster.
+<2> `SparkGraphComputer` is nearly 10x faster than `GiraphGraphComputer` on such small graphs due to the start-up time costs of Hadoop's MapReduce engine.
[[mapreducegraphcomputer]]
MapReduceGraphComputer
@@ -880,14 +891,14 @@ image::pagerank-logo.png[width=300]
The classic link:http://en.wikipedia.org/wiki/PageRank[PageRank] centrality algorithm can be executed over the TinkerPop graph from the command line using `GiraphGraphComputer`.
-NOTE: The extension `ldjson` in `hadoop-graphson.properties` refers to link:http://en.wikipedia.org/wiki/Line_Delimited_JSON[line-delimitated JSON] which is the file format used by `GraphSONWriter` when writing an link:http://en.wikipedia.org/wiki/Adjacency_list[adjacency list] representation of a graph.
+NOTE: The extension `ldjson` in `giraph-graphson.properties` refers to link:http://en.wikipedia.org/wiki/Line_Delimited_JSON[line-delimitated JSON] which is the file format used by `GraphSONWriter` when writing an link:http://en.wikipedia.org/wiki/Adjacency_list[adjacency list] representation of a graph.
[source,text]
$ hadoop fs -copyFromLocal data/tinkerpop-modern-vertices.ldjson tinkerpop-modern-vertices.ldjson
$ hadoop fs -ls
Found 2 items
-rw-r--r-- 1 marko supergroup 2356 2014-07-28 13:00 /user/marko/tinkerpop-modern-vertices.ldjson
-$ hadoop jar target/hadoop-gremlin-x.y.z-job.jar org.apache.tinkerpop.gremlin.hadoop.process.computer.giraph.GiraphGraphComputer conf/hadoop-graphson.properties
+$ hadoop jar target/hadoop-gremlin-x.y.z-job.jar org.apache.tinkerpop.gremlin.hadoop.process.computer.giraph.GiraphGraphComputer conf/giraph-graphson.properties
14/07/29 12:08:27 INFO giraph.GiraphGraphComputer: HadoopGremlin(Giraph): PageRankVertexProgram[alpha=0.85,iterations=30]
14/07/29 12:08:28 INFO mapred.JobClient: Running job: job_201407281259_0041
14/07/29 12:08:29 INFO mapred.JobClient: map 0% reduce 0%
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d9bbc3aa/docs/static/images/tinkerpop3.graffle
----------------------------------------------------------------------
diff --git a/docs/static/images/tinkerpop3.graffle b/docs/static/images/tinkerpop3.graffle
index 7005a67..1c42aeb 100644
--- a/docs/static/images/tinkerpop3.graffle
+++ b/docs/static/images/tinkerpop3.graffle
@@ -20,7 +20,7 @@
<key>GuidesVisible</key>
<string>YES</string>
<key>ImageCounter</key>
- <integer>91</integer>
+ <integer>92</integer>
<key>Images</key>
<array>
<dict>
@@ -148438,7 +148438,7 @@
<key>MasterSheets</key>
<array/>
<key>ModificationDate</key>
- <string>2015-02-17 20:34:44 +0000</string>
+ <string>2015-03-04 15:40:19 +0000</string>
<key>Modifier</key>
<string>Marko Rodriguez</string>
<key>NotesVisible</key>
@@ -212521,8 +212521,8 @@ lang:java}</string>
<integer>154</integer>
<key>Points</key>
<array>
- <string>{411.12995833633278, 249.17537096233991}</string>
- <string>{393.34156685592797, 263.8246326463925}</string>
+ <string>{411.12995833633272, 249.17537096233991}</string>
+ <string>{393.34156685592802, 263.82463264639244}</string>
</array>
<key>Style</key>
<dict>
@@ -217507,8 +217507,8 @@ lang:java}</string>
<integer>142</integer>
<key>Points</key>
<array>
- <string>{432.31558180663797, 157.68958738721886}</string>
- <string>{428.44294795917932, 154.48274935358782}</string>
+ <string>{432.31558180663797, 157.68958738721889}</string>
+ <string>{428.44294795917938, 154.48274935358785}</string>
</array>
<key>Style</key>
<dict>
@@ -228423,8 +228423,8 @@ endTime:2001}</string>
<integer>48</integer>
<key>Points</key>
<array>
- <string>{207.56428733349856, 306.90079234700096}</string>
- <string>{275.56067070483152, 339.75387073893648}</string>
+ <string>{207.56428733349856, 306.90079234700102}</string>
+ <string>{275.56067070483158, 339.75387073893654}</string>
</array>
<key>Style</key>
<dict>
@@ -247345,8 +247345,8 @@ endTime:2001}</string>
<integer>833</integer>
<key>Points</key>
<array>
- <string>{105.52237945151377, 208.31001946013788}</string>
- <string>{105.63731914533909, 212.65601170076275}</string>
+ <string>{105.52240071844524, 208.31001890112481}</string>
+ <string>{105.63735579580813, 212.65601073725625}</string>
</array>
<key>Style</key>
<dict>
@@ -247378,8 +247378,8 @@ endTime:2001}</string>
<integer>834</integer>
<key>Points</key>
<array>
- <string>{93.331419456731211, 215.70136274987894}</string>
- <string>{98.654523410053059, 217.66216048998922}</string>
+ <string>{93.331422140760893, 215.7013554490882}</string>
+ <string>{98.654531470446258, 217.66213856496285}</string>
</array>
<key>Style</key>
<dict>
@@ -247411,8 +247411,8 @@ endTime:2001}</string>
<integer>835</integer>
<key>Points</key>
<array>
- <string>{89.348688709820706, 217.22528967048856}</string>
- <string>{84.798174448812304, 225.81886034058067}</string>
+ <string>{89.348688700496226, 217.225289665547}</string>
+ <string>{84.798174407471578, 225.8188603186718}</string>
</array>
<key>Style</key>
<dict>
@@ -247444,8 +247444,8 @@ endTime:2001}</string>
<integer>836</integer>
<key>Points</key>
<array>
- <string>{100.21217949109801, 238.89697616727426}</string>
- <string>{105.91488315206389, 245.27828295846902}</string>
+ <string>{100.21217967049391, 238.89697600666904}</string>
+ <string>{105.91488366893917, 245.27828249573341}</string>
</array>
<key>Style</key>
<dict>
@@ -247477,8 +247477,8 @@ endTime:2001}</string>
<integer>837</integer>
<key>Points</key>
<array>
- <string>{120.05191083455661, 243.60070136552093}</string>
- <string>{113.08573287867719, 246.78090416563759}</string>
+ <string>{120.05187007938419, 243.600612058406}</string>
+ <string>{113.08567167448749, 246.78077004274016}</string>
</array>
<key>Style</key>
<dict>
@@ -247510,8 +247510,8 @@ endTime:2001}</string>
<integer>838</integer>
<key>Points</key>
<array>
- <string>{130.57076938427414, 262.18132870953065}</string>
- <string>{125.0691361271717, 263.81735483429048}</string>
+ <string>{130.57075957553371, 262.18129561801635}</string>
+ <string>{125.06911394289449, 263.81727999881826}</string>
</array>
<key>Style</key>
<dict>
@@ -247543,8 +247543,8 @@ endTime:2001}</string>
<integer>839</integer>
<key>Points</key>
<array>
- <string>{116.15985269992821, 260.40897205417474}</string>
- <string>{111.32857134183654, 252.54707577965456}</string>
+ <string>{116.15985323772283, 260.40897172306654}</string>
+ <string>{111.32857270515926, 252.54707494028733}</string>
</array>
<key>Style</key>
<dict>
@@ -247576,8 +247576,8 @@ endTime:2001}</string>
<integer>840</integer>
<key>Points</key>
<array>
- <string>{107.28622204465285, 264.95973627320438}</string>
- <string>{108.48506036673759, 253.19631347057245}</string>
+ <string>{107.28622204223238, 264.95973627295916}</string>
+ <string>{108.48506035802596, 253.19631346968981}</string>
</array>
<key>Style</key>
<dict>
@@ -247609,8 +247609,8 @@ endTime:2001}</string>
<integer>841</integer>
<key>Points</key>
<array>
- <string>{108.63067847230108, 291.16607227595529}</string>
- <string>{114.28915485107261, 297.78906326590112}</string>
+ <string>{108.63067853287563, 291.16607222408868}</string>
+ <string>{114.28915499943878, 297.78906313886347}</string>
</array>
<key>Style</key>
<dict>
@@ -247642,8 +247642,8 @@ endTime:2001}</string>
<integer>842</integer>
<key>Points</key>
<array>
- <string>{118.22301182449368, 298.03661130080366}</string>
- <string>{123.67765954212852, 293.07493113723433}</string>
+ <string>{118.22300926356239, 298.03660848988955}</string>
+ <string>{123.67765031202353, 293.07492100615679}</string>
</array>
<key>Style</key>
<dict>
@@ -247675,8 +247675,8 @@ endTime:2001}</string>
<integer>843</integer>
<key>Points</key>
<array>
- <string>{105.45810644915345, 280.63059748035329}</string>
- <string>{106.27117659729893, 274.00116259288291}</string>
+ <string>{105.45810348181614, 280.6305971184841}</string>
+ <string>{106.27117033250194, 274.00116182888917}</string>
</array>
<key>Style</key>
<dict>
@@ -247708,8 +247708,8 @@ endTime:2001}</string>
<integer>844</integer>
<key>Points</key>
<array>
- <string>{102.04882568769949, 226.96187010866973}</string>
- <string>{99.433623098843242, 231.55146795024805}</string>
+ <string>{102.04876562027994, 226.96183583445892}</string>
+ <string>{99.43352159305293, 231.55141003326344}</string>
</array>
<key>Style</key>
<dict>
@@ -247741,8 +247741,8 @@ endTime:2001}</string>
<integer>845</integer>
<key>Points</key>
<array>
- <string>{93.502859994908277, 287.07466026066282}</string>
- <string>{98.721663477082302, 286.85226667263396}</string>
+ <string>{93.502859609034445, 287.0746511498624}</string>
+ <string>{98.721662379279508, 286.85224075390204}</string>
</array>
<key>Style</key>
<dict>
@@ -247774,8 +247774,8 @@ endTime:2001}</string>
<integer>846</integer>
<key>Points</key>
<array>
- <string>{159.43712091461484, 205.84707608815805}</string>
- <string>{162.35482513011996, 208.67404673064399}</string>
+ <string>{159.43729496946088, 205.84689654858235}</string>
+ <string>{162.35511682212422, 208.67374581586213}</string>
</array>
<key>Style</key>
<dict>
@@ -247807,8 +247807,8 @@ endTime:2001}</string>
<integer>847</integer>
<key>Points</key>
<array>
- <string>{161.23890682040462, 213.05817495286723}</string>
- <string>{155.15924305698198, 214.74889471750689}</string>
+ <string>{161.23890229300022, 213.05815862409349}</string>
+ <string>{155.15923222256782, 214.74885574971066}</string>
</array>
<key>Style</key>
<dict>
@@ -247840,8 +247840,8 @@ endTime:2001}</string>
<integer>848</integer>
<key>Points</key>
<array>
- <string>{162.56238927114782, 222.63526834459415}</string>
- <string>{166.19618461615192, 223.00637852976277}</string>
+ <string>{162.56243434615519, 222.634824447829}</string>
+ <string>{166.19628760632531, 223.00536197237886}</string>
</array>
<key>Style</key>
<dict>
@@ -247873,8 +247873,8 @@ endTime:2001}</string>
<integer>849</integer>
<key>Points</key>
<array>
- <string>{168.70275318803442, 215.18690523880827}</string>
- <string>{169.49316236870624, 216.04595218514208}</string>
+ <string>{168.74925612066465, 215.14346548963599}</string>
+ <string>{169.54979093889287, 215.98945594579453}</string>
</array>
<key>Style</key>
<dict>
@@ -247906,8 +247906,8 @@ endTime:2001}</string>
<integer>850</integer>
<key>Points</key>
<array>
- <string>{183.92146625364489, 233.84501262485873}</string>
- <string>{184.777966566103, 235.16743307822443}</string>
+ <string>{183.93591369453469, 233.83562032755631}</string>
+ <string>{184.79430344183157, 235.15662407834361}</string>
</array>
<key>Style</key>
<dict>
@@ -247939,8 +247939,8 @@ endTime:2001}</string>
<integer>851</integer>
<key>Points</key>
<array>
- <string>{175.24959919520211, 235.44090734624476}</string>
- <string>{174.40224352986786, 239.40576969836036}</string>
+ <string>{175.24920651962063, 235.44082373783573}</string>
+ <string>{174.40171267608611, 239.40565665957135}</string>
</array>
<key>Style</key>
<dict>
@@ -247972,8 +247972,8 @@ endTime:2001}</string>
<integer>852</integer>
<key>Points</key>
<array>
- <string>{173.92627177789848, 248.37919308570534}</string>
- <string>{174.71178935421494, 255.85018566583159}</string>
+ <string>{173.92627528653674, 248.37919271894123}</string>
+ <string>{174.71179865656063, 255.85018469343584}</string>
</array>
<key>Style</key>
<dict>
@@ -248425,8 +248425,8 @@ endTime:2001}</string>
<integer>865</integer>
<key>Points</key>
<array>
- <string>{162.31406158832175, 239.7518922385681}</string>
- <string>{170.4826332287312, 257.03200730505131}</string>
+ <string>{162.31406161294416, 239.75189222692973}</string>
+ <string>{170.482633331638, 257.03200725640994}</string>
</array>
<key>Style</key>
<dict>
@@ -248458,8 +248458,8 @@ endTime:2001}</string>
<integer>866</integer>
<key>Points</key>
<array>
- <string>{144.52538331936611, 248.24429401346507}</string>
- <string>{148.15560988561037, 252.00815094247929}</string>
+ <string>{144.52746492244782, 248.24228457467933}</string>
+ <string>{148.15840363261998, 252.00545136430679}</string>
</array>
<key>Style</key>
<dict>
@@ -248491,8 +248491,8 @@ endTime:2001}</string>
<integer>867</integer>
<key>Points</key>
<array>
- <string>{153.47501853752851, 265.16923519721695}</string>
- <string>{153.4751072648958, 274.70012851650972}</string>
+ <string>{153.47501853757723, 265.16923519721695}</string>
+ <string>{153.47510726500519, 274.70012851650972}</string>
</array>
<key>Style</key>
<dict>
@@ -248524,8 +248524,8 @@ endTime:2001}</string>
<integer>868</integer>
<key>Points</key>
<array>
- <string>{151.67980536602613, 286.4472708147365}</string>
- <string>{149.25206842164624, 294.2049461674938}</string>
+ <string>{151.67980349361781, 286.44727023050422}</string>
+ <string>{149.2520640173118, 294.20494479324282}</string>
</array>
<key>Style</key>
<dict>
@@ -248662,8 +248662,8 @@ endTime:2001}</string>
<integer>872</integer>
<key>Points</key>
<array>
- <string>{158.30365522338792, 284.29038361547123}</string>
- <string>{162.57808705578339, 287.45964032138727}</string>
+ <string>{158.3037578429381, 284.29024556592418}</string>
+ <string>{162.57828050926429, 287.45938005563374}</string>
</array>
<key>Style</key>
<dict>
@@ -248695,8 +248695,8 @@ endTime:2001}</string>
<integer>873</integer>
<key>Points</key>
<array>
- <string>{172.12163448781467, 281.36249514128082}</string>
- <string>{171.05553633207043, 284.724379697725}</string>
+ <string>{172.11695509488618, 281.36101442951093}</string>
+ <string>{171.04959919239056, 284.72249366343362}</string>
</array>
<key>Style</key>
<dict>
@@ -248761,8 +248761,8 @@ endTime:2001}</string>
<integer>875</integer>
<key>Points</key>
<array>
- <string>{143.63703744290132, 202.97567080265253}</string>
- <string>{149.14924218087543, 202.34814850206837}</string>
+ <string>{143.63685851384628, 202.97409113291181}</string>
+ <string>{149.14898665675139, 202.34589980758446}</string>
</array>
<key>Style</key>
<dict>
@@ -248794,8 +248794,8 @@ endTime:2001}</string>
<integer>876</integer>
<key>Points</key>
<array>
- <string>{112.29036360771379, 216.18333743864036}</string>
- <string>{119.58847876993072, 211.51577576641182}</string>
+ <string>{112.29025455584838, 216.18316729404324}</string>
+ <string>{119.58824633061367, 211.51541312654672}</string>
</array>
<key>Style</key>
<dict>
@@ -248827,8 +248827,8 @@ endTime:2001}</string>
<integer>877</integer>
<key>Points</key>
<array>
- <string>{120.86891188436712, 229.37880071606776}</string>
- <string>{112.39552765833507, 224.2654275976009}</string>
+ <string>{120.86899520559021, 229.37866289030109}</string>
+ <string>{112.39566501990872, 224.26520038734978}</string>
</array>
<key>Style</key>
<dict>
@@ -248860,8 +248860,8 @@ endTime:2001}</string>
<integer>878</integer>
<key>Points</key>
<array>
- <string>{82.683850107693956, 237.10127663201331}</string>
- <string>{83.628199369528375, 245.16711614258455}</string>
+ <string>{82.683862237556298, 237.1012752199446}</string>
+ <string>{83.628227888568802, 245.16711282256222}</string>
</array>
<key>Style</key>
<dict>
@@ -249614,7 +249614,7 @@ endTime:2001}</string>
<key>WindowInfo</key>
<dict>
<key>CurrentSheet</key>
- <integer>22</integer>
+ <integer>42</integer>
<key>ExpandedCanvases</key>
<array/>
<key>Frame</key>
@@ -249630,9 +249630,9 @@ endTime:2001}</string>
<key>Sidebar</key>
<true/>
<key>SidebarWidth</key>
- <integer>152</integer>
+ <integer>168</integer>
<key>VisibleRegion</key>
- <string>{{-153, -14}, {882, 761}}</string>
+ <string>{{-145, -14}, {866, 761}}</string>
<key>Zoom</key>
<real>1</real>
<key>ZoomValues</key>
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d9bbc3aa/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/traversal/step/map/ComputerResultStep.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/traversal/step/map/ComputerResultStep.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/traversal/step/map/ComputerResultStep.java
index d927892..f38193c 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/traversal/step/map/ComputerResultStep.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/traversal/step/map/ComputerResultStep.java
@@ -105,7 +105,7 @@ public final class ComputerResultStep<S> extends AbstractStep<S, S> {
}
this.traversers = IteratorUtils.of(this.getTraversal().getTraverserGenerator().generate((S) sideEffects, this, 1l));
}
- } else if (endStep instanceof ReducingBarrierStep) {
+ } else if (result.memory().exists(ReducingBarrierStep.REDUCING)) {
this.traversers = IteratorUtils.of(this.getTraversal().getTraverserGenerator().generate(result.memory().get(ReducingBarrierStep.REDUCING), this, 1l));
} else {
this.traversers = result.memory().get(TraverserMapReduce.TRAVERSERS);
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d9bbc3aa/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java
----------------------------------------------------------------------
diff --git a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java
index c91d445..4a2e33c 100644
--- a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java
+++ b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java
@@ -30,6 +30,7 @@ import org.apache.tinkerpop.gremlin.groovy.plugin.RemoteAcceptor;
import org.apache.tinkerpop.gremlin.hadoop.Constants;
import org.apache.tinkerpop.gremlin.hadoop.process.computer.giraph.GiraphGraphComputer;
import org.apache.tinkerpop.gremlin.hadoop.process.computer.mapreduce.MapReduceGraphComputer;
+import org.apache.tinkerpop.gremlin.hadoop.process.computer.spark.SparkGraphComputer;
import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration;
import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
import org.apache.tinkerpop.gremlin.hadoop.structure.hdfs.HDFSTools;
@@ -68,6 +69,7 @@ public class HadoopGremlinPlugin extends AbstractGremlinPlugin {
add(IMPORT_SPACE + HDFSTools.class.getPackage().getName() + DOT_STAR);
////
add(IMPORT_SPACE + GiraphGraphComputer.class.getPackage().getName() + DOT_STAR);
+ add(IMPORT_SPACE + SparkGraphComputer.class.getPackage().getName() + DOT_STAR);
add(IMPORT_SPACE + MapReduceGraphComputer.class.getPackage().getName() + DOT_STAR);
}};
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d9bbc3aa/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/HadoopGraph.java
----------------------------------------------------------------------
diff --git a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/HadoopGraph.java b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/HadoopGraph.java
index 990fc77..769679c 100644
--- a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/HadoopGraph.java
+++ b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/HadoopGraph.java
@@ -164,7 +164,7 @@ public class HadoopGraph implements Graph, Graph.Iterators {
@Override
public void compute(final Class<? extends GraphComputer> graphComputerClass) {
- if (!graphComputerClass.equals(GiraphGraphComputer.class) || !graphComputerClass.equals(SparkGraphComputer.class))
+ if (!graphComputerClass.equals(GiraphGraphComputer.class) && !graphComputerClass.equals(SparkGraphComputer.class))
throw Graph.Exceptions.graphDoesNotSupportProvidedGraphComputer(graphComputerClass);
this.graphComputerClass = graphComputerClass;
}
@@ -203,7 +203,7 @@ public class HadoopGraph implements Graph, Graph.Iterators {
final String toString = this.configuration.containsKey(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT) ?
hadoopConfiguration.getClass(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT, OutputFormat.class).getSimpleName() :
"no-output";
- return StringFactory.graphString(this, fromString.toLowerCase() + "->" + toString.toLowerCase());
+ return StringFactory.graphString(this, fromString.toLowerCase() + "->" + toString.toLowerCase() + "[" + this.graphComputerClass.getSimpleName().toLowerCase() + "]");
}
@Override
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d9bbc3aa/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 8acac72..5711729 100644
--- a/pom.xml
+++ b/pom.xml
@@ -633,6 +633,8 @@ limitations under the License.
<imagesdir>images</imagesdir>
<encoding>UTF-8</encoding>
<toc>true</toc>
+ <toclevels>3</toclevels>
+ <toc-position>left</toc-position>
<!--<iconsdir>images/icons</iconsdir>-->
<!-- AsciiDoctor CSS3-based theme configuration -->
<stylesdir>${asciidoctor.style.dir}</stylesdir>