You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by dk...@apache.org on 2015/06/04 12:15:57 UTC

[43/43] incubator-tinkerpop git commit: more work on preprocessor

more work on preprocessor


Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/e7b5e199
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/e7b5e199
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/e7b5e199

Branch: refs/heads/preprocessor
Commit: e7b5e1990a7f6724ce1417a7bb43e5d595082f28
Parents: 44b4949
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Thu Jun 4 02:45:50 2015 +0200
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Thu Jun 4 02:45:50 2015 +0200

----------------------------------------------------------------------
 docs/preprocessor/preprocess-file.sh | 4 ++--
 docs/src/implementations.asciidoc    | 8 ++++++++
 2 files changed, 10 insertions(+), 2 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/e7b5e199/docs/preprocessor/preprocess-file.sh
----------------------------------------------------------------------
diff --git a/docs/preprocessor/preprocess-file.sh b/docs/preprocessor/preprocess-file.sh
index 2bd66d3..9b8abac 100755
--- a/docs/preprocessor/preprocess-file.sh
+++ b/docs/preprocessor/preprocess-file.sh
@@ -38,8 +38,8 @@ if [ $(grep -c '^\[gremlin' ${input}) -gt 0 ]; then
   cat ${input}.part?.groovy > ${input}.groovy && rm -f ${input}.part?.groovy
   ec=${PIPESTATUS[0]}
   if [ ${ec} -eq 0 ]; then
-    HADOOP_GREMLIN_LIBS="${CONSOLE_HOME}/ext/hadoop-gremlin/lib" bin/gremlin.sh ${input}.groovy | awk 'BEGIN {b=1} /¶IGNORE/ {b=!b} !/¶IGNORE/ {if(a&&b)print} /¶START/ {a=1}' | grep -v '^WARN ' | sed 's/^==>¶//' > ${output}
-    ec=${PIPESTATUS[0]}
+    cat ${input}.groovy | HADOOP_GREMLIN_LIBS="${CONSOLE_HOME}/ext/hadoop-gremlin/lib" bin/gremlin.sh | grep -v '^gremlin> ' | awk 'BEGIN {i=0} /^==>¶IGNORE$/ {i=!i} /^==>¶END$/ {p=0} !/^==>¶IGNORE$/ {if(!i&&p)print} /^==>¶START$/ {p=1}' | grep -v '^WARN ' | sed 's/^==>¶//' > ${output}
+    ec=${PIPESTATUS[1]}
   fi
   rm -f ${input}.groovy
   popd > /dev/null

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/e7b5e199/docs/src/implementations.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/implementations.asciidoc b/docs/src/implementations.asciidoc
index 138a981..37f53ae 100644
--- a/docs/src/implementations.asciidoc
+++ b/docs/src/implementations.asciidoc
@@ -882,6 +882,14 @@ gremlin> :remote connect tinkerpop.hadoop graph g
 gremlin> :> g.V().group().by{it.value('name')[1]}.by('name')
 ==>[a:[marko, vadas], e:[peter], i:[ripple], o:[lop, josh]]
 
+[gremlin-groovy]
+----
+graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
+g = graph.traversal(computer(SparkGraphComputer))
+:remote connect tinkerpop.hadoop graph g
+:> g.V().group().by{it.value('name')[1]}.by('name')
+----
+
 The `SparkGraphComputer` algorithm leverages Spark's caching abilities to reduce the amount of data shuffled across the wire on each iteration of the <<vertexprogram,`VertexProgram`>>. When the graph is loaded as a Spark RDD (Resilient Distributed Dataset) it is immediately cached as `graphRDD`. The `graphRDD` is a distributed adjacency list which encodes the vertex, its properties, and all its incident edges. On the first iteration, each vertex (in parallel) is passed through `VertexProgram.execute()`. This yields an output of the vertex's mutated state (i.e. updated compute keys -- `propertyX`) and its outgoing messages. This `viewOutgoingRDD` is then reduced to `viewIncomingRDD` where the outgoing messages are sent to their respective vertices. If a `MessageCombiner` exists for the vertex program, then messages are aggregated locally and globally to ultimately yield one incoming message for the vertex. This reduce sequence is the "message pass." If the vertex program does not ter
 minate on this iteration, then the `viewIncomingRDD` is joined with the cached `graphRDD` and the process continues. When there are no more iterations, there is a final join and the resultant RDD is stripped of its edges and messages. This `mapReduceRDD` is cached and is processed by each <<mapreduce,`MapReduce`>> job in the <<graphcomputer,`GraphComputer`>> computation.
 
 image::spark-algorithm.png[width=775]