You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by ok...@apache.org on 2015/10/07 16:23:22 UTC
[2/3] incubator-tinkerpop git commit: Updated Documentation to note
new parameter
Updated Documentation to note new parameter
Changed the underlying ascii to note the spark.persistContext parameter
and additionally made a new table for other Spark-Gremlin specific
properties
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/41aa9066
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/41aa9066
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/41aa9066
Branch: refs/heads/master
Commit: 41aa906602cb06eff72c861827873d48776e1dad
Parents: 9fb2539
Author: Russell Spitzer <Ru...@gmail.com>
Authored: Tue Oct 6 17:54:59 2015 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Wed Oct 7 08:23:12 2015 -0600
----------------------------------------------------------------------
docs/src/implementations.asciidoc | 18 +++++++++++++++++-
1 file changed, 17 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/41aa9066/docs/src/implementations.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/implementations.asciidoc b/docs/src/implementations.asciidoc
index 331ce4f..0c5aaa0 100644
--- a/docs/src/implementations.asciidoc
+++ b/docs/src/implementations.asciidoc
@@ -780,11 +780,17 @@ gremlin.hadoop.outputLocation=output
gremlin.hadoop.graphOutputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat
gremlin.hadoop.jarsInDistributedCache=true
####################################
-# SparkGraphComputer Configuration #
+# Spark Configuration #
####################################
spark.master=local[4]
spark.executor.memory=1g
spark.serializer=org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer
+####################################
+# SparkGraphComputer Configuration #
+####################################
+gremlin.hadoop.graphInputRDD=org.apache.tinkerpop.gremlin.spark.process.computer.io.InputRDDFormat
+gremlin.hadoop.graphOutputRDD=org.apache.tinkerpop.gremlin.spark.process.computer.io.OutputRDDFormat
+gremlin.spark.persistContext=true
#####################################
# GiraphGraphComputer Configuration #
#####################################
@@ -810,6 +816,8 @@ A review of the Hadoop-Gremlin specific properties are provided in the table bel
|gremlin.hadoop.jarsInDistributedCache |Whether to upload the Hadoop-Gremlin jars to a distributed cache (necessary if jars are not on the machines' classpaths).
|=========================================================
+
+
Along with the properties above, the numerous link:http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml[Hadoop specific properties] can be added as needed to tune and parameterize the executed Hadoop-Gremlin job on the respective Hadoop cluster.
IMPORTANT: As the size of the graphs being processed becomes large, it is important to fully understand how the underlying OLAP engine (e.g. Spark, Giraph, etc.) works and understand the numerous parameterizations offered by these systems. Such knowledge can help alleviate out of memory exceptions, slow load times, slow processing times, garbage collection issues, etc.
@@ -926,6 +934,14 @@ The `SparkGraphComputer` algorithm leverages Spark's caching abilities to reduce
image::spark-algorithm.png[width=775]
+Spark-Gremlin Provides specific properties in the table below.
+[width="100%",cols="2,10",options="header"]
+|========================================================
+|Property |Description
+|gremlin.hadoop.graphInputRDD |A class for creating RDD's from underlying graph data, defaults to Hadoop `InputFormat`
+|gremlin.hadoop.graphOutputRDD |A class for output RDD's, defaults to Hadoop `OutputFormat`
+|gremlin.spark.persistContext |Whether to create a new context for every SparkGraphComputer or to reuse an available SparkContext in the JVM (or start one if not running)
+
IMPORTANT: If the vendor/user wishes to bypass using Hadoop `InputFormats` for pulling data from the underlying graph system, it is possible to leverage Spark's RDD constructs directly. There is a `gremlin.hadoop.graphInputRDD` configuration that references a `Class<? extends InputRDD>`. An `InputRDD` provides a read method that takes a `SparkContext` and returns a graphRDD. Likewise, to bypass `OutputFormat`, use `gremlin.hadoop.graphOutputRDD` and the respective `OutputRDD` with its write-based method.
[[giraphgraphcomputer]]