You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by ok...@apache.org on 2015/09/11 16:51:40 UTC
incubator-tinkerpop git commit: tweaks to preprocess-file.sh,
updates to Hadoop sections of docs,
fixed an awkwardness with System.getenv when you deploy via a standalone uber
jar.
Repository: incubator-tinkerpop
Updated Branches:
refs/heads/master e34fc1a4f -> a03151aa8
tweaks to preprocess-file.sh, updates to Hadoop sections of docs, fixed an awkwardness with System.getenv when you deploy via a standalone uber jar.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/a03151aa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/a03151aa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/a03151aa
Branch: refs/heads/master
Commit: a03151aa8ea2144fe35cd948dee9830a0e7aa37b
Parents: e34fc1a
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Fri Sep 11 08:52:05 2015 -0600
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Fri Sep 11 08:52:05 2015 -0600
----------------------------------------------------------------------
docs/preprocessor/preprocess-file.sh | 2 +-
docs/src/implementations.asciidoc | 242 ++++++++++++-------
docs/src/index.asciidoc | 2 +-
docs/src/the-traversal.asciidoc | 2 +-
.../process/computer/GiraphGraphComputer.java | 2 +-
.../groovy/plugin/HadoopGremlinPlugin.java | 3 +-
.../process/computer/SparkGraphComputer.java | 2 +-
7 files changed, 157 insertions(+), 98 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/a03151aa/docs/preprocessor/preprocess-file.sh
----------------------------------------------------------------------
diff --git a/docs/preprocessor/preprocess-file.sh b/docs/preprocessor/preprocess-file.sh
index 7a7cfd1..263a788 100755
--- a/docs/preprocessor/preprocess-file.sh
+++ b/docs/preprocessor/preprocess-file.sh
@@ -58,7 +58,7 @@ if [ $(grep -c '^\[gremlin' ${input}) -gt 0 ]; then
awk -f ${AWK_SCRIPTS}/prepare.awk ${input} |
awk -f ${AWK_SCRIPTS}/init-code-blocks.awk |
- awk -f ${AWK_SCRIPTS}/progressbar.awk -v tpl=${AWK_SCRIPTS}/progressbar.groovy.template | HADOOP_GREMLIN_LIBS="${CONSOLE_HOME}/ext/hadoop-gremlin/lib" bin/gremlin.sh |
+ awk -f ${AWK_SCRIPTS}/progressbar.awk -v tpl=${AWK_SCRIPTS}/progressbar.groovy.template | HADOOP_GREMLIN_LIBS="${CONSOLE_HOME}/ext/giraph-gremlin/lib" bin/gremlin.sh |
${lb} awk -f ${AWK_SCRIPTS}/ignore.awk |
${lb} awk -f ${AWK_SCRIPTS}/prettify.awk |
${lb} awk -f ${AWK_SCRIPTS}/cleanup.awk > ${output}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/a03151aa/docs/src/implementations.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/implementations.asciidoc b/docs/src/implementations.asciidoc
index f694639..8e7369d 100644
--- a/docs/src/implementations.asciidoc
+++ b/docs/src/implementations.asciidoc
@@ -723,17 +723,17 @@ Hadoop-Gremlin
</dependency>
----
-image:hadoop-logo-notext.png[width=100,float=left] link:http://hadoop.apache.org/[Hadoop] is a distributed computing framework that is used to process data represented across a multi-machine compute cluster. When the data in the Hadoop cluster represents a TinkerPop3 graph, then Hadoop-Gremlin can be used to process the graph using TinkerPop3's OLTP and OLAP models of graph computing.
+image:hadoop-logo-notext.png[width=100,float=left] link:http://hadoop.apache.org/[Hadoop] is a distributed computing framework that is used to process data represented across a multi-machine compute cluster. When the data in the Hadoop cluster represents a TinkerPop3 graph, then Hadoop-Gremlin can be used to process the graph using both TinkerPop3's OLTP and OLAP graph computing models.
IMPORTANT: This section assumes that the user has a Hadoop 2.x cluster functioning. For more information on getting started with Hadoop, please see the link:http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/SingleCluster.html[Single Node Setup] tutorial. Moreover, if using `GiraphGraphComputer` or `SparkGraphComputer` it is advisable that the reader also familiarize their self with Giraph (link:http://giraph.apache.org/quick_start.html[Getting Started]) and Spark (link:http://spark.apache.org/docs/latest/quick-start.html[Quick Start]).
Installing Hadoop-Gremlin
~~~~~~~~~~~~~~~~~~~~~~~~~
-The `HADOOP_GREMLIN_LIBS` references locations that contains jars that should be uploaded to the distributed cache of Hadoop. Note that `HADOOP_GREMLIN_LIBS` can be a colon-separated (`:`) list of locations and thus will load all jars into the cluster at all provided locations.
+The `HADOOP_GREMLIN_LIBS` references locations that contains jars that should be uploaded to a respective distributed cache (link:http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html[YARN] or SparkServer). Note that the locations in `HADOOP_GREMLIN_LIBS` can be a colon-separated (`:`) and all jars from all locations will be loaded into the cluster. Typically, only the jars of the respective GraphComputer are required to be loaded (e.g. `GiraphGraphComputer` plugin lib directory).
[source,shell]
-export HADOOP_GREMLIN_LIBS=/usr/local/gremlin-console/ext/hadoop-gremlin/lib
+export HADOOP_GREMLIN_LIBS=/usr/local/gremlin-console/ext/giraph-gremlin/lib
If using <<gremlin-console,Gremlin Console>>, it is important to install the Hadoop-Gremlin plugin. Note that Hadoop-Gremlin requires a Gremlin Console restart after installing.
@@ -766,15 +766,15 @@ gremlin>
Properties Files
~~~~~~~~~~~~~~~~
-`HadoopGraph` makes heavy use of properties files which ultimately get turned into Apache configurations and Hadoop configurations. The example properties file presented below is located at `conf/hadoop/hadoop-gryo.properties`.
+`HadoopGraph` makes use of properties files which ultimately get turned into Apache configurations and/or Hadoop configurations. The example properties file presented below is located at `conf/hadoop/hadoop-gryo.properties`.
[source,text]
gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
+gremlin.hadoop.inputLocation=tinkerpop-modern.kryo
gremlin.hadoop.graphInputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat
+gremlin.hadoop.outputLocation=output
gremlin.hadoop.graphOutputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat
gremlin.hadoop.jarsInDistributedCache=true
-gremlin.hadoop.inputLocation=tinkerpop-modern.kryo
-gremlin.hadoop.outputLocation=output
####################################
# SparkGraphComputer Configuration #
####################################
@@ -804,7 +804,7 @@ A review of the Hadoop-Gremlin specific properties are provided in the table bel
|gremlin.hadoop.graphInputFormat |The format that the graph input file(s) are represented in.
|gremlin.hadoop.outputLocation |The location to write the computed HadoopGraph to.
|gremlin.hadoop.graphOutputFormat |The format that the output file(s) should be represented in.
-|gremlin.hadoop.jarsInDistributedCache |Whether to upload the Hadoop-Gremlin jars to Hadoop's distributed cache (necessary if jars are not on machines' classpaths).
+|gremlin.hadoop.jarsInDistributedCache |Whether to upload the Hadoop-Gremlin jars to a distributed cache (necessary if jars are not on the machines' classpaths).
|=========================================================
Along with the properties above, the numerous link:http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml[Hadoop specific properties] can be added as needed to tune and parameterize the executed Hadoop-Gremlin job on the respective Hadoop cluster.
@@ -814,9 +814,9 @@ IMPORTANT: As the size of the graphs being processed becomes large, it is import
OLTP Hadoop-Gremlin
~~~~~~~~~~~~~~~~~~~
-image:hadoop-pipes.png[width=180,float=left] It is possible to execute OLTP operations over a `HadoopGraph`. However, realize that the underlying HDFS files are typically not random access and thus, to retrieve a vertex, a linear scan is required. OLTP operations are useful for peeking at the graph prior to executing a long running OLAP job -- e.g. `g.V().valueMap().limit(10)`.
+image:hadoop-pipes.png[width=180,float=left] It is possible to execute OLTP operations over a `HadoopGraph`. However, realize that the underlying HDFS files are not random access and thus, to retrieve a vertex, a linear scan is required. OLTP operations are useful for peeking into the graph prior to executing a long running OLAP job -- e.g. `g.V().valueMap().limit(10)`.
-CAUTION: OLTP operations on `HadoopGraph` are not efficient. They require linear scans to execute and are unreasonable for large graphs. In such large graph situations, make use of <<traversalvertexprogram,TraversalVertexProgram>> which is the OLAP implementation of the Gremlin language. Hadoop-Gremlin provides various `GraphComputer` implementations to execute OLAP computations over a `HadoopGraph`.
+CAUTION: OLTP operations on `HadoopGraph` are not efficient. They require linear scans to execute and are unreasonable for large graphs. In such large graph situations, make use of <<traversalvertexprogram,TraversalVertexProgram>> which is the OLAP Gremlin machine.
[gremlin-groovy]
----
@@ -836,11 +836,11 @@ image:hadoop-furnace.png[width=180,float=left] Hadoop-Gremlin was designed to ex
A `Graph` in TinkerPop3 can support any number of `GraphComputer` implementations. Out of the box, Hadoop-Gremlin supports the following three implementations.
-* <<mapreducegraphcomputer,`MapReduceGraphComputer`>>: Leverages Hadoop's MapReduce to execute TinkerPop3 OLAP computations. (*coming soon*)
+* <<mapreducegraphcomputer,`MapReduceGraphComputer`>>: Leverages Hadoop's MapReduce engine to execute TinkerPop3 OLAP computations. (*coming soon*)
** The graph must fit within the total disk space of the Hadoop cluster (supports massive graphs). Message passing is coordinated via MapReduce jobs over the on-disk graph (slow traversals).
-* <<sparkgraphcomputer,`SparkGraphComputer`>>: Leverages Spark to execute TinkerPop3 OLAP computations.
+* <<sparkgraphcomputer,`SparkGraphComputer`>>: Leverages Apache Spark to execute TinkerPop3 OLAP computations.
** The graph may fit within the total RAM of the cluster (supports larger graphs). Message passing is coordinated via Spark map/reduce/join operations on in-memory and disk-cached data (average speed traversals).
-* <<giraphgraphcomputer,`GiraphGraphComputer`>>: Leverages Giraph to execute TinkerPop3 OLAP computations.
+* <<giraphgraphcomputer,`GiraphGraphComputer`>>: Leverages Apache Giraph to execute TinkerPop3 OLAP computations.
** The graph should fit within the total RAM of the Hadoop cluster (graph size restriction), though "out-of-core" processing is possible. Message passing is coordinated via ZooKeeper for the in-memory graph (speedy traversals).
TIP: image:gremlin-sugar.png[width=50,float=left] For those wanting to use the <<sugar-plugin,SugarPlugin>> with their submitted traversal, do `:remote config useSugar true` as well as `:plugin use tinkerpop.sugar` at the start of the Gremlin Console session if it is not already activated.
@@ -890,7 +890,16 @@ MapReduceGraphComputer
SparkGraphComputer
^^^^^^^^^^^^^^^^^^
-image:spark-logo.png[width=175,float=left] link:http://spark.apache.org[Spark] is an Apache Software Foundation project focused on general-purpose OLAP data processing. Spark provides a hybrid in-memory/disk-based distributed computing model that is similar to Hadoop's MapReduce model. Spark maintains a fluent function chaining DSL that is arguably easier for developers to work with than native Hadoop MapReduce. While Spark has a shorter startup time between "jobs" (a scatter/gather-step), the actual message passing algorithm (as designed by TinkerPop) is less efficient than that of Giraph. For small graphs, Spark will typically be much faster than Giraph, but as the graph becomes larger, the Hadoop MapReduce startup time incurred by Giraph will amortize as more time is spent passing messages (i.e. traversers) between the vertices of the graph.
+[source,xml]
+----
+<dependency>
+ <groupId>org.apache.tinkerpop</groupId>
+ <artifactId>spark-gremlin</artifactId>
+ <version>x.y.z</version>
+</dependency>
+----
+
+image:spark-logo.png[width=175,float=left] link:http://spark.apache.org[Spark] is an Apache Software Foundation project focused on general-purpose OLAP data processing. Spark provides a hybrid in-memory/disk-based distributed computing model that is similar to Hadoop's MapReduce model. Spark maintains a fluent function chaining DSL that is arguably easier for developers to work with than native Hadoop MapReduce. Spark-Gremlin provides an implementation of the bulk-synchronous parallel, distributed message passing algorithm within Spark and thus, any `VertexProgram` can be executed over `SparkGraphComputer`.
[gremlin-groovy]
----
@@ -920,11 +929,20 @@ IMPORTANT: If the vendor/user wishes to bypass using Hadoop `InputFormats` for p
GiraphGraphComputer
^^^^^^^^^^^^^^^^^^^
+[source,xml]
+----
+<dependency>
+ <groupId>org.apache.tinkerpop</groupId>
+ <artifactId>giraph-gremlin</artifactId>
+ <version>x.y.z</version>
+</dependency>
+----
+
image:giraph-logo.png[width=100,float=left] link:http://giraph.apache.org[Giraph] is an Apache Software Foundation project focused on OLAP-based graph processing. Giraph makes use of the distributed graph computing paradigm made popular by Google's Pregel. In Giraph, developers write "vertex programs" that get executed at each vertex in parallel. These programs communicate with one another in a bulk synchronous parallel (BSP) manner. This model aligns with TinkerPop3's `GraphComputer` API. TinkerPop3 provides an implementation of `GraphComputer` that works for Giraph called `GiraphGraphComputer`. Moreover, with TinkerPop3's <<mapreduce,MapReduce>>-framework, the standard Giraph/Pregel model is extended to support an arbitrary number of MapReduce phases to aggregate and yield results from the graph. Below are examples using `GiraphGraphComputer` from the <<gremlin-console,Gremlin-Console>>.
WARNING: Be sure that the link:http://www.slf4j.org/[SLF4J] of Hadoop matches that of Giraph or else there will be conflicts. Simply copy the following jars to the `lib/` of the machines in the Hadoop cluster: `slf4j-api-a.b.c.jar` and `slf4j-log4j12-a.b.c.jar`.
-WARNING: Giraph uses a large number of Hadoop counters. The default for Hadoop is 120. In `mapred-site.xml` it is possible to increase the limit it via the `mapreduce.job.counters.limit` property. A good value to use is 1000. This is a cluster-wide property so be sure to restart the cluster after updating.
+WARNING: Giraph uses a large number of Hadoop counters. The default for Hadoop is 120. In `mapred-site.xml` it is possible to increase the limit it via the `mapreduce.job.counters.max` property. A good value to use is 1000. This is a cluster-wide property so be sure to restart the cluster after updating.
WARNING: The maximum number of workers can be no larger than the number of map-slots in the Hadoop cluster minus 1. For example, if the Hadoop cluster has 4 map slots, then `giraph.maxWorkers` can not be larger than 3. One map-slot is reserved for the master compute node and all other slots can be allocated as workers to execute the VertexPrograms on the vertices of the graph.
@@ -951,7 +969,8 @@ IMPORTANT: The examples above do not use lambdas (i.e. closures in Gremlin-Groov
[gremlin-groovy]
----
graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
-:remote connect tinkerpop.hadoop graph
+g = graph.traversal(computer(GiraphGraphComputer))
+:remote connect tinkerpop.hadoop graph g
:> g.V().group().by{it.value('name')[1]}.by('name')
result
result.memory.runtime
@@ -1086,7 +1105,8 @@ The distributed file system of Hadoop is called link:http://en.wikipedia.org/wik
[gremlin-groovy]
----
graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
-:remote connect tinkerpop.hadoop graph
+g = graph.traversal(computer(SparkGraphComputer))
+:remote connect tinkerpop.hadoop graph g
:> g.V().group().by{it.value('name')[1]}.by('name')
hdfs.ls()
hdfs.ls('output')
@@ -1119,90 +1139,130 @@ image::pagerank-logo.png[width=300]
The classic link:http://en.wikipedia.org/wiki/PageRank[PageRank] centrality algorithm can be executed over the TinkerPop graph from the command line using `GiraphGraphComputer`.
+WARNING: Be sure that the `HADOOP_GREMLIN_LIBS` references the location `lib` directory of the respective `GraphComputer` engine being used or else the requisite dependencies will not be uploaded to the Hadoop cluster.
+
[source,text]
-$ hadoop fs -copyFromLocal data/tinkerpop-modern.json tinkerpop-modern.json
-$ hadoop fs -ls
+----
+$ hdfs dfs -copyFromLocal data/tinkerpop-modern.json tinkerpop-modern.json
+$ hdfs dfs -ls
Found 2 items
-rw-r--r-- 1 marko supergroup 2356 2014-07-28 13:00 /user/marko/tinkerpop-modern.json
-$ hadoop jar target/giraph-gremlin-x.y.z-job.jar org.apache.tinkerpop.gremlin.giraph.process.computer.giraph.GiraphGraphComputer conf/hadoop-graphson.properties
-14/07/29 12:08:27 INFO giraph.GiraphGraphComputer: HadoopGremlin(Giraph): PageRankVertexProgram[alpha=0.85,iterations=30]
-14/07/29 12:08:28 INFO mapred.JobClient: Running job: job_201407281259_0041
-14/07/29 12:08:29 INFO mapred.JobClient: map 0% reduce 0%
-14/07/29 12:08:51 INFO mapred.JobClient: map 66% reduce 0%
-14/07/29 12:08:52 INFO mapred.JobClient: map 100% reduce 0%
-14/07/29 12:08:54 INFO mapred.JobClient: Job complete: job_201407281259_0041
-14/07/29 12:08:54 INFO mapred.JobClient: Counters: 57
-14/07/29 12:08:54 INFO mapred.JobClient: Map-Reduce Framework
-14/07/29 12:08:54 INFO mapred.JobClient: Spilled Records=0
-14/07/29 12:08:54 INFO mapred.JobClient: Map input records=3
-14/07/29 12:08:54 INFO mapred.JobClient: SPLIT_RAW_BYTES=132
-14/07/29 12:08:54 INFO mapred.JobClient: Map output records=0
-14/07/29 12:08:54 INFO mapred.JobClient: Total committed heap usage (bytes)=347602944
-14/07/29 12:08:54 INFO mapred.JobClient: Giraph Timers
-14/07/29 12:08:54 INFO mapred.JobClient: Shutdown (milliseconds)=385
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 1 (milliseconds)=89
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 23 (milliseconds)=28
-14/07/29 12:08:54 INFO mapred.JobClient: Input superstep (milliseconds)=1127
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 27 (milliseconds)=30
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 10 (milliseconds)=34
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 5 (milliseconds)=43
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 22 (milliseconds)=31
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 14 (milliseconds)=35
-14/07/29 12:08:54 INFO mapred.JobClient: Total (milliseconds)=4023
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 2 (milliseconds)=50
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 18 (milliseconds)=29
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 11 (milliseconds)=35
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 24 (milliseconds)=32
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 28 (milliseconds)=32
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 15 (milliseconds)=34
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 6 (milliseconds)=37
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 19 (milliseconds)=31
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 25 (milliseconds)=27
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 8 (milliseconds)=33
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 12 (milliseconds)=44
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 20 (milliseconds)=31
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 16 (milliseconds)=31
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 9 (milliseconds)=36
-14/07/29 12:08:54 INFO mapred.JobClient: Setup (milliseconds)=1119
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 3 (milliseconds)=50
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 7 (milliseconds)=38
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 13 (milliseconds)=36
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 29 (milliseconds)=37
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 26 (milliseconds)=40
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 0 (milliseconds)=293
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 21 (milliseconds)=46
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 17 (milliseconds)=32
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep 4 (milliseconds)=39
-14/07/29 12:08:54 INFO mapred.JobClient: File Input Format Counters
-14/07/29 12:08:54 INFO mapred.JobClient: Bytes Read=0
-14/07/29 12:08:54 INFO mapred.JobClient: Giraph Stats
-14/07/29 12:08:54 INFO mapred.JobClient: Aggregate finished vertices=0
-14/07/29 12:08:54 INFO mapred.JobClient: Aggregate edges=0
-14/07/29 12:08:54 INFO mapred.JobClient: Sent messages=6
-14/07/29 12:08:54 INFO mapred.JobClient: Current workers=2
-14/07/29 12:08:54 INFO mapred.JobClient: Last checkpointed superstep=0
-14/07/29 12:08:54 INFO mapred.JobClient: Current master task partition=0
-14/07/29 12:08:54 INFO mapred.JobClient: Superstep=30
-14/07/29 12:08:54 INFO mapred.JobClient: Aggregate vertices=6
-14/07/29 12:08:54 INFO mapred.JobClient: FileSystemCounters
-14/07/29 12:08:54 INFO mapred.JobClient: HDFS_BYTES_READ=2488
-14/07/29 12:08:54 INFO mapred.JobClient: FILE_BYTES_WRITTEN=250470
-14/07/29 12:08:54 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=2719
-14/07/29 12:08:54 INFO mapred.JobClient: Job Counters
-14/07/29 12:08:54 INFO mapred.JobClient: Launched map tasks=3
-14/07/29 12:08:54 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
-14/07/29 12:08:54 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
-14/07/29 12:08:54 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=31907
-14/07/29 12:08:54 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
-14/07/29 12:08:54 INFO mapred.JobClient: File Output Format Counters
-14/07/29 12:08:54 INFO mapred.JobClient: Bytes Written=0
-$ hadoop fs -cat output/~g/*
+$ hadoop jar target/giraph-gremlin-x.y.z-job.jar org.apache.tinkerpop.gremlin.giraph.process.computer.GiraphGraphComputer ../hadoop-gremlin/conf/hadoop-graphson.properties
+15/09/11 08:02:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
+15/09/11 08:02:11 INFO computer.GiraphGraphComputer: HadoopGremlin(Giraph): PageRankVertexProgram[alpha=0.85,iterations=30]
+15/09/11 08:02:11 INFO job.GiraphJob: run: Since checkpointing is disabled (default), do not allow any task retries (setting mapred.map.max.attempts = 0, old value = 4)
+15/09/11 08:02:11 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
+15/09/11 08:02:12 INFO mapreduce.JobSubmitter: number of splits:3
+15/09/11 08:02:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1441915907347_0028
+15/09/11 08:02:12 INFO impl.YarnClientImpl: Submitted application application_1441915907347_0028
+15/09/11 08:02:12 INFO mapreduce.Job: The url to track the job: http://markos-macbook:8088/proxy/application_1441915907347_0028/
+15/09/11 08:02:12 INFO job.GiraphJob: Tracking URL: http://markos-macbook:8088/proxy/application_1441915907347_0028/
+15/09/11 08:02:12 INFO job.GiraphJob: Waiting for resources... Job will start only when it gets all 3 mappers
+15/09/11 08:03:54 INFO mapreduce.Job: Running job: job_1441915907347_0028
+15/09/11 08:03:55 INFO mapreduce.Job: Job job_1441915907347_0028 running in uber mode : false
+15/09/11 08:03:55 INFO mapreduce.Job: map 33% reduce 0%
+15/09/11 08:03:57 INFO mapreduce.Job: map 67% reduce 0%
+15/09/11 08:04:01 INFO mapreduce.Job: map 100% reduce 0%
+15/09/11 08:06:17 INFO mapreduce.Job: Job job_1441915907347_0028 completed successfully
+15/09/11 08:06:17 INFO mapreduce.Job: Counters: 80
+ File System Counters
+ FILE: Number of bytes read=0
+ FILE: Number of bytes written=483918
+ FILE: Number of read operations=0
+ FILE: Number of large read operations=0
+ FILE: Number of write operations=0
+ HDFS: Number of bytes read=1465
+ HDFS: Number of bytes written=1760
+ HDFS: Number of read operations=39
+ HDFS: Number of large read operations=0
+ HDFS: Number of write operations=20
+ Job Counters
+ Launched map tasks=3
+ Other local map tasks=3
+ Total time spent by all maps in occupied slots (ms)=458105
+ Total time spent by all reduces in occupied slots (ms)=0
+ Total time spent by all map tasks (ms)=458105
+ Total vcore-seconds taken by all map tasks=458105
+ Total megabyte-seconds taken by all map tasks=469099520
+ Map-Reduce Framework
+ Map input records=3
+ Map output records=0
+ Input split bytes=132
+ Spilled Records=0
+ Failed Shuffles=0
+ Merged Map outputs=0
+ GC time elapsed (ms)=1594
+ CPU time spent (ms)=0
+ Physical memory (bytes) snapshot=0
+ Virtual memory (bytes) snapshot=0
+ Total committed heap usage (bytes)=527958016
+ Giraph Stats
+ Aggregate edges=0
+ Aggregate finished vertices=0
+ Aggregate sent message message bytes=13535
+ Aggregate sent messages=186
+ Aggregate vertices=6
+ Current master task partition=0
+ Current workers=2
+ Last checkpointed superstep=0
+ Sent message bytes=438
+ Sent messages=6
+ Superstep=31
+ Giraph Timers
+ Initialize (ms)=2996
+ Input superstep (ms)=5209
+ Setup (ms)=59
+ Shutdown (ms)=9324
+ Superstep 0 GiraphComputation (ms)=3861
+ Superstep 1 GiraphComputation (ms)=4027
+ Superstep 10 GiraphComputation (ms)=4000
+ Superstep 11 GiraphComputation (ms)=4004
+ Superstep 12 GiraphComputation (ms)=3999
+ Superstep 13 GiraphComputation (ms)=4000
+ Superstep 14 GiraphComputation (ms)=4005
+ Superstep 15 GiraphComputation (ms)=4003
+ Superstep 16 GiraphComputation (ms)=4001
+ Superstep 17 GiraphComputation (ms)=4007
+ Superstep 18 GiraphComputation (ms)=3998
+ Superstep 19 GiraphComputation (ms)=4006
+ Superstep 2 GiraphComputation (ms)=4007
+ Superstep 20 GiraphComputation (ms)=3996
+ Superstep 21 GiraphComputation (ms)=4006
+ Superstep 22 GiraphComputation (ms)=4002
+ Superstep 23 GiraphComputation (ms)=3998
+ Superstep 24 GiraphComputation (ms)=4003
+ Superstep 25 GiraphComputation (ms)=4001
+ Superstep 26 GiraphComputation (ms)=4003
+ Superstep 27 GiraphComputation (ms)=4005
+ Superstep 28 GiraphComputation (ms)=4002
+ Superstep 29 GiraphComputation (ms)=4001
+ Superstep 3 GiraphComputation (ms)=3988
+ Superstep 30 GiraphComputation (ms)=4248
+ Superstep 4 GiraphComputation (ms)=4010
+ Superstep 5 GiraphComputation (ms)=3998
+ Superstep 6 GiraphComputation (ms)=3996
+ Superstep 7 GiraphComputation (ms)=4005
+ Superstep 8 GiraphComputation (ms)=4009
+ Superstep 9 GiraphComputation (ms)=3994
+ Total (ms)=138788
+ Zookeeper base path
+ /_hadoopBsp/job_1441915907347_0028=0
+ Zookeeper halt node
+ /_hadoopBsp/job_1441915907347_0028/_haltComputation=0
+ Zookeeper server:port
+ localhost:22181=0
+ File Input Format Counters
+ Bytes Read=0
+ File Output Format Counters
+ Bytes Written=0
+$ hdfs dfs -cat output/~g/*
{"id":1,"label":"person","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":39,"value":0.15000000000000002}],"name":[{"id":0,"value":"marko"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":10,"value":3.0}],"age":[{"id":1,"value":29}]}}
{"id":5,"label":"software","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":35,"value":0.23181250000000003}],"name":[{"id":8,"value":"ripple"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":6,"value":0.0}],"lang":[{"id":9,"value":"java"}]}}
{"id":3,"label":"software","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":39,"value":0.4018125}],"name":[{"id":4,"value":"lop"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":10,"value":0.0}],"lang":[{"id":5,"value":"java"}]}}
{"id":4,"label":"person","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":39,"value":0.19250000000000003}],"name":[{"id":6,"value":"josh"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":10,"value":2.0}],"age":[{"id":7,"value":32}]}}
{"id":2,"label":"person","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":35,"value":0.19250000000000003}],"name":[{"id":2,"value":"vadas"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":6,"value":0.0}],"age":[{"id":3,"value":27}]}}
{"id":6,"label":"person","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":35,"value":0.15000000000000002}],"name":[{"id":10,"value":"peter"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":6,"value":1.0}],"age":[{"id":11,"value":35}]}}
+----
Vertex 4 ("josh") is isolated below:
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/a03151aa/docs/src/index.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/index.asciidoc b/docs/src/index.asciidoc
index 4ed383d..5a01291 100644
--- a/docs/src/index.asciidoc
+++ b/docs/src/index.asciidoc
@@ -16,7 +16,7 @@ limitations under the License.
////
image::apache-tinkerpop-logo.png[width=500]
-x.y.z
+*x.y.z*
:toc-position: left
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/a03151aa/docs/src/the-traversal.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/the-traversal.asciidoc b/docs/src/the-traversal.asciidoc
index 08fcac7..120615c 100644
--- a/docs/src/the-traversal.asciidoc
+++ b/docs/src/the-traversal.asciidoc
@@ -1898,7 +1898,7 @@ Other `Graph` implementations, such as Neo4j, generate element identifiers autom
graph = Neo4jGraph.open('/tmp/neo4j')
strategy = ElementIdStrategy.build().create()
g = GraphTraversalSource.build().with(strategy).create(graph)
-// g.addV().property(id, '42a').id()
+g.addV().property(id, '42a').id()
----
IMPORTANT: The key that is used to store the assigned identifier should be indexed in the underlying graph database. If it is not indexed, then lookups for the elements that use these identifiers will perform a linear scan.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/a03151aa/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
----------------------------------------------------------------------
diff --git a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
index 48c4ab9..bdea4f1 100644
--- a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
+++ b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
@@ -187,7 +187,7 @@ public final class GiraphGraphComputer extends AbstractHadoopGraphComputer imple
private void loadJars(final FileSystem fs) {
final String hadoopGremlinLibsRemote = "hadoop-gremlin-libs";
if (this.giraphConfiguration.getBoolean(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, true)) {
- final String hadoopGremlinLocalLibs = System.getProperty(Constants.HADOOP_GREMLIN_LIBS);
+ final String hadoopGremlinLocalLibs = null == System.getProperty(Constants.HADOOP_GREMLIN_LIBS) ? System.getenv(Constants.HADOOP_GREMLIN_LIBS) : System.getProperty(Constants.HADOOP_GREMLIN_LIBS);
if (null == hadoopGremlinLocalLibs)
this.logger.warn(Constants.HADOOP_GREMLIN_LIBS + " is not set -- proceeding regardless");
else {
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/a03151aa/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java
----------------------------------------------------------------------
diff --git a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java
index 97bbdb7..5b0e609 100644
--- a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java
+++ b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java
@@ -97,8 +97,7 @@ public final class HadoopGremlinPlugin extends AbstractGremlinPlugin {
if (null == System.getenv(Constants.HADOOP_GREMLIN_LIBS))
HadoopGraph.LOGGER.warn("Be sure to set the environmental variable: " + Constants.HADOOP_GREMLIN_LIBS);
else {
- System.setProperty(Constants.HADOOP_GREMLIN_LIBS, System.getenv(Constants.HADOOP_GREMLIN_LIBS));
- HadoopGraph.LOGGER.info(Constants.HADOOP_GREMLIN_LIBS + " is set to: " + System.getProperty(Constants.HADOOP_GREMLIN_LIBS));
+ HadoopGraph.LOGGER.info(Constants.HADOOP_GREMLIN_LIBS + " is set to: " + System.getenv(Constants.HADOOP_GREMLIN_LIBS));
}
} catch (final Exception e) {
throw new PluginInitializationException(e.getMessage(), e);
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/a03151aa/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
index 4c9cec3..be5c19e 100644
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
@@ -194,7 +194,7 @@ public final class SparkGraphComputer extends AbstractHadoopGraphComputer {
private void loadJars(final JavaSparkContext sparkContext, final Configuration hadoopConfiguration) {
if (hadoopConfiguration.getBoolean(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, true)) {
- final String hadoopGremlinLocalLibs = System.getProperty(Constants.HADOOP_GREMLIN_LIBS);
+ final String hadoopGremlinLocalLibs = null == System.getProperty(Constants.HADOOP_GREMLIN_LIBS) ? System.getenv(Constants.HADOOP_GREMLIN_LIBS) : System.getProperty(Constants.HADOOP_GREMLIN_LIBS);
if (null == hadoopGremlinLocalLibs)
this.logger.warn(Constants.HADOOP_GREMLIN_LIBS + " is not set -- proceeding regardless");
else {