You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by sp...@apache.org on 2015/11/22 06:59:16 UTC
[01/22] incubator-tinkerpop git commit: Added comments to pom around
asciidoc generation.
Repository: incubator-tinkerpop
Updated Branches:
refs/heads/master 4a995c2ee -> 2fa626482
Added comments to pom around asciidoc generation.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/79b05567
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/79b05567
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/79b05567
Branch: refs/heads/master
Commit: 79b055678012c5b8ec41d75fc50b0866542ea303
Parents: d4de86b
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Fri Nov 20 05:54:32 2015 -0500
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Fri Nov 20 05:54:32 2015 -0500
----------------------------------------------------------------------
pom.xml | 21 ++++++++++++++++-----
1 file changed, 16 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/79b05567/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 96b70e3..4f29a70 100644
--- a/pom.xml
+++ b/pom.xml
@@ -584,10 +584,19 @@ limitations under the License.
</activation>
<properties>
+ <!-- the source asciidocs are copied from /docs to asciidoc.source.dir by bin/process-docs.sh
+ where they can the scripts embedded in the text are then executed and their results shoved
+ back into the doc -->
<asciidoc.source.dir>${project.basedir}/target/postprocess-asciidoc</asciidoc.source.dir>
+
+ <!-- once code processing is over, the documents are basically ready for the formatter and are
+ copied by way of the maven-resources-plugin to asciidoc.input.dir where they will be processed
+ by the asciidoctor plugin -->
<asciidoc.input.dir>${project.basedir}/target/doc-source</asciidoc.input.dir>
- <asciidoctor.style.dir>${asciidoc.input.dir}/stylesheets</asciidoctor.style.dir>
+
+ <!-- the output directory for the asciidoctor plugin - finalized documents can be found here -->
<htmlsingle.output.dir>${project.basedir}/target/docs/htmlsingle</htmlsingle.output.dir>
+ <asciidoctor.style.dir>${asciidoc.input.dir}/stylesheets</asciidoctor.style.dir>
</properties>
<build>
@@ -610,6 +619,7 @@ limitations under the License.
<artifactId>maven-resources-plugin</artifactId>
<inherited>false</inherited>
<executions>
+ <!-- copies docs already handled by the preprocessor to their staging area for asciidoc formatting -->
<execution>
<id>copy-docs-to-work-area</id>
<phase>generate-resources</phase>
@@ -626,6 +636,7 @@ limitations under the License.
</resources>
</configuration>
</execution>
+ <!-- copies images/resources to staging area for asciidoc formatting -->
<execution>
<id>copy-static-assets-to-htmlsingle</id>
<phase>process-resources</phase>
@@ -681,7 +692,7 @@ limitations under the License.
<inherited>false</inherited>
<executions>
<execution>
- <id>asciidoc-to-docbook</id>
+ <id>reference-book</id>
<phase>generate-resources</phase>
<goals>
<goal>process-asciidoc</goal>
@@ -708,7 +719,7 @@ limitations under the License.
</configuration>
</execution>
<execution>
- <id>asciidoc-to-upgrade-book</id>
+ <id>upgrade-book</id>
<phase>generate-resources</phase>
<goals>
<goal>process-asciidoc</goal>
@@ -735,7 +746,7 @@ limitations under the License.
</configuration>
</execution>
<execution>
- <id>asciidoc-to-developer-book</id>
+ <id>developer-book</id>
<phase>generate-resources</phase>
<goals>
<goal>process-asciidoc</goal>
@@ -762,7 +773,7 @@ limitations under the License.
</configuration>
</execution>
<execution>
- <id>asciidoc-to-tutorials-book</id>
+ <id>tutorial-getting-started</id>
<phase>generate-resources</phase>
<goals>
<goal>process-asciidoc</goal>
[19/22] incubator-tinkerpop git commit: removed the maven replacer
plugin; version numbers are now being replaced by the AsciiDocs post-processor
Posted by sp...@apache.org.
removed the maven replacer plugin; version numbers are now being replaced by the AsciiDocs post-processor
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/cc0c2baf
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/cc0c2baf
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/cc0c2baf
Branch: refs/heads/master
Commit: cc0c2baff2bef1518f5ffb3e69a2d30304bd8be2
Parents: 3b7d7c9
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Fri Nov 20 22:11:18 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Fri Nov 20 22:11:18 2015 +0100
----------------------------------------------------------------------
docs/postprocessor/postprocess.sh | 13 +++++++++----
pom.xml | 28 ----------------------------
2 files changed, 9 insertions(+), 32 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cc0c2baf/docs/postprocessor/postprocess.sh
----------------------------------------------------------------------
diff --git a/docs/postprocessor/postprocess.sh b/docs/postprocessor/postprocess.sh
index fd14f4c..cef9b93 100755
--- a/docs/postprocessor/postprocess.sh
+++ b/docs/postprocessor/postprocess.sh
@@ -20,10 +20,15 @@
pushd "$(dirname $0)/../.." > /dev/null
-if [ -f "target/docs/htmlsingle/index.html" ]; then
- awk -f "docs/postprocessor/processor.awk" target/docs/htmlsingle/index.html \
- | perl -0777 -pe 's/<span class="comment">\/\*\n \*\/<\/span>//igs' \
- > /tmp/index.html && mv /tmp/index.html target/docs/htmlsingle/
+TP_VERSION=$(cat pom.xml | grep -A1 '<artifactId>tinkerpop</artifactId>' | grep -o 'version>[^<]*' | grep -o '>.*' | cut -d '>' -f2 | head -n1)
+
+if [ -d "target/docs" ]; then
+ find target/docs -name index.html | while read file ; do
+ awk -f "docs/postprocessor/processor.awk" "${file}" \
+ | perl -0777 -pe 's/<span class="comment">\/\*\n \*\/<\/span>//igs' \
+ | sed "s/x\.y\.z/${TP_VERSION}/" \
+ > "${file}.tmp" && mv "${file}.tmp" "${file}"
+ done
fi
popd > /dev/null
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cc0c2baf/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index c098598..0ed7b98 100644
--- a/pom.xml
+++ b/pom.xml
@@ -658,34 +658,6 @@ limitations under the License.
</execution>
</executions>
</plugin>
- <!-- replaces x.y.z in asciidoc with the tinkerpop version from the pom. using an attribute
- in the asciidoctor-maven-plugin doesn't work properly in all cases for some reasons,
- specifically in the code examples -->
- <plugin>
- <groupId>com.google.code.maven-replacer-plugin</groupId>
- <artifactId>replacer</artifactId>
- <version>1.5.3</version>
- <inherited>false</inherited>
- <executions>
- <execution>
- <phase>generate-resources</phase>
- <goals>
- <goal>replace</goal>
- </goals>
- </execution>
- </executions>
- <configuration>
- <includes>
- <include>${asciidoc.input.dir}/*.asciidoc</include>
- </includes>
- <replacements>
- <replacement>
- <token>x\.y\.z</token>
- <value>${project.version}</value>
- </replacement>
- </replacements>
- </configuration>
- </plugin>
<plugin>
<groupId>org.asciidoctor</groupId>
<artifactId>asciidoctor-maven-plugin</artifactId>
[22/22] incubator-tinkerpop git commit: Update changelog.
Posted by sp...@apache.org.
Update changelog.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/2fa62648
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/2fa62648
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/2fa62648
Branch: refs/heads/master
Commit: 2fa6264821bee0eda9df93d7dccca71b77e66284
Parents: bbae082
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Sun Nov 22 00:58:47 2015 -0500
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Sun Nov 22 00:58:47 2015 -0500
----------------------------------------------------------------------
CHANGELOG.asciidoc | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/2fa62648/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index 5f3018a..5122f74 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -26,6 +26,7 @@ image::https://raw.githubusercontent.com/apache/incubator-tinkerpop/master/docs/
TinkerPop 3.1.1 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Split TinkerPop documentation into different directories.
* Added `explain()`-step which yields a `TraversalExplanation` with a pretty `toString()` detailing the compilation process.
* Fixed a traversal strategy ordering bug in `AdjacentToIncidentStrategy` and `IncidentToAdjacentStrategy`.
* Made a number of changes to improve traversal startup and execution performance.
[14/22] incubator-tinkerpop git commit: Made subdirectories for
various "books" in the docs.
Posted by sp...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/developer-release.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/developer-release.asciidoc b/docs/src/developer-release.asciidoc
deleted file mode 100644
index a5ddcc8..0000000
--- a/docs/src/developer-release.asciidoc
+++ /dev/null
@@ -1,184 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-Release Process
-===============
-
-This document describes the steps required to release a version of TinkerPop. The release is handled by a "release
-manager" (a committer fulfills this role), who ensures that the steps in this document are executed. The process is
-multi-phased and can therefore take several weeks to complete given the time needed for Apache voting and community
-feedback. Once a release point has been identified, the following phases represent the flow of "release":
-
-* Pre-flight check.
-* Optionally, produce a release candidate for community feedback.
-* Submit the official release for PMC vote.
-* Submit the official release for Incubator vote.
-* Release and promote.
-
-NOTE: It might be helpful to use this document as generated from the currently release as opposed to one generate
-from a previous version or from recent `SNAPSHOT`. When using one generated for release, all the "versions" in the
-commands end up being set to the version that is being released, making cut and paste of those commands less labor
-intensive and error prone.
-
-Pre-flight Check
-----------------
-
-The "pre-flight check" is a list of things performed by the release manager during the weeks leading up to a scheduled
-day to release. These checks will help to ensure that that release day goes smoothly by identifying problems up early
-and communicating with other members of the community.
-
-. Fourteen days before release, issue an email to the dev mailing list to remind the community of the pending release.
-.. Note any important issues open in JIRA in that post.
-.. Request review and update of the "upgrade documentation" and CHANGELOG.
-. Seven days before release, announce the code freeze on the dev mailing list to remind the community that the branch
-under release is protected. Tweaks to documentation and other odds and ends related to release are still allowed
-during this period.
-. At some point during the week:
-.. Run the full integration test suite: `mvn clean install -DskipIntegrationTests=false -DincludeNeo4j`
-.. Deploy a final SNAPSHOT to the snapshot repository.
-.. Review LICENSE and NOTICE files to make sure that no <<dependencies,changes are needed>>.
-.. Review javadoc filters on the "Core API" docs to be sure nothing needs to change.
-.. Review JIRA tickets in the release and ensure that:
-... All tickets categorized by having a "Component" assigned.
-... All tickets are either of type "Bug" or "Enhancement".
-. When all documentation changes are in place, use `bin/publish-docs.sh` to deploy a final `SNAPSHOT` representation
-of the docs and thus validate that there are no issues with the documentation generation process. Request review
-of the published documentation on the dev mailing list.
-
-Release Candidate
------------------
-
-A release candidate is an unofficial release that is represented by a tagged version in the Git repository. It is
-offered in cases where there is significant change in a particular version and the potential for upgrades and problems
-might be high.
-
-. `mvn clean install -DincludeNeo4j`
-.. `mvn verify -DskipIntegrationTests=false -DincludeNeo4j`
-.. `mvn verify -DskipPerformanceTests=false`
-. `bin/publish-docs.sh <username>` - note that under a release candidate the documentation is published as SNAPSHOT
-. `mvn versions:set -DnewVersion=x.y.z -DgenerateBackupPoms=false` to update the project files to reference a non-SNAPSHOT version
-. `git diff` and review the updated files (expect all `pom.xml` files and this README)
-. `git commit -a -m "TinkerPop x.y.z release"` and `git push`
-. `git tag -a -m "TinkerPop x.y.z release" x.y.z` and `git push --tags`
-. `mvn clean install -Dmaven.test.skip=true`
-. `mvn versions:set -DnewVersion=x.y.z-SNAPSHOT -DgenerateBackupPoms=false` to go back to SNAPSHOT
-. `git commit -a -m "Returned to x.y.z-SNAPSHOT"` and `git push`
-. Announce the release candidate to `dev` mailing list and await feedback
-. Repeat as required or proceed to the next phase
-
-PMC Vote
---------
-
-A positive vote for a particular release from the TinkerPop PMC is required to move to the following phase.
-
-. By this point, the testing performed during the code freeze should have validated the release. If however there
-are additional tests to perform that the release manager feels are relevant, they should be performed now. In other
-words, there is no need to rebuild the `SNAPSHOT` yet another time unless there are circumstances that would call its
-validity into question.
-. Update `CHANGELOG.asciidoc`:
-.. Update the release date
-.. Generate the JIRA release notes report for the current version and append them to the `CHANGELOG.asciidoc`.
-... Use an "advanced" search to filter out JIRA issues already released on other versions. For example: `fixVersion
-= 3.1.0-incubating AND fixVersion not in (3.0.2-incubating, 3.0.1-incubating`.
-... Consider use of an "Excel" export to organize, sort and prepare the JIRA tickets to be pasted to `CHANGELOG.asciidoc`
-... Be sure to include a link to other versions in the `CHANGELOG.asciidoc` that were previously released while the
-current release was under development as this new release will have those changes included within it. Please see
-3.1.0-incubating for an example.
-.. Organize "breaking" changes to be clearly marked (use JIRA and the "breaking" label to identify those)
-. Update "upgrade documentation":
-.. Update the release date.
-.. Update the link to CHANGELOG.asciidoc
-. `mvn versions:set -DnewVersion=x.y.z -DgenerateBackupPoms=false` to update project files to reference the non-SNAPSHOT version
-. `git diff` and review the updated files (expect all `pom.xml` files and this README)
-. `git commit -a -m "TinkerPop x.y.z release"` and `git push`
-. `mvn clean install -Dmaven.test.skip=true` - need to build first so that the right version of the console is used with `bin/publish-docs.sh`
-. `bin/process-docs.sh` and validate the generated documentation locally
-. `bin/publish-docs.sh <username>` - Note that this step requires no additional processing as the previous step.
-handled document generation and this step now merely needs to upload what was generated.
-. `mvn deploy -Papache-release -DcreateChecksum=true -Dmaven.test.skip=true` - deploy signed artifacts with checksums to link:https://repository.apache.org/[Apache Nexus]. Review (artifacts versions, file sizes, anything that might be out of place - request another committer to review as well) but do NOT close/release the staging repository at this time.
-. Review generated artifacts to be sure they have both javadocs and asciidocs present
-. Upload artifacts to `https://dist.apache.org/repos/dist/dev/incubator/tinkerpop` for `[VOTE]` review.
-.. `svn co --depth empty https://dist.apache.org/repos/dist/dev/incubator/tinkerpop/ dev` and `mkdir dev/x.y.z`
-.. `cp ~/.m2/repository/org/apache/tinkerpop/gremlin-console/x.y.z/gremlin-console-x.y.z-distribution.zip* dev/x.y.z`
-.. `cp ~/.m2/repository/org/apache/tinkerpop/gremlin-server/x.y.z/gremlin-server-x.y.z-distribution.zip* dev/x.y.z`
-.. `cp ~/.m2/repository/org/apache/tinkerpop/tinkerpop/x.y.z/tinkerpop-x.y.z-source-release.zip* dev/x.y.z`
-.. `cd dev/x.y.z`
-.. pass:[<code>ls * | xargs -n1 -I {} echo "mv apache-{} {}" | sed -e 's/distribution/bin/' -e 's/source-release/src/' -e s'/^\(.*\) \(.*\) \(.*\)$/\1 \3 \2/' | /bin/bash</code>]
-.. `cd ..; svn add x.y.z/; svn ci -m "TinkerPop x.y.z release"`
-. Execute `bin/validate-distribution.sh` and any other relevant testing.
-. `git tag -a -m "TinkerPop x.y.z release" x.y.z` and `git push --tags`
-. Perform JIRA administration tasks:
-.. "Release" the current version and set the "release date"
-.. If there is to be a follow on release in the current line of code, create that new version specifying the "start date"
-. Submit for `[VOTE]` at `dev@tinkerpop.incubator.apache.org` (see email template below)
-. *Wait for vote acceptance* (72 hours)
-
-Incubator Vote
---------------
-
-A positive vote for a particular release from the Apache Incubator is required to move to the following phase.
-
-. Submit for `[VOTE]` at `general@incubator.apache.org` (see email template below)
-.. Include the vote tally: "Apache TinkerPop (http://tinkerpop.incubator.apache.org/) would like to release TinkerPop x.y.z. We had a dev@ VOTE which resulted in a tally of +1 (3), 0 (0), and -1 (0). We now present our artifacts for vote by Incubator."
-. *Wait for vote acceptance* (72 hours)
-
-Release & Promote
------------------
-
-. Close the staging repository at link:https://repository.apache.org/[Apache Nexus]) and then release.
-. `svn co --depth empty https://dist.apache.org/repos/dist/dev/incubator/tinkerpop dev; svn up dev/x.y.z`
-. `svn co --depth empty https://dist.apache.org/repos/dist/release/incubator/tinkerpop release; mkdir release/x.y.z`
-. `cd release; svn add x.y.z/; svn ci -m "TinkerPop x.y.z release"`
-. Update homepage with references to latest distribution and to other internal links elsewhere on the page.
-. Wait for Apache Central to sync the jars and src (link:http://repo1.maven.org/maven2/org/apache/tinkerpop/tinkerpop/[http://repo1.maven.org/maven2/org/apache/tinkerpop/tinkerpop/]).
-. Announce release on `dev@`/`gremlin-users@` mailing lists and tweet from `@apachetinkerpop`
-
-Example `[VOTE]` email:
-
-```
-[VOTE] TinkerPop x.y.z Release
-
-Hello,
-
-The release artifacts can be found at this location:
- https://dist.apache.org/repos/dist/dev/incubator/tinkerpop/x.y.z/
-
-The source distribution is provided by:
- apache-tinkerpop-x.y.z-src.zip
-
-Two binary distributions are provided for user convenience:
- apache-gremlin-console-x.y.z-bin.zip
- apache-gremlin-server-x.y.z-bin.zip
-
-The online docs can be found here:
- http://tinkerpop.incubator.apache.org/docs/x.y.z/ (user docs)
- http://tinkerpop.incubator.apache.org/docs/x.y.z/upgrade.html (upgrade docs)
- http://tinkerpop.incubator.apache.org/javadocs/x.y.z/core/ (core javadoc)
- http://tinkerpop.incubator.apache.org/javadocs/x.y.z/full/ (full javadoc)
-
-The tag in Apache Git can be found here:
- https://git-wip-us.apache.org/repos/asf?p=incubator-tinkerpop.git;...
-
-The release notes are available here:
- https://github.com/apache/incubator-tinkerpop/blob/master/CHANGELOG.asciidoc#...
-
-The [VOTE] will be open for the next 72 hours --- closing <DayOfTheWeek> (<Month> <Day> <Year>) at <Time> <TimeZone>.
-
-My vote is +1.
-
-Thank you very much,
-<TinkerPop Committer Name>
-```
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/developer.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/developer.asciidoc b/docs/src/developer.asciidoc
deleted file mode 100644
index b5b9648..0000000
--- a/docs/src/developer.asciidoc
+++ /dev/null
@@ -1,33 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-image::apache-tinkerpop-logo.png[width=500]
-
-:toc-position: left
-
-Developer Documentation
-=======================
-
-This document contains information for TinkerPop developers, contributors, and community members. It focuses on
-technical information and other internal processes related to the project.
-
-include::developer-contributing.asciidoc[]
-
-include::developer-release.asciidoc[]
-
-include::developer-administration.asciidoc[]
-
-include::developer-meetings.asciidoc[]
\ No newline at end of file
[12/22] incubator-tinkerpop git commit: Made subdirectories for
various "books" in the docs.
Posted by sp...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/implementations.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/implementations.asciidoc b/docs/src/implementations.asciidoc
deleted file mode 100644
index b5c2c4d..0000000
--- a/docs/src/implementations.asciidoc
+++ /dev/null
@@ -1,1764 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-[[implementations]]
-Implementations
-===============
-
-image::gremlin-racecar.png[width=325]
-
-[[graph-system-provider-requirements]]
-Graph System Provider Requirements
-----------------------------------
-
-image:tinkerpop-enabled.png[width=140,float=left] At the core of TinkerPop3 is a Java8 API. The implementation of this
-core API and its validation via the `gremlin-test` suite is all that is required of a graph system provider wishing to
-provide a TinkerPop3-enabled graph engine. Once a graph system has a valid implementation, then all the applications
-provided by TinkerPop (e.g. Gremlin Console, Gremlin Server, etc.) and 3rd-party developers (e.g. Gremlin-Scala,
-Gremlin-JS, etc.) will integrate properly. Finally, please feel free to use the logo on the left to promote your
-TinkerPop3 implementation.
-
-Implementing Gremlin-Core
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The classes that a graph system provider should focus on implementing are itemized below. It is a good idea to study
-the <<tinkergraph-gremlin,TinkerGraph>> (in-memory OLTP and OLAP in `tinkergraph-gremlin`), <<neo4j-gremlin,Neo4jGraph>>
-(OTLP w/ transactions in `neo4j-gremlin`) and/or <<hadoop-gremlin,HadoopGraph>> (OLAP in `hadoop-gremlin`)
-implementations for ideas and patterns.
-
-. Online Transactional Processing Graph Systems (*OLTP*)
- .. Structure API: `Graph`, `Element`, `Vertex`, `Edge`, `Property` and `Transaction` (if transactions are supported).
- .. Process API: `TraversalStrategy` instances for optimizing Gremlin traversals to the provider's graph system (i.e. `TinkerGraphStepStrategy`).
-. Online Analytics Processing Graph Systems (*OLAP*)
- .. Everything required of OTLP is required of OLAP (but not vice versa).
- .. GraphComputer API: `GraphComputer`, `Messenger`, `Memory`.
-
-Please consider the following implementation notes:
-
-* Be sure your `Graph` implementation is named as `XXXGraph` (e.g. TinkerGraph, Neo4jGraph, HadoopGraph, etc.).
-* Use `StringHelper` to ensuring that the `toString()` representation of classes are consistent with other implementations.
-* Ensure that your implementation's `Features` (Graph, Vertex, etc.) are correct so that test cases handle particulars accordingly.
-* Use the numerous static method helper classes such as `ElementHelper`, `GraphComputerHelper`, `VertexProgramHelper`, etc.
-* There are a number of default methods on the provided interfaces that are semantically correct. However, if they are
-not efficient for the implementation, override them.
-* Implement the `structure/` package interfaces first and then, if desired, interfaces in the `process/` package interfaces.
-* `ComputerGraph` is a `Wrapper` system that ensure proper semantics during a GraphComputer computation.
-
-[[oltp-implementations]]
-OLTP Implementations
-^^^^^^^^^^^^^^^^^^^^
-
-image:pipes-character-1.png[width=110,float=right] The most important interfaces to implement are in the `structure/`
-package. These include interfaces like Graph, Vertex, Edge, Property, Transaction, etc. The `StructureStandardSuite`
-will ensure that the semantics of the methods implemented are correct. Moreover, there are numerous `Exceptions`
-classes with static exceptions that should be thrown by the graph system so that all the exceptions and their
-messages are consistent amongst all TinkerPop3 implementations.
-
-[[olap-implementations]]
-OLAP Implementations
-^^^^^^^^^^^^^^^^^^^^
-
-image:furnace-character-1.png[width=110,float=right] Implementing the OLAP interfaces may be a bit more complicated.
-Note that before OLAP interfaces are implemented, it is necessary for the OLTP interfaces to be, at minimal,
-implemented as specified in <<oltp-implementations,OLTP Implementations>>. A summary of each required interface
-implementation is presented below:
-
-. `GraphComputer`: A fluent builder for specifying an isolation level, a VertexProgram, and any number of MapReduce jobs to be submitted.
-. `Memory`: A global blackboard for ANDing, ORing, INCRing, and SETing values for specified keys.
-. `Messenger`: The system that collects and distributes messages being propagated by vertices executing the VertexProgram application.
-. `MapReduce.MapEmitter`: The system that collects key/value pairs being emitted by the MapReduce applications map-phase.
-. `MapReduce.ReduceEmitter`: The system that collects key/value pairs being emitted by the MapReduce applications combine- and reduce-phases.
-
-NOTE: The VertexProgram and MapReduce interfaces in the `process/computer/` package are not required by the graph
-system. Instead, these are interfaces to be implemented by application developers writing VertexPrograms and MapReduce jobs.
-
-IMPORTANT: TinkerPop3 provides three OLAP implementations: <<tinkergraph-gremlin,TinkerGraphComputer>> (TinkerGraph),
-<<giraphgraphcomputer,GiraphGraphComputer>> (HadoopGraph), and <<sparkgraphcomputer,`SparkGraphComputer`>> (Hadoop).
-Given the complexity of the OLAP system, it is good to study and copy many of the patterns used in these reference
-implementations.
-
-Implementing GraphComputer
-++++++++++++++++++++++++++
-
-image:furnace-character-3.png[width=150,float=right] The most complex method in GraphComputer is the `submit()`-method. The method must do the following:
-
-. Ensure the the GraphComputer has not already been executed.
-. Ensure that at least there is a VertexProgram or 1 MapReduce job.
-. If there is a VertexProgram, validate that it can execute on the GraphComputer given the respectively defined features.
-. Create the Memory to be used for the computation.
-. Execute the VertexProgram.setup() method once and only once.
-. Execute the VertexProgram.execute() method for each vertex.
-. Execute the VertexProgram.terminate() method once and if true, repeat VertexProgram.execute().
-. When VertexProgram.terminate() returns true, move to MapReduce job execution.
-. MapReduce jobs are not required to be executed in any specified order.
-. For each Vertex, execute MapReduce.map(). Then (if defined) execute MapReduce.combine() and MapReduce.reduce().
-. Update Memory with runtime information.
-. Construct a new `ComputerResult` containing the compute Graph and Memory.
-
-Implementing Memory
-+++++++++++++++++++
-
-image:gremlin-brain.png[width=175,float=left] The Memory object is initially defined by `VertexProgram.setup()`.
-The memory data is available in the first round of the `VertexProgram.execute()` method. Each Vertex, when executing
-the VertexProgram, can update the Memory in its round. However, the update is not seen by the other vertices until
-the next round. At the end of the first round, all the updates are aggregated and the new memory data is available
-on the second round. This process repeats until the VertexProgram terminates.
-
-Implementing Messenger
-++++++++++++++++++++++
-
-The Messenger object is similar to the Memory object in that a vertex can read and write to the Messenger. However,
-the data it reads are the messages sent to the vertex in the previous step and the data it writes are the messages
-that will be readable by the receiving vertices in the subsequent round.
-
-Implementing MapReduce Emitters
-+++++++++++++++++++++++++++++++
-
-image:hadoop-logo-notext.png[width=150,float=left] The MapReduce framework in TinkerPop3 is similar to the model
-popularized by link:http://apache.hadoop.org[Hadoop]. The primary difference is that all Mappers process the vertices
-of the graph, not an arbitrary key/value pair. However, the vertices' edges can not be accessed -- only their
-properties. This greatly reduces the amount of data needed to be pushed through the MapReduce engine as any edge
-information required, can be computed in the VertexProgram.execute() method. Moreover, at this stage, vertices can
-not be mutated, only their token and property data read. A Gremlin OLAP system needs to provide implementations for
-to particular classes: `MapReduce.MapEmitter` and `MapReduce.ReduceEmitter`. TinkerGraph's implementation is provided
-below which demonstrates the simplicity of the algorithm (especially when the data is all within the same JVM).
-
-[source,java]
-----
-public class TinkerMapEmitter<K, V> implements MapReduce.MapEmitter<K, V> {
-
- public Map<K, Queue<V>> reduceMap;
- public Queue<KeyValue<K, V>> mapQueue;
- private final boolean doReduce;
-
- public TinkerMapEmitter(final boolean doReduce) { <1>
- this.doReduce = doReduce;
- if (this.doReduce)
- this.reduceMap = new ConcurrentHashMap<>();
- else
- this.mapQueue = new ConcurrentLinkedQueue<>();
- }
-
- @Override
- public void emit(K key, V value) {
- if (this.doReduce)
- this.reduceMap.computeIfAbsent(key, k -> new ConcurrentLinkedQueue<>()).add(value); <2>
- else
- this.mapQueue.add(new KeyValue<>(key, value)); <3>
- }
-
- protected void complete(final MapReduce<K, V, ?, ?, ?> mapReduce) {
- if (!this.doReduce && mapReduce.getMapKeySort().isPresent()) { <4>
- final Comparator<K> comparator = mapReduce.getMapKeySort().get();
- final List<KeyValue<K, V>> list = new ArrayList<>(this.mapQueue);
- Collections.sort(list, Comparator.comparing(KeyValue::getKey, comparator));
- this.mapQueue.clear();
- this.mapQueue.addAll(list);
- } else if (mapReduce.getMapKeySort().isPresent()) {
- final Comparator<K> comparator = mapReduce.getMapKeySort().get();
- final List<Map.Entry<K, Queue<V>>> list = new ArrayList<>();
- list.addAll(this.reduceMap.entrySet());
- Collections.sort(list, Comparator.comparing(Map.Entry::getKey, comparator));
- this.reduceMap = new LinkedHashMap<>();
- list.forEach(entry -> this.reduceMap.put(entry.getKey(), entry.getValue()));
- }
- }
-}
-----
-
-<1> If the MapReduce job has a reduce, then use one data structure (`reduceMap`), else use another (`mapList`). The
-difference being that a reduction requires a grouping by key and therefore, the `Map<K,Queue<V>>` definition. If no
-reduction/grouping is required, then a simple `Queue<KeyValue<K,V>>` can be leveraged.
-<2> If reduce is to follow, then increment the Map with a new value for the key. `MapHelper` is a TinkerPop3 class
-with static methods for adding data to a Map.
-<3> If no reduce is to follow, then simply append a KeyValue to the queue.
-<4> When the map phase is complete, any map-result sorting required can be executed at this point.
-
-[source,java]
-----
-public class TinkerReduceEmitter<OK, OV> implements MapReduce.ReduceEmitter<OK, OV> {
-
- protected Queue<KeyValue<OK, OV>> reduceQueue = new ConcurrentLinkedQueue<>();
-
- @Override
- public void emit(final OK key, final OV value) {
- this.reduceQueue.add(new KeyValue<>(key, value));
- }
-
- protected void complete(final MapReduce<?, ?, OK, OV, ?> mapReduce) {
- if (mapReduce.getReduceKeySort().isPresent()) {
- final Comparator<OK> comparator = mapReduce.getReduceKeySort().get();
- final List<KeyValue<OK, OV>> list = new ArrayList<>(this.reduceQueue);
- Collections.sort(list, Comparator.comparing(KeyValue::getKey, comparator));
- this.reduceQueue.clear();
- this.reduceQueue.addAll(list);
- }
- }
-}
-----
-
-The method `MapReduce.reduce()` is defined as:
-
-[source,java]
-public void reduce(final OK key, final Iterator<OV> values, final ReduceEmitter<OK, OV> emitter) { ... }
-
-In other words, for the TinkerGraph implementation, iterate through the entrySet of the `reduceMap` and call the
-`reduce()` method on each entry. The `reduce()` method can emit key/value pairs which are simply aggregated into a
-`Queue<KeyValue<OK,OV>>` in an analogous fashion to `TinkerMapEmitter` when no reduce is to follow. These two emitters
-are tied together in `TinkerGraphComputer.submit()`.
-
-[source,java]
-----
-...
-for (final MapReduce mapReduce : mapReducers) {
- if (mapReduce.doStage(MapReduce.Stage.MAP)) {
- final TinkerMapEmitter<?, ?> mapEmitter = new TinkerMapEmitter<>(mapReduce.doStage(MapReduce.Stage.REDUCE));
- final SynchronizedIterator<Vertex> vertices = new SynchronizedIterator<>(this.graph.vertices());
- workers.setMapReduce(mapReduce);
- workers.mapReduceWorkerStart(MapReduce.Stage.MAP);
- workers.executeMapReduce(workerMapReduce -> {
- while (true) {
- final Vertex vertex = vertices.next();
- if (null == vertex) return;
- workerMapReduce.map(ComputerGraph.mapReduce(vertex), mapEmitter);
- }
- });
- workers.mapReduceWorkerEnd(MapReduce.Stage.MAP);
-
- // sort results if a map output sort is defined
- mapEmitter.complete(mapReduce);
-
- // no need to run combiners as this is single machine
- if (mapReduce.doStage(MapReduce.Stage.REDUCE)) {
- final TinkerReduceEmitter<?, ?> reduceEmitter = new TinkerReduceEmitter<>();
- final SynchronizedIterator<Map.Entry<?, Queue<?>>> keyValues = new SynchronizedIterator((Iterator) mapEmitter.reduceMap.entrySet().iterator());
- workers.mapReduceWorkerStart(MapReduce.Stage.REDUCE);
- workers.executeMapReduce(workerMapReduce -> {
- while (true) {
- final Map.Entry<?, Queue<?>> entry = keyValues.next();
- if (null == entry) return;
- workerMapReduce.reduce(entry.getKey(), entry.getValue().iterator(), reduceEmitter);
- }
- });
- workers.mapReduceWorkerEnd(MapReduce.Stage.REDUCE);
- reduceEmitter.complete(mapReduce); // sort results if a reduce output sort is defined
- mapReduce.addResultToMemory(this.memory, reduceEmitter.reduceQueue.iterator()); <1>
- } else {
- mapReduce.addResultToMemory(this.memory, mapEmitter.mapQueue.iterator()); <2>
- }
- }
-}
-...
-----
-
-<1> Note that the final results of the reducer are provided to the Memory as specified by the application developer's
-`MapReduce.addResultToMemory()` implementation.
-<2> If there is no reduce stage, the the map-stage results are inserted into Memory as specified by the application
-developer's `MapReduce.addResultToMemory()` implementation.
-
-[[io-implementations]]
-IO Implementations
-^^^^^^^^^^^^^^^^^^
-
-If a `Graph` requires custom serializers for IO to work properly, implement the `Graph.io` method. A typical example
-of where a `Graph` would require such a custom serializers is if their identifier system uses non-primitive values,
-such as OrientDB's `Rid` class. From basic serialization of a single `Vertex` all the way up the stack to Gremlin
-Server, the need to know how to handle these complex identifiers is an important requirement.
-
-The first step to implementing custom serializers is to first implement the `IoRegistry` interface and register the
-custom classes and serializers to it. Each `Io` implementation has different requirements for what it expects from the
-`IoRegistry`:
-
-* *GraphML* - No custom serializers expected/allowed.
-* *GraphSON* - Register a Jackson `SimpleModule`. The `SimpleModule` encapsulates specific classes to be serialized,
-so it does not need to be registered to a specific class in the `IoRegistry` (use `null`).
-* *Gryo* - Expects registration of one of three objects:
-** Register just the custom class with a `null` Kryo `Serializer` implementation - this class will use default "field-level" Kryo serialization.
-** Register the custom class with a specific Kryo `Serializer' implementation.
-** Register the custom class with a `Function<Kryo, Serializer>` for those cases where the Kryo `Serializer` requires the `Kryo` instance to get constructed.
-
-This implementation should provide a zero-arg constructor as the stack may require instantiation via reflection.
-Consider extending `AbstractIoRegistry` for convenience as follows:
-
-[source,java]
-----
-public class MyGraphIoRegistry extends AbstractIoRegistry {
- public MyGraphIoRegistry() {
- register(GraphSONIo.class, null, new MyGraphSimpleModule());
- register(GryoIo.class, MyGraphIdClass.class, new MyGraphIdSerializer());
- }
-}
-----
-
-In the `Graph.io` method, provide the `IoRegistry` object to the supplied `Builder` and call the `create` method to
-return that `Io` instance as follows:
-
-[source,java]
-----
-public <I extends Io> I io(final Io.Builder<I> builder) {
- return (I) builder.graph(this).registry(myGraphIoRegistry).create();
-}}
-----
-
-In this way, `Graph` implementations can pre-configure custom serializers for IO interactions and users will not need
-to know about those details. Following this pattern will ensure proper execution of the test suite as well as
-simplified usage for end-users.
-
-IMPORTANT: Proper implementation of IO is critical to successful `Graph` operations in Gremlin Server. The Test Suite
-does have "serialization" tests that provide some assurance that an implementation is working properly, but those
-tests cannot make assertions against any specifics of a custom serializer. It is the responsibility of the
-implementer to test the specifics of their custom serializers.
-
-TIP: Consider separating serializer code into its own module, if possible, so that clients that use the `Graph`
-implementation remotely don't need a full dependency on the entire `Graph` - just the IO components and related
-classes being serialized.
-
-[[validating-with-gremlin-test]]
-Validating with Gremlin-Test
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-image:gremlin-edumacated.png[width=225]
-
-[source,xml]
-<dependency>
- <groupId>org.apache.tinkerpop</groupId>
- <artifactId>gremlin-test</artifactId>
- <version>x.y.z</version>
-</dependency>
-<dependency>
- <groupId>org.apache.tinkerpop</groupId>
- <artifactId>gremlin-groovy-test</artifactId>
- <version>x.y.z</version>
-</dependency>
-
-The operational semantics of any OLTP or OLAP implementation are validated by `gremlin-test` and functional
-interoperability with the Groovy environment is ensured by `gremlin-groovy-test`. To implement these tests, provide
-test case implementations as shown below, where `XXX` below denotes the name of the graph implementation (e.g.
-TinkerGraph, Neo4jGraph, HadoopGraph, etc.).
-
-[source,java]
-----
-// Structure API tests
-@RunWith(StructureStandardSuite.class)
-@GraphProviderClass(provider = XXXGraphProvider.class, graph = XXXGraph.class)
-public class XXXStructureStandardTest {}
-
-// Process API tests
-@RunWith(ProcessComputerSuite.class)
-@GraphProviderClass(provider = XXXGraphProvider.class, graph = XXXGraph.class)
-public class XXXProcessComputerTest {}
-
-@RunWith(ProcessStandardSuite.class)
-@GraphProviderClass(provider = XXXGraphProvider.class, graph = XXXGraph.class)
-public class XXXProcessStandardTest {}
-
-@RunWith(GroovyEnvironmentSuite.class)
-@GraphProviderClass(provider = XXXProvider.class, graph = TinkerGraph.class)
-public class XXXGroovyEnvironmentTest {}
-
-@RunWith(GroovyProcessStandardSuite.class)
-@GraphProviderClass(provider = XXXGraphProvider.class, graph = TinkerGraph.class)
-public class XXXGroovyProcessStandardTest {}
-
-@RunWith(GroovyProcessComputerSuite.class)
-@GraphProviderClass(provider = XXXGraphComputerProvider.class, graph = TinkerGraph.class)
-public class XXXGroovyProcessComputerTest {}
-----
-
-The above set of tests represent the minimum test suite set to implement. There are other "integration" and
-"performance" tests that should be considered optional. Implementing those tests requires the same pattern as shown above.
-
-IMPORTANT: It is as important to look at "ignored" tests as it is to look at ones that fail. The `gremlin-test`
-suite utilizes the `Feature` implementation exposed by the `Graph` to determine which tests to execute. If a test
-utilizes features that are not supported by the graph, it will ignore them. While that may be fine, implementers
-should validate that the ignored tests are appropriately bypassed and that there are no mistakes in their feature
-definitions. Moreover, implementers should consider filling gaps in their own test suites, especially when
-IO-related tests are being ignored.
-
-The only test-class that requires any code investment is the `GraphProvider` implementation class. This class is a
-used by the test suite to construct `Graph` configurations and instances and provides information about the
-implementation itself. In most cases, it is best to simply extend `AbstractGraphProvider` as it provides many
-default implementations of the `GraphProvider` interface.
-
-Finally, specify the test suites that will be supported by the `Graph` implementation using the `@Graph.OptIn`
-annotation. See the `TinkerGraph` implementation below as an example:
-
-[source,java]
-----
-@Graph.OptIn(Graph.OptIn.SUITE_STRUCTURE_STANDARD)
-@Graph.OptIn(Graph.OptIn.SUITE_PROCESS_STANDARD)
-@Graph.OptIn(Graph.OptIn.SUITE_PROCESS_COMPUTER)
-@Graph.OptIn(Graph.OptIn.SUITE_GROOVY_PROCESS_STANDARD)
-@Graph.OptIn(Graph.OptIn.SUITE_GROOVY_PROCESS_COMPUTER)
-@Graph.OptIn(Graph.OptIn.SUITE_GROOVY_ENVIRONMENT)
-public class TinkerGraph implements Graph {
-----
-
-Only include annotations for the suites the implementation will support. Note that implementing the suite, but
-not specifying the appropriate annotation will prevent the suite from running (an obvious error message will appear
-in this case when running the mis-configured suite).
-
-There are times when there may be a specific test in the suite that the implementation cannot support (despite the
-features it implements) or should not otherwise be executed. It is possible for implementers to "opt-out" of a test
-by using the `@Graph.OptOut` annotation. The following is an example of this annotation usage as taken from
-`HadoopGraph`:
-
-[source, java]
-----
-@Graph.OptIn(Graph.OptIn.SUITE_PROCESS_STANDARD)
-@Graph.OptIn(Graph.OptIn.SUITE_PROCESS_COMPUTER)
-@Graph.OptOut(
- test = "org.apache.tinkerpop.gremlin.process.graph.step.map.MatchTest$Traversals",
- method = "g_V_matchXa_hasXname_GarciaX__a_inXwrittenByX_b__a_inXsungByX_bX",
- reason = "Hadoop-Gremlin is OLAP-oriented and for OLTP operations, linear-scan joins are required. This particular tests takes many minutes to execute.")
-@Graph.OptOut(
- test = "org.apache.tinkerpop.gremlin.process.graph.step.map.MatchTest$Traversals",
- method = "g_V_matchXa_inXsungByX_b__a_inXsungByX_c__b_outXwrittenByX_d__c_outXwrittenByX_e__d_hasXname_George_HarisonX__e_hasXname_Bob_MarleyXX",
- reason = "Hadoop-Gremlin is OLAP-oriented and for OLTP operations, linear-scan joins are required. This particular tests takes many minutes to execute.")
-@Graph.OptOut(
- test = "org.apache.tinkerpop.gremlin.process.computer.GraphComputerTest",
- method = "shouldNotAllowBadMemoryKeys",
- reason = "Hadoop does a hard kill on failure and stops threads which stops test cases. Exception handling semantics are correct though.")
-@Graph.OptOut(
- test = "org.apache.tinkerpop.gremlin.process.computer.GraphComputerTest",
- method = "shouldRequireRegisteringMemoryKeys",
- reason = "Hadoop does a hard kill on failure and stops threads which stops test cases. Exception handling semantics are correct though.")
-public class HadoopGraph implements Graph {
-----
-
-The above examples show how to ignore individual tests. It is also possible to:
-
-* Ignore an entire test case (i.e. all the methods within the test) by setting the `method` to "*".
-* Ignore a "base" test class such that test that extend from those classes will all be ignored. This style of
-ignoring is useful for Gremlin "process" tests that have bases classes that are extended by various Gremlin flavors (e.g. groovy).
-* Ignore a `GraphComputer` test based on the type of `GraphComputer` being used. Specify the "computer" attribute on
-the `OptOut` (which is an array specification) which should have a value of the `GraphComputer` implementation class
-that should ignore that test. This attribute should be left empty for "standard" execution and by default all
-`GraphComputer` implementations will be included in the `OptOut` so if there are multiple implementations, explicitly
-specify the ones that should be excluded.
-
-Also note that some of the tests in the Gremlin Test Suite are parameterized tests and require an additional level of
-specificity to be properly ignored. To ignore these types of tests, examine the name template of the parameterized
-tests. It is defined by a Java annotation that looks like this:
-
-[source, java]
-@Parameterized.Parameters(name = "expect({0})")
-
-The annotation above shows that the name of each parameterized test will be prefixed with "expect" and have
-parentheses wrapped around the first parameter (at index 0) value supplied to each test. This information can
-only be garnered by studying the test set up itself. Once the pattern is determined and the specific unique name of
-the parameterized test is identified, add it to the `specific` property on the `OptOut` annotation in addition to
-the other arguments.
-
-These annotations help provide users a level of transparency into test suite compliance (via the
-xref:describe-graph[describeGraph()] utility function). It also allows implementers to have a lot of flexibility in
-terms of how they wish to support TinkerPop. For example, maybe there is a single test case that prevents an
-implementer from claiming support of a `Feature`. The implementer could choose to either not support the `Feature`
-or to support it but "opt-out" of the test with a "reason" as to why so that users understand the limitation.
-
-IMPORTANT: Before using `OptOut` be sure that the reason for using it is sound and it is more of a last resort.
-It is possible that a test from the suite doesn't properly represent the expectations of a feature, is too broad or
-narrow for the semantics it is trying to enforce or simply contains a bug. Please consider raising issues in the
-developer mailing list with such concerns before assuming `OptOut` is the only answer.
-
-IMPORTANT: There are no tests that specifically validate complete compliance with Gremlin Server. Generally speaking,
-a `Graph` that passes the full Test Suite, should be compliant with Gremlin Server. The one area where problems can
-occur is in serialization. Always ensure that IO is properly implemented, that custom serializers are tested fully
-and ultimately integration test the `Graph` with an actual Gremlin Server instance.
-
-CAUTION: Configuring tests to run in parallel might result in errors that are difficult to debug as there is some
-shared state in test execution around graph configuration. It is therefore recommended that parallelism be turned
-off for the test suite (the Maven SureFire Plugin is configured this way by default). It may also be important to
-include this setting, `<reuseForks>false</reuseForks>`, in the SureFire configuration if tests are failing in an
-unexplainable way.
-
-Accessibility via GremlinPlugin
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-image:gremlin-plugin.png[width=100,float=left] The applications distributed with TinkerPop3 do not distribute with
-any graph system implementations besides TinkerGraph. If your implementation is stored in a Maven repository (e.g.
-Maven Central Repository), then it is best to provide a `GremlinPlugin` implementation so the respective jars can be
-downloaded according and when required by the user. Neo4j's GremlinPlugin is provided below for reference.
-
-[source,java]
-----
-public class Neo4jGremlinPlugin implements GremlinPlugin {
-
- private static final String IMPORT = "import ";
- private static final String DOT_STAR = ".*";
-
- private static final Set<String> IMPORTS = new HashSet<String>() {{
- add(IMPORT + Neo4jGraph.class.getPackage().getName() + DOT_STAR);
- }};
-
- @Override
- public String getName() {
- return "neo4j";
- }
-
- @Override
- public void pluginTo(final PluginAcceptor pluginAcceptor) {
- pluginAcceptor.addImports(IMPORTS);
- }
-}
-----
-
-With the above plugin implementations, users can now download respective binaries for Gremlin Console, Gremlin Server, etc.
-
-[source,groovy]
-gremlin> g = Neo4jGraph.open('/tmp/neo4j')
-No such property: Neo4jGraph for class: groovysh_evaluate
-Display stack trace? [yN]
-gremlin> :install org.apache.tinkerpop neo4j-gremlin x.y.z
-==>loaded: [org.apache.tinkerpop, neo4j-gremlin, …]
-gremlin> :plugin use tinkerpop.neo4j
-==>tinkerpop.neo4j activated
-gremlin> g = Neo4jGraph.open('/tmp/neo4j')
-==>neo4jgraph[EmbeddedGraphDatabase [/tmp/neo4j]]
-
-In-Depth Implementations
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-image:gremlin-painting.png[width=200,float=right] The graph system implementation details presented thus far are
-minimum requirements necessary to yield a valid TinkerPop3 implementation. However, there are other areas that a
-graph system provider can tweak to provide an implementation more optimized for their underlying graph engine. Typical
-areas of focus include:
-
-* Traversal Strategies: A <<traversalstrategy,TraversalStrategy>> can be used to alter a traversal prior to its
-execution. A typical example is converting a pattern of `g.V().has('name','marko')` into a global index lookup for
-all vertices with name "marko". In this way, a `O(|V|)` lookup becomes an `O(log(|V|))`. Please review
-`TinkerGraphStepStrategy` for ideas.
-* Step Implementations: Every <<graph-traversal-steps,step>> is ultimately referenced by the `GraphTraversal`
-interface. It is possible to extend `GraphTraversal` to use a graph system specific step implementation.
-
-
-[[tinkergraph-gremlin]]
-TinkerGraph-Gremlin
--------------------
-
-[source,xml]
-----
-<dependency>
- <groupId>org.apache.tinkerpop</groupId>
- <artifactId>tinkergraph-gremlin</artifactId>
- <version>x.y.z</version>
-</dependency>
-----
-
-image:tinkerpop-character.png[width=100,float=left] TinkerGraph is a single machine, in-memory (with optional
-persistence), non-transactional graph engine that provides both OLTP and OLAP functionality. It is deployed with
-TinkerPop3 and serves as the reference implementation for other providers to study in order to understand the
-semantics of the various methods of the TinkerPop3 API. Constructing a simple graph in Java8 is presented below.
-
-[source,java]
-Graph g = TinkerGraph.open();
-Vertex marko = g.addVertex("name","marko","age",29);
-Vertex lop = g.addVertex("name","lop","lang","java");
-marko.addEdge("created",lop,"weight",0.6d);
-
-The above graph creates two vertices named "marko" and "lop" and connects them via a created-edge with a weight=0.6
-property. Next, the graph can be queried as such.
-
-[source,java]
-g.V().has("name","marko").out("created").values("name")
-
-The `g.V().has("name","marko")` part of the query can be executed in two ways.
-
- * A linear scan of all vertices filtering out those vertices that don't have the name "marko"
- * A `O(log(|V|))` index lookup for all vertices with the name "marko"
-
-Given the initial graph construction in the first code block, no index was defined and thus, a linear scan is executed.
-However, if the graph was constructed as such, then an index lookup would be used.
-
-[source,java]
-Graph g = TinkerGraph.open();
-g.createIndex("name",Vertex.class)
-
-The execution times for a vertex lookup by property is provided below for both no-index and indexed version of
-TinkerGraph over the Grateful Dead graph.
-
-[gremlin-groovy]
-----
-graph = TinkerGraph.open()
-g = graph.traversal()
-graph.io(graphml()).readGraph('data/grateful-dead.xml')
-clock(1000) {g.V().has('name','Garcia').iterate()} <1>
-graph = TinkerGraph.open()
-g = graph.traversal()
-graph.createIndex('name',Vertex.class)
-graph.io(graphml()).readGraph('data/grateful-dead.xml')
-clock(1000){g.V().has('name','Garcia').iterate()} <2>
-----
-
-<1> Determine the average runtime of 1000 vertex lookups when no `name`-index is defined.
-<2> Determine the average runtime of 1000 vertex lookups when a `name`-index is defined.
-
-IMPORTANT: Each graph system will have different mechanism by which indices and schemas are defined. TinkerPop3
-does not require any conformance in this area. In TinkerGraph, the only definitions are around indices. With other
-graph systems, property value types, indices, edge labels, etc. may be required to be defined _a priori_ to adding
-data to the graph.
-
-NOTE: TinkerGraph is distributed with Gremlin Server and is therefore automatically available to it for configuration.
-
-Configuration
-~~~~~~~~~~~~~
-
-TinkerGraph has several settings that can be provided on creation via `Configuration` object:
-
-[width="100%",cols="2,10",options="header"]
-|=========================================================
-|Property |Description
-|gremlin.graph |`org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph`
-|gremlin.tinkergraph.vertexIdManager |The `IdManager` implementation to use for vertices.
-|gremlin.tinkergraph.edgeIdManager |The `IdManager` implementation to use for edges.
-|gremlin.tinkergraph.vertexPropertyIdManager |The `IdManager` implementation to use for vertex properties.
-|gremlin.tinkergraph.defaultVertexPropertyCardinality |The default `VertexProperty.Cardinality` to use when `Vertex.property(k,v)` is called.
-|gremlin.tinkergraph.graphLocation |The path and file name for where TinkerGraph should persist the graph data. If a
-value is specified here, the the `gremlin.tinkergraph.graphFormat` should also be specified. If this value is not
-included (default), then the graph will stay in-memory and not be loaded/persisted to disk.
-|gremlin.tinkergraph.graphFormat |The format to use to serialize the graph which may be one of the following:
-`graphml`, `graphson`, or `gryo`. If a value is specified here, the the `gremlin.tinkergraph.graphLocation` should
-also be specified. If this value is not included (default), then the graph will stay in-memory and not be
-loaded/persisted to disk.
-|=========================================================
-
-The `IdManager` settings above refer to how TinkerGraph will control identifiers for vertices, edges and vertex
-properties. There are several options for each of these settings: `ANY`, `LONG`, `INTEGER`, `UUID`, or the fully
-qualified class name of an `IdManager` implementation on the classpath. When not specified, the default values
-for all settings is `ANY`, meaning that the graph will work with any object on the JVM as the identifier and will
-generate new identifiers from `Long` when the identifier is not user supplied. TinkerGraph will also expect the
-user to understand the types used for identifiers when querying, meaning that `g.V(1)` and `g.V(1L)` could return
-two different vertices. `LONG`, `INTEGER` and `UUID` settings will try to coerce identifier values to the expected
-type as well as generate new identifiers with that specified type.
-
-If the TinkerGraph is configured for persistence with `gremlin.tinkergraph.graphLocation` and
-`gremlin.tinkergraph.graphFormat`, then the graph will be written to the specified location with the specified
-format when `Graph.close()` is called. In addition, if these settings are present, TinkerGraph will attempt to
-load the graph from the specified location.
-
-It is important to consider the data being imported to TinkerGraph with respect to `defaultVertexPropertyCardinality`
-setting. For example, if a `.gryo` file is known to contain multi-property data, be sure to set the default
-cardinality to `list` or else the data will import as `single`. Consider the following:
-
-[gremlin-groovy]
-----
-graph = TinkerGraph.open()
-graph.io(gryo()).readGraph("data/tinkerpop-crew.kryo")
-g = graph.traversal()
-g.V().properties()
-conf = new BaseConfiguration()
-conf.setProperty("gremlin.tinkergraph.defaultVertexPropertyCardinality","list")
-graph = TinkerGraph.open(conf)
-graph.io(gryo()).readGraph("data/tinkerpop-crew.kryo")
-g = graph.traversal()
-g.V().properties()
-----
-
-[[neo4j-gremlin]]
-Neo4j-Gremlin
--------------
-
-[source,xml]
-----
-<dependency>
- <groupId>org.apache.tinkerpop</groupId>
- <artifactId>neo4j-gremlin</artifactId>
- <version>x.y.z</version>
-</dependency>
-<!-- neo4j-tinkerpop-api-impl is NOT Apache 2 licensed - more information below -->
-<dependency>
- <groupId>org.neo4j</groupId>
- <artifactId>neo4j-tinkerpop-api-impl</artifactId>
- <version>0.1-2.2</version>
-</dependency>
-----
-
-link:http://neotechnology.com[Neo Technology] are the developers of the OLTP-based link:http://neo4j.org[Neo4j graph database].
-
-CAUTION: Unless under a commercial agreement with Neo Technology, Neo4j is licensed
-link:http://en.wikipedia.org/wiki/Affero_General_Public_License[AGPL]. The `neo4j-gremlin` module is licensed Apache2
-because it only references the Apache2-licensed Neo4j API (not its implementation). Note that neither the
-<<gremlin-console,Gremlin Console>> nor <<gremlin-server,Gremlin Server>> distribute with the Neo4j implementation
-binaries. To access the binaries, use the `:install` command to download binaries from
-link:http://search.maven.org/[Maven Central Repository].
-
-[source,groovy]
-----
-gremlin> :install org.apache.tinkerpop neo4j-gremlin x.y.z
-==>Loaded: [org.apache.tinkerpop, neo4j-gremlin, x.y.z] - restart the console to use [tinkerpop.neo4j]
-gremlin> :q
-...
-gremlin> :plugin use tinkerpop.neo4j
-==>tinkerpop.neo4j activated
-gremlin> graph = Neo4jGraph.open('/tmp/neo4j')
-==>neo4jgraph[EmbeddedGraphDatabase [/tmp/neo4j]]
-----
-
-NOTE: Neo4j link:http://docs.neo4j.org/chunked/stable/ha.html[High Availability] is currently not supported by
-Neo4j-Gremlin.
-
-TIP: To host Neo4j in <<gremlin-server,Gremlin Server>>, the dependencies must first be "installed" or otherwise
-copied to the Gremlin Server path. The automated method for doing this would be to execute
-`bin/gremlin-server.sh -i org.apache.tinkerpop neo4j-gremlin x.y.z`.
-
-Indices
-~~~~~~~
-
-Neo4j 2.x indices leverage vertex labels to partition the index space. TinkerPop3 does not provide method interfaces
-for defining schemas/indices for the underlying graph system. Thus, in order to create indices, it is important to
-call the Neo4j API directly.
-
-NOTE: `Neo4jGraphStep` will attempt to discern which indices to use when executing a traversal of the form `g.V().has()`.
-
-The Gremlin-Console session below demonstrates Neo4j indices. For more information, please refer to the Neo4j documentation:
-
-* Manipulating indices with link:http://docs.neo4j.org/chunked/stable/query-schema-index.html[Cypher].
-* Manipulating indices with the Neo4j link:http://docs.neo4j.org/chunked/stable/tutorials-java-embedded-new-index.html[Java API].
-
-[gremlin-groovy]
-----
-graph = Neo4jGraph.open('/tmp/neo4j')
-graph.cypher("CREATE INDEX ON :person(name)")
-graph.tx().commit() <1>
-graph.addVertex(label,'person','name','marko')
-graph.addVertex(label,'dog','name','puppy')
-g = graph.traversal()
-g.V().hasLabel('person').has('name','marko').values('name')
-graph.close()
-----
-
-<1> Schema mutations must happen in a different transaction than graph mutations
-
-Below demonstrates the runtime benefits of indices and demonstrates how if there is no defined index (only vertex
-labels), a linear scan of the vertex-label partition is still faster than a linear scan of all vertices.
-
-[gremlin-groovy]
-----
-graph = Neo4jGraph.open('/tmp/neo4j')
-graph.io(graphml()).readGraph('data/grateful-dead.xml')
-g = graph.traversal()
-g.tx().commit()
-clock(1000) {g.V().hasLabel('artist').has('name','Garcia').iterate()} <1>
-graph.cypher("CREATE INDEX ON :artist(name)") <2>
-g.tx().commit()
-Thread.sleep(5000) <3>
-clock(1000) {g.V().hasLabel('artist').has('name','Garcia').iterate()} <4>
-clock(1000) {g.V().has('name','Garcia').iterate()} <5>
-graph.cypher("DROP INDEX ON :artist(name)") <6>
-g.tx().commit()
-graph.close()
-----
-
-<1> Find all artists whose name is Garcia which does a linear scan of the artist vertex-label partition.
-<2> Create an index for all artist vertices on their name property.
-<3> Neo4j indices are eventually consistent so this stalls to give the index time to populate itself.
-<4> Find all artists whose name is Garcia which uses the pre-defined schema index.
-<5> Find all vertices whose name is Garcia which requires a linear scan of all the data in the graph.
-<6> Drop the created index.
-
-Multi/Meta-Properties
-~~~~~~~~~~~~~~~~~~~~~
-
-`Neo4jGraph` supports both multi- and meta-properties (see <<_vertex_properties,vertex properties>>). These features
-are not native to Neo4j and are implemented using "hidden" Neo4j nodes. For example, when a vertex has multiple
-"name" properties, each property is a new node (multi-properties) which can have properties attached to it
-(meta-properties). As such, the native, underlying representation may become difficult to query directly using
-another graph language such as <<_cypher,Cypher>>. The default setting is to disable multi- and meta-properties.
-However, if this feature is desired, then it can be activated via `gremlin.neo4j.metaProperties` and
-`gremlin.neo4j.multiProperties` configurations being set to `true`. Once the configuration is set, it can not be
-changed for the lifetime of the graph.
-
-[gremlin-groovy]
-----
-conf = new BaseConfiguration()
-conf.setProperty('gremlin.neo4j.directory','/tmp/neo4j')
-conf.setProperty('gremlin.neo4j.multiProperties',true)
-conf.setProperty('gremlin.neo4j.metaProperties',true)
-graph = Neo4jGraph.open(conf)
-g = graph.traversal()
-g.addV('name','michael','name','michael hunger','name','mhunger')
-g.V().properties('name').property('acl', 'public')
-g.V(0).valueMap()
-g.V(0).properties()
-g.V(0).properties().valueMap()
-graph.close()
-----
-
-WARNING: `Neo4jGraph` without multi- and meta-properties is in 1-to-1 correspondence with the native, underlying Neo4j
-representation. It is recommended that if the user does not require multi/meta-properties, then they should not
-enable them. Without multi- and meta-properties enabled, Neo4j can be interacted with with other tools and technologies
-that do not leverage TinkerPop.
-
-IMPORTANT: When using a multi-property enabled `Neo4jGraph`, vertices may represent their properties on "hidden
-nodes" adjacent to the vertex. If a vertex property key/value is required for indexing, then two indices are
-required -- e.g. `CREATE INDEX ON :person(name)` and `CREATE INDEX ON :vertexProperty(name)`
-(see <<_indices,Neo4j indices>>).
-
-Cypher
-~~~~~~
-
-image::gremlin-loves-cypher.png[width=400]
-
-NeoTechnology are the creators of the graph pattern-match query language link:http://www.neo4j.org/learn/cypher[Cypher].
-It is possible to leverage Cypher from within Gremlin by using the `Neo4jGraph.cypher()` graph traversal method.
-
-[gremlin-groovy]
-----
-graph = Neo4jGraph.open('/tmp/neo4j')
-graph.io(gryo()).readGraph('data/tinkerpop-modern.kryo')
-graph.cypher('MATCH (a {name:"marko"}) RETURN a')
-graph.cypher('MATCH (a {name:"marko"}) RETURN a').select('a').out('knows').values('name')
-graph.close()
-----
-
-Thus, like <<match-step,`match()`>>-step in Gremlin, it is possible to do a declarative pattern match and then move
-back into imperative Gremlin.
-
-TIP: For those developers using <<gremlin-server,Gremlin Server>> against Neo4j, it is possible to do Cypher queries
-by simply placing the Cypher string in `graph.cypher(...)` before submission to the server.
-
-Multi-Label
-~~~~~~~~~~~
-
-TinkerPop3 requires every `Element` to have a single, immutable string label (i.e. a `Vertex`, `Edge`, and
-`VertexProperty`). In Neo4j, a `Node` (vertex) can have an
-link:http://neo4j.com/docs/stable/graphdb-neo4j-labels.html[arbitrary number of labels] while a `Relationship`
-(edge) can have one and only one. Furthermore, in Neo4j, `Node` labels are mutable while `Relationship` labels are
-not. In order to handle this mismatch, three `Neo4jVertex` specific methods exist in Neo4j-Gremlin.
-
-[source,java]
-public Set<String> labels() // get all the labels of the vertex
-public void addLabel(String label) // add a label to the vertex
-public void removeLabel(String label) // remove a label from the vertex
-
-An example use case is presented below.
-
-[gremlin-groovy]
-----
-graph = Neo4jGraph.open('/tmp/neo4j')
-vertex = (Neo4jVertex) graph.addVertex('human::animal') <1>
-vertex.label() <2>
-vertex.labels() <3>
-vertex.addLabel('organism') <4>
-vertex.label()
-vertex.removeLabel('human') <5>
-vertex.labels()
-vertex.addLabel('organism') <6>
-vertex.labels()
-vertex.removeLabel('human') <7>
-vertex.label()
-g = graph.traversal()
-g.V().has(label,'organism') <8>
-g.V().has(label,of('organism')) <9>
-g.V().has(label,of('organism')).has(label,of('animal'))
-g.V().has(label,of('organism').and(of('animal')))
-graph.close()
-----
-
-<1> Typecasting to a `Neo4jVertex` is only required in Java.
-<2> The standard `Vertex.label()` method returns all the labels in alphabetical order concatenated using `::`.
-<3> `Neo4jVertex.labels()` method returns the individual labels as a set.
-<4> `Neo4jVertex.addLabel()` method adds a single label.
-<5> `Neo4jVertex.removeLabel()` method removes a single label.
-<6> Labels are unique and thus duplicate labels don't exist.
-<7> If a label that does not exist is removed, nothing happens.
-<8> `P.eq()` does a full string match and should only be used if multi-labels are not leveraged.
-<9> `LabelP.of()` is specific to `Neo4jGraph` and used for multi-label matching.
-
-IMPORTANT: `LabelP.of()` is only required if multi-labels are leveraged. `LabelP.of()` is used when
-filtering/looking-up vertices by their label(s) as the standard `P.eq()` does a direct match on the `::`-representation
-of `vertex.label()`
-
-Loading with BulkLoaderVertexProgram
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The <<bulkloadervertexprogram, BulkLoaderVertexProgram>> is a generalized bulk loader that can be used to load
-large amounts of data to and from Neo4j. The following code demonstrates how to load the modern graph from TinkerGraph
-into Neo4j:
-
-[gremlin-groovy]
-----
-wgConf = 'conf/neo4j-standalone.properties'
-modern = TinkerFactory.createModern()
-blvp = BulkLoaderVertexProgram.build().
- keepOriginalIds(false).
- writeGraph(wgConf).create(modern)
-modern.compute().workers(1).program(blvp).submit().get()
-graph = GraphFactory.open(wgConf)
-g = graph.traversal()
-g.V().valueMap()
-graph.close()
-----
-
-[source,properties]
-----
-# neo4j-standalone.properties
-
-gremlin.graph=org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jGraph
-gremlin.neo4j.directory=/tmp/neo4j
-gremlin.neo4j.conf.node_auto_indexing=true
-gremlin.neo4j.conf.relationship_auto_indexing=true
-----
-
-[[hadoop-gremlin]]
-Hadoop-Gremlin
---------------
-
-[source,xml]
-----
-<dependency>
- <groupId>org.apache.tinkerpop</groupId>
- <artifactId>hadoop-gremlin</artifactId>
- <version>x.y.z</version>
-</dependency>
-----
-
-image:hadoop-logo-notext.png[width=100,float=left] link:http://hadoop.apache.org/[Hadoop] is a distributed
-computing framework that is used to process data represented across a multi-machine compute cluster. When the
-data in the Hadoop cluster represents a TinkerPop3 graph, then Hadoop-Gremlin can be used to process the graph
-using both TinkerPop3's OLTP and OLAP graph computing models.
-
-IMPORTANT: This section assumes that the user has a Hadoop 2.x cluster functioning. For more information on getting
-started with Hadoop, please see the
-link:http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/SingleCluster.html[Single Node Setup]
-tutorial. Moreover, if using `GiraphGraphComputer` or `SparkGraphComputer` it is advisable that the reader also
-familiarize their self with Giraph (link:http://giraph.apache.org/quick_start.html[Getting Started]) and Spark
-(link:http://spark.apache.org/docs/latest/quick-start.html[Quick Start]).
-
-Installing Hadoop-Gremlin
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The `HADOOP_GREMLIN_LIBS` references locations that contains jars that should be uploaded to a respective
-distributed cache (link:http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html[YARN] or SparkServer).
-Note that the locations in `HADOOP_GREMLIN_LIBS` can be a colon-separated (`:`) and all jars from all locations will
-be loaded into the cluster. Typically, only the jars of the respective GraphComputer are required to be loaded (e.g.
-`GiraphGraphComputer` plugin lib directory).
-
-[source,shell]
-export HADOOP_GREMLIN_LIBS=/usr/local/gremlin-console/ext/giraph-gremlin/lib
-
-If using <<gremlin-console,Gremlin Console>>, it is important to install the Hadoop-Gremlin plugin. Note that
-Hadoop-Gremlin requires a Gremlin Console restart after installing.
-
-[source,text]
-----
-$ bin/gremlin.sh
-
- \,,,/
- (o o)
------oOOo-(3)-oOOo-----
-plugin activated: tinkerpop.server
-plugin activated: tinkerpop.utilities
-plugin activated: tinkerpop.tinkergraph
-gremlin> :install org.apache.tinkerpop hadoop-gremlin x.y.z
-==>loaded: [org.apache.tinkerpop, hadoop-gremlin, x.y.z] - restart the console to use [tinkerpop.hadoop]
-gremlin> :q
-$ bin/gremlin.sh
-
- \,,,/
- (o o)
------oOOo-(3)-oOOo-----
-plugin activated: tinkerpop.server
-plugin activated: tinkerpop.utilities
-plugin activated: tinkerpop.tinkergraph
-gremlin> :plugin use tinkerpop.hadoop
-==>tinkerpop.hadoop activated
-gremlin>
-----
-
-Properties Files
-~~~~~~~~~~~~~~~~
-
-`HadoopGraph` makes use of properties files which ultimately get turned into Apache configurations and/or
-Hadoop configurations. The example properties file presented below is located at `conf/hadoop/hadoop-gryo.properties`.
-
-[source,text]
-gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
-gremlin.hadoop.inputLocation=tinkerpop-modern.kryo
-gremlin.hadoop.graphInputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat
-gremlin.hadoop.outputLocation=output
-gremlin.hadoop.graphOutputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat
-gremlin.hadoop.jarsInDistributedCache=true
-####################################
-# Spark Configuration #
-####################################
-spark.master=local[4]
-spark.executor.memory=1g
-spark.serializer=org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer
-####################################
-# SparkGraphComputer Configuration #
-####################################
-gremlin.spark.graphInputRDD=org.apache.tinkerpop.gremlin.spark.structure.io.InputRDDFormat
-gremlin.spark.graphOutputRDD=org.apache.tinkerpop.gremlin.spark.structure.io.OutputRDDFormat
-gremlin.spark.persistContext=true
-#####################################
-# GiraphGraphComputer Configuration #
-#####################################
-giraph.minWorkers=2
-giraph.maxWorkers=2
-giraph.useOutOfCoreGraph=true
-giraph.useOutOfCoreMessages=true
-mapreduce.map.java.opts=-Xmx1024m
-mapreduce.reduce.java.opts=-Xmx1024m
-giraph.numInputThreads=2
-giraph.numComputeThreads=2
-
-A review of the Hadoop-Gremlin specific properties are provided in the table below. For the respective OLAP
-engines (<<sparkgraphcomputer,`SparkGraphComputer`>> or <<giraphgraphcomputer,`GiraphGraphComputer`>>) refer
-to their respective documentation for configuration options.
-
-[width="100%",cols="2,10",options="header"]
-|=========================================================
-|Property |Description
-|gremlin.graph |The class of the graph to construct using GraphFactory.
-|gremlin.hadoop.inputLocation |The location of the input file(s) for Hadoop-Gremlin to read the graph from.
-|gremlin.hadoop.graphInputFormat |The format that the graph input file(s) are represented in.
-|gremlin.hadoop.outputLocation |The location to write the computed HadoopGraph to.
-|gremlin.hadoop.graphOutputFormat |The format that the output file(s) should be represented in.
-|gremlin.hadoop.jarsInDistributedCache |Whether to upload the Hadoop-Gremlin jars to a distributed cache (necessary if jars are not on the machines' classpaths).
-|=========================================================
-
-
-
-Along with the properties above, the numerous link:http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml[Hadoop specific properties]
-can be added as needed to tune and parameterize the executed Hadoop-Gremlin job on the respective Hadoop cluster.
-
-IMPORTANT: As the size of the graphs being processed becomes large, it is important to fully understand how the
-underlying OLAP engine (e.g. Spark, Giraph, etc.) works and understand the numerous parameterizations offered by
-these systems. Such knowledge can help alleviate out of memory exceptions, slow load times, slow processing times,
-garbage collection issues, etc.
-
-OLTP Hadoop-Gremlin
-~~~~~~~~~~~~~~~~~~~
-
-image:hadoop-pipes.png[width=180,float=left] It is possible to execute OLTP operations over a `HadoopGraph`.
-However, realize that the underlying HDFS files are not random access and thus, to retrieve a vertex, a linear scan
-is required. OLTP operations are useful for peeking into the graph prior to executing a long running OLAP job -- e.g.
-`g.V().valueMap().limit(10)`.
-
-CAUTION: OLTP operations on `HadoopGraph` are not efficient. They require linear scans to execute and are unreasonable
-for large graphs. In such large graph situations, make use of <<traversalvertexprogram,TraversalVertexProgram>>
-which is the OLAP Gremlin machine.
-
-[gremlin-groovy]
-----
-hdfs.copyFromLocal('data/tinkerpop-modern.kryo', 'tinkerpop-modern.kryo')
-hdfs.ls()
-graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
-g = graph.traversal()
-g.V().count()
-g.V().out().out().values('name')
-g.V().group().by{it.value('name')[1]}.by('name').next()
-----
-
-OLAP Hadoop-Gremlin
-~~~~~~~~~~~~~~~~~~~
-
-image:hadoop-furnace.png[width=180,float=left] Hadoop-Gremlin was designed to execute OLAP operations via
-`GraphComputer`. The OLTP examples presented previously are reproduced below, but using `TraversalVertexProgram`
-for the execution of the Gremlin traversal.
-
-A `Graph` in TinkerPop3 can support any number of `GraphComputer` implementations. Out of the box, Hadoop-Gremlin
-supports the following three implementations.
-
-* <<mapreducegraphcomputer,`MapReduceGraphComputer`>>: Leverages Hadoop's MapReduce engine to execute TinkerPop3 OLAP
-computations. (*coming soon*)
-** The graph must fit within the total disk space of the Hadoop cluster (supports massive graphs). Message passing is
-coordinated via MapReduce jobs over the on-disk graph (slow traversals).
-* <<sparkgraphcomputer,`SparkGraphComputer`>>: Leverages Apache Spark to execute TinkerPop3 OLAP computations.
-** The graph may fit within the total RAM of the cluster (supports larger graphs). Message passing is coordinated via
-Spark map/reduce/join operations on in-memory and disk-cached data (average speed traversals).
-* <<giraphgraphcomputer,`GiraphGraphComputer`>>: Leverages Apache Giraph to execute TinkerPop3 OLAP computations.
-** The graph should fit within the total RAM of the Hadoop cluster (graph size restriction), though "out-of-core"
-processing is possible. Message passing is coordinated via ZooKeeper for the in-memory graph (speedy traversals).
-
-TIP: image:gremlin-sugar.png[width=50,float=left] For those wanting to use the <<sugar-plugin,SugarPlugin>> with
-their submitted traversal, do `:remote config useSugar true` as well as `:plugin use tinkerpop.sugar` at the start of
-the Gremlin Console session if it is not already activated.
-
-Note that `SparkGraphComputer` and `GiraphGraphComputer` are loaded via their respective plugins. Typically only
-one plugin or the other is loaded depending on the desired `GraphComputer` to use.
-
-[source,text]
-----
-$ bin/gremlin.sh
-
- \,,,/
- (o o)
------oOOo-(3)-oOOo-----
-plugin activated: tinkerpop.server
-plugin activated: tinkerpop.utilities
-plugin activated: tinkerpop.tinkergraph
-plugin activated: tinkerpop.hadoop
-gremlin> :install org.apache.tinkerpop giraph-gremlin x.y.z
-==>loaded: [org.apache.tinkerpop, giraph-gremlin, x.y.z] - restart the console to use [tinkerpop.giraph]
-gremlin> :install org.apache.tinkerpop spark-gremlin x.y.z
-==>loaded: [org.apache.tinkerpop, spark-gremlin, x.y.z] - restart the console to use [tinkerpop.spark]
-gremlin> :q
-$ bin/gremlin.sh
-
- \,,,/
- (o o)
------oOOo-(3)-oOOo-----
-plugin activated: tinkerpop.server
-plugin activated: tinkerpop.utilities
-plugin activated: tinkerpop.tinkergraph
-plugin activated: tinkerpop.hadoop
-gremlin> :plugin use tinkerpop.giraph
-==>tinkerpop.giraph activated
-gremlin> :plugin use tinkerpop.spark
-==>tinkerpop.spark activated
-----
-
-WARNING: Hadoop, Spark, and Giraph all depend on many of the same libraries (e.g. ZooKeeper, Snappy, Netty, Guava,
-etc.). Unfortunately, typically these dependencies are not to the same versions of the respective libraries. As such,
-it is best to *not* have both Spark and Giraph plugins loaded in the same console session nor in the same Java
-project (though intelligent `<exclusion>`-usage can help alleviate conflicts in a Java project).
-
-[[mapreducegraphcomputer]]
-MapReduceGraphComputer
-^^^^^^^^^^^^^^^^^^^^^^
-
-*COMING SOON*
-
-[[sparkgraphcomputer]]
-SparkGraphComputer
-^^^^^^^^^^^^^^^^^^
-
-[source,xml]
-----
-<dependency>
- <groupId>org.apache.tinkerpop</groupId>
- <artifactId>spark-gremlin</artifactId>
- <version>x.y.z</version>
-</dependency>
-----
-
-image:spark-logo.png[width=175,float=left] link:http://spark.apache.org[Spark] is an Apache Software Foundation
-project focused on general-purpose OLAP data processing. Spark provides a hybrid in-memory/disk-based distributed
-computing model that is similar to Hadoop's MapReduce model. Spark maintains a fluent function chaining DSL that is
-arguably easier for developers to work with than native Hadoop MapReduce. Spark-Gremlin provides an implementation of
-the bulk-synchronous parallel, distributed message passing algorithm within Spark and thus, any `VertexProgram` can be
-executed over `SparkGraphComputer`.
-
-[gremlin-groovy]
-----
-graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
-g = graph.traversal(computer(SparkGraphComputer))
-g.V().count()
-g.V().out().out().values('name')
-----
-
-For using lambdas in Gremlin-Groovy, simply provide `:remote connect` a `TraversalSource` which leverages SparkGraphComputer.
-
-[gremlin-groovy]
-----
-graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
-g = graph.traversal(computer(SparkGraphComputer))
-:remote connect tinkerpop.hadoop graph g
-:> g.V().group().by{it.value('name')[1]}.by('name')
-----
-
-The `SparkGraphComputer` algorithm leverages Spark's caching abilities to reduce the amount of data shuffled across
-the wire on each iteration of the <<vertexprogram,`VertexProgram`>>. When the graph is loaded as a Spark RDD
-(Resilient Distributed Dataset) it is immediately cached as `graphRDD`. The `graphRDD` is a distributed adjacency
-list which encodes the vertex, its properties, and all its incident edges. On the first iteration, each vertex
-(in parallel) is passed through `VertexProgram.execute()`. This yields an output of the vertex's mutated state
-(i.e. updated compute keys -- `propertyX`) and its outgoing messages. This `viewOutgoingRDD` is then reduced to
-`viewIncomingRDD` where the outgoing messages are sent to their respective vertices. If a `MessageCombiner` exists
-for the vertex program, then messages are aggregated locally and globally to ultimately yield one incoming message
-for the vertex. This reduce sequence is the "message pass." If the vertex program does not terminate on this
-iteration, then the `viewIncomingRDD` is joined with the cached `graphRDD` and the process continues. When there
-are no more iterations, there is a final join and the resultant RDD is stripped of its edges and messages. This
-`mapReduceRDD` is cached and is processed by each <<mapreduce,`MapReduce`>> job in the
-<<graphcomputer,`GraphComputer`>> computation.
-
-image::spark-algorithm.png[width=775]
-
-[width="100%",cols="2,10",options="header"]
-|========================================================
-|Property |Description
-|gremlin.spark.graphInputRDD |A class for creating RDD's from underlying graph data, defaults to Hadoop `InputFormat`.
-|gremlin.spark.graphOutputRDD |A class for output RDD's, defaults to Hadoop `OutputFormat`.
-|gremlin.spark.persistContext |Whether to create a new `SparkContext` for every `SparkGraphComputer` or to reuse an existing one.
-|========================================================
-
-If the provider/user wishes to not use Hadoop `InputFormats`, it is possible to leverage Spark's RDD
-constructs directly. There is a `gremlin.spark.graphInputRDD` configuration that references a `Class<? extends
-InputRDD>`. An `InputRDD` provides a read method that takes a `SparkContext` and returns a graphRDD. Likewise, use
-`gremlin.spark.graphOutputRDD` and the respective `OutputRDD`.
-
-It is possible to persist the graph RDD between jobs within the `SparkContext` (e.g. SparkServer) by leveraging `PersistedOutputRDD`.
-Note that `gremlin.spark.persistContext` should be set to `true` or else the persisted RDD will be destroyed when the `SparkContext` closes.
-The persisted RDD is named by the `gremlin.hadoop.outputLocation` configuration (i.e. named in `SparkContext.getPersistedRDDs()`).
-Finally, `PersistedInputRDD` is used with respective `gremlin.hadoop.inputLocation` to retrieve the persisted RDD from the `SparkContext`.
-
-When using a persistent `Spark Context` the configuration used by the original Spark Configuration will be inherited by all threaded
-references to that Spark Context. The exception to this rule are those properties which have a specific thread local effect.
-
-.Thread Local Properties
-. spark.jobGroup.id
-. spark.job.description
-. spark.job.interruptOnCancel
-. spark.scheduler.pool
-
-Loading with BulkLoaderVertexProgram
-++++++++++++++++++++++++++++++++++++
-
-The <<bulkloadervertexprogram, BulkLoaderVertexProgram>> is a generalized bulk loader that can be used to load large
-amounts of data to and from different `Graph` implementations. The following code demonstrates how to load the
-Grateful Dead graph from HadoopGraph into TinkerGraph over Spark:
-
-[gremlin-groovy]
-----
-hdfs.copyFromLocal('data/grateful-dead.kryo', 'data/grateful-dead.kryo')
-readGraph = GraphFactory.open('conf/hadoop/hadoop-grateful-gryo.properties')
-writeGraph = 'conf/tinkergraph-gryo.properties'
-blvp = BulkLoaderVertexProgram.build().
- keepOriginalIds(false).
- writeGraph(writeGraph).create(readGraph)
-readGraph.compute(SparkGraphComputer).workers(1).program(blvp).submit().get()
-:set max-iteration 10
-graph = GraphFactory.open(writeGraph)
-g = graph.traversal()
-g.V().valueMap()
-graph.close()
-----
-
-[source,properties]
-----
-# hadoop-grateful-gryo.properties
-
-#
-# Hadoop Graph Configuration
-#
-gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
-gremlin.hadoop.graphInputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat
-gremlin.hadoop.memoryOutputFormat=org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat
-gremlin.hadoop.inputLocation=data/grateful-dead.kryo
-gremlin.hadoop.outputLocation=output
-gremlin.hadoop.deriveMemory=false
-gremlin.hadoop.jarsInDistributedCache=true
-
-#
-# SparkGraphComputer Configuration
-#
-spark.master=local[1]
-spark.executor.memory=1g
-spark.serializer=org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer
-----
-
-[source,properties]
-----
-# tinkergraph-gryo.properties
-
-gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph
-gremlin.tinkergraph.graphFormat=gryo
-gremlin.tinkergraph.graphLocation=/tmp/tinkergraph.kryo
-----
-
-IMPORTANT: The path to TinkerGraph jars needs to be included in the `HADOOP_GREMLIN_LIBS` for the above example to work.
-
-[[giraphgraphcomputer]]
-GiraphGraphComputer
-^^^^^^^^^^^^^^^^^^^
-
-[source,xml]
-----
-<dependency>
- <groupId>org.apache.tinkerpop</groupId>
- <artifactId>giraph-gremlin</artifactId>
- <version>x.y.z</version>
-</dependency>
-----
-
-image:giraph-logo.png[width=100,float=left] link:http://giraph.apache.org[Giraph] is an Apache Software Foundation
-project focused on OLAP-based graph processing. Giraph makes use of the distributed graph computing paradigm made
-popular by Google's Pregel. In Giraph, developers write "vertex programs" that get executed at each vertex in
-parallel. These programs communicate with one another in a bulk synchronous parallel (BSP) manner. This model aligns
-with TinkerPop3's `GraphComputer` API. TinkerPop3 provides an implementation of `GraphComputer` that works for Giraph
-called `GiraphGraphComputer`. Moreover, with TinkerPop3's <<mapreduce,MapReduce>>-framework, the standard
-Giraph/Pregel model is extended to support an arbitrary number of MapReduce phases to aggregate and yield results
-from the graph. Below are examples using `GiraphGraphComputer` from the <<gremlin-console,Gremlin-Console>>.
-
-WARNING: Giraph uses a large number of Hadoop counters. The default for Hadoop is 120. In `mapred-site.xml` it is
-possible to increase the limit it via the `mapreduce.job.counters.max` property. A good value to use is 1000. This
-is a cluster-wide property so be sure to restart the cluster after updating.
-
-WARNING: The maximum number of workers can be no larger than the number of map-slots in the Hadoop cluster minus 1.
-For example, if the Hadoop cluster has 4 map slots, then `giraph.maxWorkers` can not be larger than 3. One map-slot
-is reserved for the master compute node and all other slots can be allocated as workers to execute the VertexPrograms
-on the vertices of the graph.
-
-If `GiraphGraphComputer` will be used as the `GraphComputer` for `HadoopGraph` then its `lib` directory should be
-specified in `HADOOP_GREMLIN_LIBS`.
-
-[source,shell]
-export HADOOP_GREMLIN_LIBS=$HADOOP_GREMLIN_LIBS:/usr/local/gremlin-console/ext/giraph-gremlin/lib
-
-Or, the user can specify the directory in the Gremlin Console.
-
-[source,groovy]
-System.setProperty('HADOOP_GREMLIN_LIBS',System.getProperty('HADOOP_GREMLIN_LIBS') + ':' + '/usr/local/gremlin-console/ext/giraph-gremlin/lib')
-
-[gremlin-groovy]
-----
-graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
-g = graph.traversal(computer(GiraphGraphComputer))
-g.V().count()
-g.V().out().out().values('name')
-----
-
-IMPORTANT: The examples above do not use lambdas (i.e. closures in Gremlin-Groovy). This makes the traversal
-serializable and thus, able to be distributed to all machines in the Hadoop cluster. If a lambda is required in a
-traversal, then the traversal must be sent as a `String` and compiled locally at each machine in the cluster. The
-following example demonstrates the `:remote` command which allows for submitting Gremlin traversals as a `String`.
-
-[gremlin-groovy]
-----
-graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
-g = graph.traversal(computer(GiraphGraphComputer))
-:remote connect tinkerpop.hadoop graph g
-:> g.V().group().by{it.value('name')[1]}.by('name')
-result
-result.memory.runtime
-result.memory.keys()
-result.memory.get('~reducing')
-----
-
-NOTE: If the user explicitly specifies `giraph.maxWorkers` and/or `giraph.numComputeThreads` in the configuration,
-then these values will be used by Giraph. However, if these are not specified and the user never calls
-`GraphComputer.workers()` then `GiraphGraphComputer` will try to compute the number of workers/threads to use based
-on the cluster's profile.
-
-Loading with BulkLoaderVertexProgram
-++++++++++++++++++++++++++++++++++++
-
-The <<bulkloadervertexprogram, BulkLoaderVertexProgram>> is a generalized bulk loader that can be used to load
-large amounts of data to and from different `Graph` implementations. The following code demonstrates how to load
-the Grateful Dead graph from HadoopGraph into TinkerGraph over Giraph:
-
-[gremlin-groovy]
-----
-hdfs.copyFromLocal('data/grateful-dead.kryo', 'data/grateful-dead.kryo')
-readGraph = GraphFactory.open('conf/hadoop/hadoop-grateful-gryo.properties')
-writeGraph = 'conf/tinkergraph-gryo.properties'
-blvp = BulkLoaderVertexProgram.build().
- keepOriginalIds(false).
- writeGraph(writeGraph).create(readGraph)
-readGraph.compute(GiraphGraphComputer).workers(1).program(blvp).submit().get()
-:set max-iteration 10
-graph = GraphFactory.open(writeGraph)
-g = graph.traversal()
-g.V().valueMap()
-graph.close()
-----
-
-[source,properties]
-----
-# hadoop-grateful-gryo.properties
-
-#
-# Hadoop Graph Configuration
-#
-gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
-gremlin.hadoop.graphInputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat
-gremlin.hadoop.graphOutputFormat=org.apache.hadoop.mapreduce.lib.output.NullOutputFormat
-gremlin.hadoop.memoryOutputFormat=org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat
-gremlin.hadoop.inputLocation=data/grateful-dead.kryo
-gremlin.hadoop.outputLocation=output
-gremlin.hadoop.deriveMemory=false
-gremlin.hadoop.jarsInDistributedCache=true
-
-#
-# GiraphGraphComputer Configuration
-#
-giraph.minWorkers=1
-giraph.maxWorkers=1
-giraph.useOutOfCoreGraph=true
-giraph.useOutOfCoreMessages=true
-mapred.map.child.java.opts=-Xmx1024m
-mapred.reduce.child.java.opts=-Xmx1024m
-giraph.numInputThreads=4
-giraph.numComputeThreads=4
-giraph.maxMessagesInMemory=100000
-----
-
-[source,properties]
-----
-# tinkergraph-gryo.properties
-
-gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph
-gremlin.tinkergraph.graphFormat=gryo
-gremlin.tinkergraph.graphLocation=/tmp/tinkergraph.kryo
-----
-
-NOTE: The path to TinkerGraph needs to be included in the `HADOOP_GREMLIN_LIBS` for the above example to work.
-
-Input/Output Formats
-~~~~~~~~~~~~~~~~~~~~
-
-image:adjacency-list.png[width=300,float=right] Hadoop-Gremlin provides various I/O formats -- i.e. Hadoop
-`InputFormat` and `OutputFormat`. All of the formats make use of an link:http://en.wikipedia.org/wiki/Adjacency_list[adjacency list]
-representation of the graph where each "row" represents a single vertex, its properties, and its incoming and
-outgoing edges.
-
-{empty} +
-
-[[gryo-io-format]]
-Gryo I/O Format
-^^^^^^^^^^^^^^^
-
-* **InputFormat**: `org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat`
-* **OutputFormat**: `org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat`
-
-<<gryo-reader-writer,Gryo>> is a binary graph format that leverages link:https://github.com/EsotericSoftware/kryo[Kryo]
-to make a compact, binary representation of a vertex. It is recommended that users leverage Gryo given its space/time
-savings over text-based representations.
-
-NOTE: The `GryoInputFormat` is splittable.
-
-[[graphson-io-format]]
-GraphSON I/O Format
-^^^^^^^^^^^^^^^^^^^
-
-* **InputFormat**: `org.apache.tinkerpop.gremlin.hadoop.structure.io.graphson.GraphSONInputFormat`
-* **OutputFormat**: `org.apache.tinkerpop.gremlin.hadoop.structure.io.graphson.GraphSONOutputFormat`
-
-<<graphson-reader-writer,GraphSON>> is a JSON based graph format. GraphSON is a space-expensive graph format in that
-it is a text-based markup language. However, it is convenient for many developers to work with as its structure is
-simple (easy to create and parse).
-
-The data below represents an adjacency list representation of the classic TinkerGraph toy graph in GraphSON format.
-
-[source,json]
-{"id":1,"label":"person","outE":{"created":[{"id":9,"inV":3,"properties":{"weight":0.4}}],"knows":[{"id":7,"inV":2,"properties":{"weight":0.5}},{"id":8,"inV":4,"properties":{"weight":1.0}}]},"properties":{"name":[{"id":0,"value":"marko"}],"age":[{"id":1,"value":29}]}}
-{"id":2,"label":"person","inE":{"knows":[{"id":7,"outV":1,"properties":{"weight":0.5}}]},"properties":{"name":[{"id":2,"value":"vadas"}],"age":[{"id":3,"value":27}]}}
-{"id":3,"label":"software","inE":{"created":[{"id":9,"outV":1,"properties":{"weight":0.4}},{"id":11,"outV":4,"properties":{"weight":0.4}},{"id":12,"outV":6,"properties":{"weight":0.2}}]},"properties":{"name":[{"id":4,"value":"lop"}],"lang":[{"id":5,"value":"java"}]}}
-{"id":4,"label":"person","inE":{"knows":[{"id":8,"outV":1,"properties":{"weight":1.0}}]},"outE":{"created":[{"id":10,"inV":5,"properties":{"weight":1.0}},{"id":11,"inV":3,"properties":{"weight":0.4}}]},"properties":{"name":[{"id":6,"value":"josh"}],"age":[{"id":7,"value":32}]}}
-{"id":5,"label":"software","inE":{"created":[{"id":10,"outV":4,"properties":{"weight":1.0}}]},"properties":{"name":[{"id":8,"value":"ripple"}],"lang":[{"id":9,"value":"java"}]}}
-{"id":6,"label":"person","outE":{"created":[{"id":12,"inV":3,"properties":{"weight":0.2}}]},"properties":{"name":[{"id":10,"value":"peter"}],"age":[{"id":11,"value":35}]}}
-
-[[script-io-format]]
-Script I/O Format
-^^^^^^^^^^^^^^^^^
-
-* **InputFormat**: `org.apache.tinkerpop.gremlin.hadoop.structure.io.script.ScriptInputFormat`
-* **OutputFormat**: `org.apache.tinkerpop.gremlin.hadoop.structure.io.script.ScriptOutputFormat`
-
-`ScriptInputFormat` and `ScriptOutputFormat` take an arbitrary script and use that script to either read or write
-`Vertex` objects, respectively. This can be considered the most general `InputFormat`/`OutputFormat` possible in that
-Hadoop-Gremlin uses the user provided script for all reading/writing.
-
-ScriptInputFormat
-+++++++++++++++++
-
-The data below represents an adjacency list representation of the classic TinkerGraph toy graph. First line reads,
-"vertex `1`, labeled `person` having 2 property values (`marko` and `29`) has 3 outgoing edges; the first edge is
-labeled `knows`, connects the current vertex `1` with vertex `2` and has a property value `0.4`, and so on."
-
-[source]
-1:person:marko:29 knows:2:0.5,knows:4:1.0,created:3:0.4
-2:person:vadas:27
-3:project:lop:java
-4:person:josh:32 created:3:0.4,created:5:1.0
-5:project:ripple:java
-6:person:peter:35 created:3:0.2
-
-There is no corresponding `InputFormat` that can parse this particular file (or some adjacency list variant of it).
-As such, `ScriptInputFormat` can be used. With `ScriptInputFormat` a script is stored in HDFS and leveraged by each
-mapper in the Hadoop job. The script must have the following method defined:
-
-[source,groovy]
-def parse(String line, ScriptElementFactory factory) { ... }
-
-`ScriptElementFactory` provides the following 4 methods:
-
-[source,java]
-Vertex vertex(Object id); // get or create the vertex with the given id
-Vertex vertex(Object id, String label); // get or create the vertex with the given id and label
-Edge edge(Vertex out, Vertex in); // create an edge between the two given vertices
-Edge edge(Vertex out, Vertex in, String label); // create an edge between the two given vertices using the given label
-
-An appropriate `parse()` for the above adjacency list file is:
-
-[source,groovy]
-def parse(line, factory) {
- def parts = line.split(/ /)
- def (id, label, name, x) = parts[0].split(/:/).toList()
- def v1 = factory.vertex(id, label)
- if (name != null) v1.property('name', name) // first value is always the name
- if (x != null) {
- // second value depends on the vertex label; it's either
- // the age of a person or the language of a project
- if (label.equals('project')) v1.property('lang', x)
- else v1.property('age', Integer.valueOf(x))
- }
- if (parts.length == 2) {
- parts[1].split(/,/).grep { !it.isEmpty() }.each {
- def (eLabel, refId, weight) = it.split(/:/).toList()
- def v2 = factory.vertex(refId)
- def edge = factory.edge(v1, v2, eLabel)
- edge.property('weight', Double.valueOf(weight))
- }
- }
- return v1
-}
-
-The resultant `Vertex` denotes whether the line parsed yielded a valid Vertex. As such, if the line is not valid
-(e.g. a comment line, a skip line, etc.), then simply return `null`.
-
-ScriptOutputFormat Support
-++++++++++++++++++++++++++
-
-The principle above can also be used to convert a vertex to an arbitrary `String` representation that is ultimately
-streamed back to a file in HDFS. This is the role of `ScriptOutputFormat`. `ScriptOutputFormat` requires that the
-provided script maintains a method with the following signature:
-
-[source,groovy]
-def stringify(Vertex vertex) { ... }
-
-An appropriate `stringify()` to produce output in the same format that was shown in the `ScriptInputFormat` sample is:
-
-[source,groovy]
-def stringify(vertex) {
- def v = vertex.values('name', 'age', 'lang').inject(vertex.id(), vertex.label()).join(':')
- def outE = vertex.outE().map {
- def e = it.get()
- e.values('weight').inject(e.label(), e.inV().next().id()).join(':')
- }.join(',')
- return [v, outE].join('\t')
-}
-
-Interacting with HDFS
-~~~~~~~~~~~~~~~~~~~~~
-
-The distributed file system of Hadoop is called link:http://en.wikipedia.org/wiki/Apache_Hadoop#Hadoop_distributed_file_system[HDFS].
-The results of any OLAP operation are stored in HDFS accessible via `hdfs`.
-
-[gremlin-groovy]
-----
-graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
-g = graph.traversal(computer(SparkGraphComputer))
-:remote connect tinkerpop.hadoop graph g
-:> g.V().group().by{it.value('name')[1]}.by('name')
-hdfs.ls()
-hdfs.ls('output')
-hdfs.ls('output/~reducing')
-hdfs.head('output/~reducing', ObjectWritable)
-----
-
-A list of the HDFS methods available are itemized below. Note that these methods are also available for the 'local' variable:
-
-[width="100%",cols="13,10",options="header"]
-|=========================================================
-| Method| Description
-|hdfs.ls(String path)| List the contents of the supplied directory.
-|hdfs.cp(String from, String to)| Copy the specified path to the specified path.
-|hdfs.exists(String path)| Whether the specified path exists.
-|hdfs.rm(String path)| Remove the specified path.
-|hdfs.rmr(String path)| Remove the specified path and its contents recurssively.
-|hdfs.copyToLocal(String from, String to)| Copy the specified HDFS path to the specified local path.
-|hdfs.copyFromLocal(String from, String to)| Copy the specified local path to the specified HDFS path.
-|hdfs.mergeToLocal(String from, String to)| Merge the files in path to the specified local path.
-|hdfs.head(String path)| Display the data in the path as text.
-|hdfs.head(String path, int lineCount)| Text display only the first `lineCount`-number of lines in the path.
-|hdfs.head(String path, int totalKeyValues, Class<Writable> writableClass)| Display the path interpreting the key values as respective writable.
-|=========================================================
-
-A Command Line Example
-~~~~~~~~~~~~~~~~~~~~~~
-
-image::pagerank-logo.png[width=300]
-
-The classic link:http://en.wikipedia.org/wiki/PageRank[PageRank] centrality algorithm can be executed over the
-TinkerPop graph from the command line using `GiraphGraphComputer`.
-
-WARNING: Be sure that the `HADOOP_GREMLIN_LIBS` references the location `lib` directory of the respective
-`GraphComputer` engine being used or else the requisite dependencies will not be uploaded to the Hadoop cluster.
-
-[source,text]
-----
-$ hdfs dfs -copyFromLocal data/tinkerpop-modern.json tinkerpop-modern.json
-$ hdfs dfs -ls
-Found 2 items
--rw-r--r-- 1 marko supergroup 2356 2014-07-28 13:00 /user/marko/tinkerpop-modern.json
-$ hadoop jar target/giraph-gremlin-x.y.z-job.jar org.apache.tinkerpop.gremlin.giraph.process.computer.GiraphGraphComputer ../hadoop-gremlin/conf/hadoop-graphson.properties
-15/09/11 08:02:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
-15/09/11 08:02:11 INFO computer.GiraphGraphComputer: HadoopGremlin(Giraph): PageRankVertexProgram[alpha=0.85,iterations=30]
-15/09/11 08:02:12 INFO mapreduce.JobSubmitter: number of splits:3
-15/09/11 08:02:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1441915907347_0028
-15/09/11 08:02:12 INFO impl.YarnClientImpl: Submitted application application_1441915907347_0028
-15/09/11 08:02:12 INFO job.GiraphJob: Tracking URL: http://markos-macbook:8088/proxy/application_1441915907347_0028/
-15/09/11 08:02:12 INFO job.GiraphJob: Waiting for resources... Job will start only when it gets all 3 mappers
-15/09/11 08:03:54 INFO mapreduce.Job: Running job: job_1441915907347_0028
-15/09/11 08:03:55 INFO mapreduce.Job: Job job_1441915907347_0028 running in uber mode : false
-15/09/11 08:03:55 INFO mapreduce.Job: map 33% reduce 0%
-15/09/11 08:03:57 INFO mapreduce.Job: map 67% reduce 0%
-15/09/11 08:04:01 INFO mapreduce.Job: map 100% reduce 0%
-15/09/11 08:06:17 INFO mapreduce.Job: Job job_1441915907347_0028 completed successfully
-15/09/11 08:06:17 INFO mapreduce.Job: Counters: 80
- File System Counters
- FILE: Number of bytes read=0
- FILE: Number of bytes written=483918
- FILE: Number of read operations=0
- FILE: Number of large read operations=0
- FILE: Number of write operations=0
- HDFS: Number of bytes read=1465
- HDFS: Number of bytes written=1760
- HDFS: Number of read operations=39
- HDFS: Number of large read operations=0
- HDFS: Number of write operations=20
- Job Counters
- Launched map tasks=3
- Other local map tasks=3
- Total time spent by all maps in occupied slots (ms)=458105
- Total time spent by all reduces in occupied slots (ms)=0
- Total time spent by all map tasks (ms)=458105
- Total vcore-seconds taken by all map tasks=458105
- Total megabyte-seconds taken by all map tasks=469099520
- Map-Reduce Framework
- Map input records=3
- Map output records=0
- Input split bytes=132
- Spilled Records=0
- Failed Shuffles=0
- Merged Map outputs=0
- GC time elapsed (ms)=1594
- CPU time spent (ms)=0
- Physical memory (bytes) snapshot=0
- Virtual memory (bytes) snapshot=0
- Total committed heap usage (bytes)=527958016
- Giraph Stats
- Aggregate edges=0
- Aggregate finished vertices=0
- Aggregate sent message message bytes=13535
- Aggregate sent messages=186
- Aggregate vertices=6
- Current master task partition=0
- Current workers=2
- Last checkpointed superstep=0
- Sent message bytes=438
- Sent messages=6
- Superstep=31
- Giraph Timers
- Initialize (ms)=2996
- Input superstep (ms)=5209
- Setup (ms)=59
- Shutdown (ms)=9324
- Superstep 0 GiraphComputation (ms)=3861
- Superstep 1 GiraphComputation (ms)=4027
- Superstep 10 GiraphComputation (ms)=4000
- Superstep 11 GiraphComputation (ms)=4004
- Superstep 12 GiraphComputation (ms)=3999
- Superstep 13 GiraphComputation (ms)=4000
- Superstep 14 GiraphComputation (ms)=4005
- Superstep 15 GiraphComputation (ms)=4003
- Superstep 16 GiraphComputation (ms)=4001
- Superstep 17 GiraphComputation (ms)=4007
- Superstep 18 GiraphComputation (ms)=3998
- Superstep 19 GiraphComputation (ms)=4006
- Superstep 2 GiraphComputation (ms)=4007
- Superstep 20 GiraphComputation (ms)=3996
- Superstep 21 GiraphComputation (ms)=4006
- Superstep 22 GiraphComputation (ms)=4002
- Superstep 23 GiraphComputation (ms)=3998
- Superstep 24 GiraphComputation (ms)=4003
- Superstep 25 GiraphComputation (ms)=4001
- Superstep 26 GiraphComputation (ms)=4003
- Superstep 27 GiraphComputation (ms)=4005
- Superstep 28 GiraphComputation (ms)=4002
- Superstep 29 GiraphComputation (ms)=4001
- Superstep 3 GiraphComputation (ms)=3988
- Superstep 30 GiraphComputation (ms)=4248
- Superstep 4 GiraphComputation (ms)=4010
- Superstep 5 GiraphComputation (ms)=3998
- Superstep 6 GiraphComputation (ms)=3996
- Superstep 7 GiraphComputation (ms)=4005
- Superstep 8 GiraphComputation (ms)=4009
- Superstep 9 GiraphComputation (ms)=3994
- Total (ms)=138788
- File Input Format Counters
- Bytes Read=0
- File Output Format Counters
- Bytes Written=0
-$ hdfs dfs -cat output/~g/*
-{"id":1,"label":"person","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":39,"value":0.15000000000000002}],"name":[{"id":0,"value":"marko"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":10,"value":3.0}],"age":[{"id":1,"value":29}]}}
-{"id":5,"label":"software","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":35,"value":0.23181250000000003}],"name":[{"id":8,"value":"ripple"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":6,"value":0.0}],"lang":[{"id":9,"value":"java"}]}}
-{"id":3,"label":"software","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":39,"value":0.4018125}],"name":[{"id":4,"value":"lop"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":10,"value":0.0}],"lang":[{"id":5,"value":"java"}]}}
-{"id":4,"label":"person","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":39,"value":0.19250000000000003}],"name":[{"id":6,"value":"josh"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":10,"value":2.0}],"age":[{"id":7,"value":32}]}}
-{"id":2,"label":"person","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":35,"value":0.19250000000000003}],"name":[{"id":2,"value":"vadas"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":6,"value":0.0}],"age":[{"id":3,"value":27}]}}
-{"id":6,"label":"person","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":35,"value":0.15000000000000002}],"name":[{"id":10,"value":"peter"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":6,"value":1.0}],"age":[{"id":11,"value":35}]}}
-----
-
-Vertex 4 ("josh") is isolated below:
-
-[source,js]
-----
-{
- "id":4,
- "label":"person",
- "properties": {
- "gremlin.pageRankVertexProgram.pageRank":[{"id":39,"value":0.19250000000000003}],
- "name":[{"id":6,"value":"josh"}],
- "gremlin.pageRankVertexProgram.edgeCount":[{"id":10,"value":2.0}],
- "age":[{"id":7,"value":32}]}
- }
-}
-----
-
-Hadoop-Gremlin for Graph System Providers
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Hadoop-Gremlin is centered around `InputFormats` and `OutputFormats`. If a 3rd-party graph system provider wishes to
-leverage Hadoop-Gremlin (and its respective `GraphComputer` engines), then they need to provide, at minimum, a
-Hadoop2 `InputFormat<NullWritable,VertexWritable>` for their graph system. If the provider wishes to persist computed
-results back to their graph system (and not just to HDFS via a `FileOutputFormat`), then a graph system specific
-`OutputFormat<NullWritable,VertexWritable>` must be developed as well.
-
-Conceptually, `HadoopGraph` is a wrapper around a `Configuration` object. There is no "data" in the `HadoopGraph` as
-the `InputFormat` specifies where and how to get the graph data at OLAP (and OLTP) runtime. Thus, `HadoopGraph` is a
-small object with little overhead. Graph system providers should realize `HadoopGraph` as the gateway to the OLAP
-features offered by Hadoop-Gremlin. For example, a graph system specific `Graph.compute(Class<? extends GraphComputer>
-graphComputerClass)`-method may look as follows:
-
-[source,java]
-----
-public <C extends GraphComputer> C compute(final Class<C> graphComputerClass) throws IllegalArgumentException {
- try {
- if (AbstractHadoopGraphComputer.class.isAssignableFrom(graphComputerClass))
- return graphComputerClass.getConstruc
<TRUNCATED>
[06/22] incubator-tinkerpop git commit: Made subdirectories for
various "books" in the docs.
Posted by sp...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/reference/the-traversal.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/reference/the-traversal.asciidoc b/docs/src/reference/the-traversal.asciidoc
new file mode 100644
index 0000000..0b82823
--- /dev/null
+++ b/docs/src/reference/the-traversal.asciidoc
@@ -0,0 +1,2378 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+[[traversal]]
+The Traversal
+=============
+
+image::gremlin-running.png[width=125]
+
+At the most general level there is `Traversal<S,E>` which implements `Iterator<E>`, where the `S` stands for start and
+the `E` stands for end. A traversal is composed of four primary components:
+
+ . `Step<S,E>`: an individual function applied to `S` to yield `E`. Steps are chained within a traversal.
+ . `TraversalStrategy`: interceptor methods to alter the execution of the traversal (e.g. query re-writing).
+ . `TraversalSideEffects`: key/value pairs that can be used to store global information about the traversal.
+ . `Traverser<T>`: the object propagating through the `Traversal` currently representing an object of type `T`.
+
+The classic notion of a graph traversal is provided by `GraphTraversal<S,E>` which extends `Traversal<S,E>`.
+`GraphTraversal` provides an interpretation of the graph data in terms of vertices, edges, etc. and thus, a graph
+traversal link:http://en.wikipedia.org/wiki/Domain-specific_language[DSL].
+
+IMPORTANT: The underlying `Step` implementations provided by TinkerPop should encompass most of the functionality
+required by a DSL author. It is important that DSL authors leverage the provided steps as then the common optimization
+and decoration strategies can reason on the underlying traversal sequence. If new steps are introduced, then common
+traversal strategies may not function properly.
+
+[[graph-traversal-steps]]
+Graph Traversal Steps
+---------------------
+
+image::step-types.png[width=650]
+
+A `GraphTraversal<S,E>` is spawned from a `GraphTraversalSource`. It can also be spawned anonymously (i.e. empty)
+via `__`. A graph traversal is composed of an ordered list of steps. All the steps provided by `GraphTraversal`
+inherit from the more general forms diagrammed above. A list of all the steps (and their descriptions) are provided
+in the TinkerPop3 link:http://tinkerpop.incubator.apache.org/javadocs/3.0.2-incubating/core/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversal.html[GraphTraversal JavaDoc].
+The following subsections will demonstrate the GraphTraversal steps using the <<gremlin-console,Gremlin Console>>.
+
+NOTE: To reduce the verbosity of the expression, it is good to
+`import static org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__.*`. This way, instead of doing `__.inE()`
+for an anonymous traversal, it is possible to simply write `inE()`. Be aware of language-specific reserved keywords
+when using anonymous traversals. For example, `in` and `as` are reserved keywords in Groovy, therefore you must use
+the verbose syntax `__.in()` and `__.as()` to avoid collisions.
+
+[[lambda-steps]]
+Lambda Steps
+~~~~~~~~~~~~
+
+CAUTION: Lambda steps are presented for educational purposes as they represent the foundational constructs of the
+Gremlin language. In practice, lambda steps should be avoided and traversal verification strategies exist to disallow t
+heir use unless explicitly "turned off." For more information on the problems with lambdas, please read
+<<a-note-on-lambdas,A Note on Lambdas>>.
+
+There are four generic steps by which all other specific steps described later extend.
+
+[width="100%",cols="10,12",options="header"]
+|=========================================================
+| Step| Description
+| `map(Function<Traverser<S>, E>)` | map the traverser to some object of type `E` for the next step to process.
+| `flatMap(Function<Traverser<S>, Iterator<E>>)` | map the traverser to an iterator of `E` objects that are streamed to the next step.
+| `filter(Predicate<Traverser<S>>)` | map the traverser to either true or false, where false will not pass the traverser to the next step.
+| `sideEffect(Consumer<Traverser<S>>)` | perform some operation on the traverser and pass it to the next step.
+| `branch(Function<Traverser<S>,M>)` | split the traverser to all the traversals indexed by the `M` token.
+|=========================================================
+
+The `Traverser<S>` object provides access to:
+
+ . The current traversed `S` object -- `Traverser.get()`.
+ . The current path traversed by the traverser -- `Traverser.path()`.
+ .. A helper shorthand to get a particular path-history object -- `Traverser.path(String) == Traverser.path().get(String)`.
+ . The number of times the traverser has gone through the current loop -- `Traverser.loops()`.
+ . The number of objects represented by this traverser -- `Traverser.bulk()`.
+ . The local data structure associated with this traverser -- `Traverser.sack()`.
+ . The side-effects associated with the traversal -- `Traverser.sideEffects()`.
+ .. A helper shorthand to get a particular side-effect -- `Traverser.sideEffect(String) == Traverser.sideEffects().get(String)`.
+
+image:map-lambda.png[width=150,float=right]
+[gremlin-groovy,modern]
+----
+g.V(1).out().values('name') <1>
+g.V(1).out().map {it.get().value('name')} <2>
+----
+
+<1> An outgoing traversal from vertex 1 to the name values of the adjacent vertices.
+<2> The same operation, but using a lambda to access the name property values.
+
+image:filter-lambda.png[width=160,float=right]
+[gremlin-groovy,modern]
+----
+g.V().filter {it.get().label() == 'person'} <1>
+g.V().hasLabel('person') <2>
+----
+
+<1> A filter that only allows the vertex to pass if it has an age-property.
+<2> The more specific `has()`-step is implemented as a `filter()` with respective predicate.
+
+
+image:side-effect-lambda.png[width=175,float=right]
+[gremlin-groovy,modern]
+----
+g.V().hasLabel('person').sideEffect(System.out.&println) <1>
+----
+
+<1> Whatever enters `sideEffect()` is passed to the next step, but some intervening process can occur.
+
+image:branch-lambda.png[width=180,float=right]
+[gremlin-groovy,modern]
+----
+g.V().branch(values('name')).
+ option('marko', values('age')).
+ option(none, values('name')) <1>
+g.V().choose(has('name','marko'),
+ values('age'),
+ values('name')) <2>
+----
+
+<1> If the vertex is "marko", get his age, else get the name of the vertex.
+<2> The more specific boolean-based `choose()`-step is implemented as a `branch()`.
+
+[[addedge-step]]
+AddEdge Step
+~~~~~~~~~~~~
+
+link:http://en.wikipedia.org/wiki/Automated_reasoning[Reasoning] is the process of making explicit what is implicit
+in the data. What is explicit in a graph are the objects of the graph -- i.e. vertices and edges. What is implicit
+in the graph is the traversal. In other words, traversals expose meaning where the meaning is determined by the
+traversal definition. For example, take the concept of a "co-developer." Two people are co-developers if they have
+worked on the same project together. This concept can be represented as a traversal and thus, the concept of
+"co-developers" can be derived. Moreover, what was once implicit can be made explicit via the `addE()`-step
+(*map*/*sideEffect*).
+
+image::addedge-step.png[width=450]
+
+[gremlin-groovy,modern]
+----
+g.V(1).as('a').out('created').in('created').where(neq('a')).
+ addE('co-developer').from('a').property('year',2009) <1>
+g.V(3,4,5).aggregate('x').has('name','josh').as('a').
+ select('x').unfold().hasLabel('software').addE('createdBy').to('a') <2>
+g.V().as('a').out('created').addE('createdBy').to('a').property('acl','public') <3>
+g.V(1).as('a').out('knows').
+ addE('livesNear').from('a').property('year',2009).
+ inV().inE('livesNear').values('year') <4>
+g.V().match(
+ __.as('a').out('knows').as('b'),
+ __.as('a').out('created').as('c'),
+ __.as('b').out('created').as('c')).
+ addE('friendlyCollaborator').from('a').to('b').
+ property(id,13).property('project',select('c').values('name')) <5>
+g.E(13).valueMap()
+----
+
+<1> Add a co-developer edge with a year-property between marko and his collaborators.
+<2> Add incoming createdBy edges from the josh-vertex to the lop- and ripple-vertices.
+<3> Add an inverse createdBy edge for all created edges.
+<4> The newly created edge is a traversable object.
+<5> Two arbitrary bindings in a traversal can be joined `from()`->`to()`, where `id` can be provided for graphs that
+supports user provided ids.
+
+[[addvertex-step]]
+AddVertex Step
+~~~~~~~~~~~~~~
+
+The `addV()`-step is used to add vertices to the graph (*map*/*sideEffect*). For every incoming object, a vertex is
+created. Moreover, `GraphTraversalSource` maintains an `addV()` method.
+
+[gremlin-groovy,modern]
+----
+g.addV('person').property('name','stephen')
+g.V().values('name')
+g.V().outE('knows').addV().property('name','nothing')
+g.V().has('name','nothing')
+g.V().has('name','nothing').bothE()
+----
+
+[[addproperty-step]]
+AddProperty Step
+~~~~~~~~~~~~~~~~
+
+The `property()`-step is used to add properties to the elements of the graph (*sideEffect*). Unlike `addV()` and
+`addE()`, `property()` is a full sideEffect step in that it does not return the property it created, but the element
+that streamed into it. Moreover, if `property()` follows an `addV()` or `addE()`, then it is "folded" into the
+previous step to enable vertex and edge creation with all its properties in one creation operation.
+
+[gremlin-groovy,modern]
+----
+g.V(1).property('country','usa')
+g.V(1).property('city','santa fe').property('state','new mexico').valueMap()
+g.V(1).property(list,'age',35) <1>
+g.V(1).valueMap()
+g.V(1).property('friendWeight',outE('knows').values('weight').sum(),'acl','private') <2>
+g.V(1).properties('friendWeight').valueMap() <3>
+----
+
+<1> For vertices, a cardinality can be provided for <<vertex properties,vertex-properties>>.
+<2> It is possible to select the property value (as well as key) via a traversal.
+<3> For vertices, the `property()`-step can add meta-properties.
+
+
+[[aggregate-step]]
+Aggregate Step
+~~~~~~~~~~~~~~
+
+image::aggregate-step.png[width=800]
+
+The `aggregate()`-step (*sideEffect*) is used to aggregate all the objects at a particular point of traversal into a
+`Collection`. The step uses link:http://en.wikipedia.org/wiki/Eager_evaluation[eager evaluation] in that no objects
+continue on until all previous objects have been fully aggregated (as opposed to <<store-step,`store()`>> which
+link:http://en.wikipedia.org/wiki/Lazy_evaluation[lazily] fills a collection). The eager evaluation nature is crucial
+in situations where everything at a particular point is required for future computation. An example is provided below.
+
+[gremlin-groovy,modern]
+----
+g.V(1).out('created') <1>
+g.V(1).out('created').aggregate('x') <2>
+g.V(1).out('created').aggregate('x').in('created') <3>
+g.V(1).out('created').aggregate('x').in('created').out('created') <4>
+g.V(1).out('created').aggregate('x').in('created').out('created').
+ where(without('x')).values('name') <5>
+----
+
+<1> What has marko created?
+<2> Aggregate all his creations.
+<3> Who are marko's collaborators?
+<4> What have marko's collaborators created?
+<5> What have marko's collaborators created that he hasn't created?
+
+In link:http://en.wikipedia.org/wiki/Recommender_system[recommendation systems], the above pattern is used:
+
+ "What has userA liked? Who else has liked those things? What have they liked that userA hasn't already liked?"
+
+Finally, `aggregate()`-step can be modulated via `by()`-projection.
+
+[gremlin-groovy,modern]
+----
+g.V().out('knows').aggregate('x').cap('x')
+g.V().out('knows').aggregate('x').by('name').cap('x')
+----
+
+[[and-step]]
+And Step
+~~~~~~~~
+
+The `and()`-step ensures that all provided traversals yield a result (*filter*). Please see <<or-step,`or()`>> for or-semantics.
+
+[gremlin-groovy,modern]
+----
+g.V().and(
+ outE('knows'),
+ values('age').is(lt(30))).
+ values('name')
+----
+
+The `and()`-step can take an arbitrary number of traversals. All traversals must produce at least one output for the
+original traverser to pass to the next step.
+
+An link:http://en.wikipedia.org/wiki/Infix_notation[infix notation] can be used as well. Though, with infix notation,
+only two traversals can be and'd together.
+
+[gremlin-groovy,modern]
+----
+g.V().where(outE('created').and().outE('knows')).values('name')
+----
+
+[[as-step]]
+As Step
+~~~~~~~
+
+The `as()`-step is not a real step, but a "step modulator" similar to <<by-step,`by()`>> and <<option-step,`option()`>>.
+With `as()`, it is possible to provide a label to the step that can later be accessed by steps and data structures
+that make use of such labels -- e.g., <<select-step,`select()`>>, <<match-step,`match()`>>, and path.
+
+[gremlin-groovy,modern]
+----
+g.V().as('a').out('created').as('b').select('a','b') <1>
+g.V().as('a').out('created').as('b').select('a','b').by('name') <2>
+----
+
+<1> Select the objects labeled "a" and "b" from the path.
+<2> Select the objects labeled "a" and "b" from the path and, for each object, project its name value.
+
+A step can have any number of labels associated with it. This is useful for referencing the same step multiple times in a future step.
+
+[gremlin-groovy,modern]
+----
+g.V().hasLabel('software').as('a','b','c').
+ select('a','b','c').
+ by('name').
+ by('lang').
+ by(__.in('created').values('name').fold())
+----
+
+[[barrier-step]]
+Barrier Step
+~~~~~~~~~~~~
+
+The `barrier()`-step (*barrier*) turns the the lazy traversal pipeline into a bulk-synchronous pipeline. This step is
+useful in the following situations:
+
+ * When everything prior to `barrier()` needs to be executed before moving onto the steps after the `barrier()` (i.e. ordering).
+ * When "stalling" the traversal may lead to a "bulking optimization" in traversals that repeatedly touch many of the same elements (i.e. optimizing).
+
+[gremlin-groovy,modern]
+----
+g.V().sideEffect{println "first: ${it}"}.sideEffect{println "second: ${it}"}.iterate()
+g.V().sideEffect{println "first: ${it}"}.barrier().sideEffect{println "second: ${it}"}.iterate()
+----
+
+The theory behind a "bulking optimization" is simple. If there are one million traversers at vertex 1, then there is
+no need to calculate one million `both()`-computations. Instead, represent those one million traversers as a single
+traverser with a `Traverser.bulk()` equal to one million and execute `both()` once. A bulking optimization example is
+made more salient on a larger graph. Therefore, the example below leverages the <<grateful-dead,Grateful Dead graph>>.
+
+[gremlin-groovy]
+----
+graph = TinkerGraph.open()
+graph.io(graphml()).readGraph('data/grateful-dead.xml')
+g = graph.traversal(standard())
+clockWithResult(1){g.V().both().both().both().count().next()} <1>
+clockWithResult(1){g.V().repeat(both()).times(3).count().next()} <2>
+clockWithResult(1){g.V().both().barrier().both().barrier().both().barrier().count().next()} <3>
+----
+
+<1> A non-bulking traversal where each traverser is processed.
+<2> Each traverser entering `repeat()` has its recursion bulked.
+<3> A bulking traversal where implicit traversers are not processed.
+
+If `barrier()` is provided an integer argument, then the barrier will only hold `n`-number of unique traversers in its
+barrier before draining the aggregated traversers to the next step. This is useful in the aforementioned bulking
+optimization scenario, but reduces the risk of an out-of-memory exception.
+
+The non-default `LazyBarrierStrategy` inserts `barrier()`-steps in a traversal where appropriate in order to gain the
+"bulking optimization."
+
+[gremlin-groovy]
+----
+graph = TinkerGraph.open()
+graph.io(graphml()).readGraph('data/grateful-dead.xml')
+g = graph.traversal(GraphTraversalSource.build().with(LazyBarrierStrategy.instance()).engine(StandardTraversalEngine.build()))
+clockWithResult(1){g.V().both().both().both().count().next()}
+g.V().both().both().both().count().iterate().toString() <1>
+----
+
+<1> With `LazyBarrierStrategy` activated, `barrier()` steps are automatically inserted where appropriate.
+
+[[by-step]]
+By Step
+~~~~~~~
+
+The `by()`-step is not an actual step, but instead is a "step-modulator" similar to <<as-step,`as()`>> and
+<<option-step,`option()`>>. If a step is able to accept traversals, functions, comparators, etc. then `by()` is the
+means by which they are added. The general pattern is `step().by()...by()`. Some steps can only accept one `by()`
+while others can take an arbitrary amount.
+
+[gremlin-groovy,modern]
+----
+g.V().group().by(bothE().count()) <1>
+g.V().group().by(bothE().count()).by('name') <2>
+g.V().group().by(bothE().count()).by(count()) <3>
+----
+
+<1> `by(outE().count())` will group the elements by their edge count (*traversal*).
+<2> `by('name')` will process the grouped elements by their name (*element property projection*).
+<3> `by(count())` will count the number of elements in each group (*traversal*).
+
+[cap-step]]
+Cap Step
+~~~~~~~~
+
+The `cap()`-step (*barrier*) iterates the traversal up to itself and emits the sideEffect referenced by the provided
+key. If multiple keys are provided, then a `Map<String,Object>` of sideEffects is emitted.
+
+[gremlin-groovy,modern]
+----
+g.V().groupCount('a').by(label).cap('a') <1>
+g.V().groupCount('a').by(label).groupCount('b').by(outE().count()).cap('a','b') <2>
+----
+
+<1> Group and count verticies by their label. Emit the side effect labeled 'a', which is the group count by label.
+<2> Same as statement 1, but also emit the side effect labeled 'b' which groups vertices by the number of out edges.
+
+[[coalesce-step]]
+Coalesce Step
+~~~~~~~~~~~~~
+
+The `coalesce()`-step evaluates the provided traversals in order and returns the first traversal that emits at
+least one element.
+
+[gremlin-groovy,modern]
+----
+g.V(1).coalesce(outE('knows'), outE('created')).inV().path().by('name').by(label)
+g.V(1).coalesce(outE('created'), outE('knows')).inV().path().by('name').by(label)
+g.V(1).next().property('nickname', 'okram')
+g.V().hasLabel('person').coalesce(values('nickname'), values('name'))
+----
+
+[[count-step]]
+Count Step
+~~~~~~~~~~
+
+image::count-step.png[width=195]
+
+The `count()`-step (*map*) counts the total number of represented traversers in the streams (i.e. the bulk count).
+
+[gremlin-groovy,modern]
+----
+g.V().count()
+g.V().hasLabel('person').count()
+g.V().hasLabel('person').outE('created').count().path() <1>
+g.V().hasLabel('person').outE('created').count().map {it.get() * 10}.path() <2>
+----
+
+<1> `count()`-step is a <<a-note-on-barrier-steps,reducing barrier step>> meaning that all of the previous traversers are folded into a new traverser.
+<2> The path of the traverser emanating from `count()` starts at `count()`.
+
+IMPORTANT: `count(local)` counts the current, local object (not the objects in the traversal stream). This works for
+`Collection`- and `Map`-type objects. For any other object, a count of 1 is returned.
+
+[[choose-step]]
+Choose Step
+~~~~~~~~~~~
+
+image::choose-step.png[width=700]
+
+The `choose()`-step (*branch*) routes the current traverser to a particular traversal branch option. With `choose()`,
+it is possible to implement if/else-based semantics as well as more complicated selections.
+
+[gremlin-groovy,modern]
+----
+g.V().hasLabel('person').
+ choose(values('age').is(lte(30)),
+ __.in(),
+ __.out()).values('name') <1>
+g.V().hasLabel('person').
+ choose(values('age')).
+ option(27, __.in()).
+ option(32, __.out()).values('name') <2>
+----
+
+<1> If the traversal yields an element, then do `in`, else do `out` (i.e. true/false-based option selection).
+<2> Use the result of the traversal as a key to the map of traversal options (i.e. value-based option selection).
+
+However, note that `choose()` can have an arbitrary number of options and moreover, can take an anonymous traversal as its choice function.
+
+[gremlin-groovy,modern]
+----
+g.V().hasLabel('person').
+ choose(values('name')).
+ option('marko', values('age')).
+ option('josh', values('name')).
+ option('vadas', valueMap()).
+ option('peter', label())
+----
+
+The `choose()`-step can leverage the `Pick.none` option match. For anything that does not match a specified option, the `none`-option is taken.
+
+[gremlin-groovy,modern]
+----
+g.V().hasLabel('person').
+ choose(values('name')).
+ option('marko', values('age')).
+ option(none, values('name'))
+----
+
+[[coin-step]]
+Coin Step
+~~~~~~~~~
+
+To randomly filter out a traverser, use the `coin()`-step (*filter*). The provided double argument biases the "coin toss."
+
+[gremlin-groovy,modern]
+----
+g.V().coin(0.5)
+g.V().coin(0.0)
+g.V().coin(1.0)
+----
+
+[[constant-step]]
+Constant Step
+~~~~~~~~~~~~~
+
+To specify a constant value for a traverser, use the `constant()`-step (*map*). This is often useful with conditional
+steps like <<choose-step,`choose()`-step>> or <<coalesce-step,`coalesce()`-step>>.
+
+[gremlin-groovy,modern]
+----
+g.V().choose(__.hasLabel('person'),
+ __.values('name'),
+ __.constant('inhuman')) <1>
+g.V().coalesce(
+ __.hasLabel('person').values('name'),
+ __.constant('inhuman')) <2>
+----
+
+<1> Show the names of people, but show "inhuman" for other vertices.
+<2> Same as statement 1 (unless there is a person vertex with no name).
+
+[[cyclicpath-step]]
+CyclicPath Step
+~~~~~~~~~~~~~~~
+
+image::cyclicpath-step.png[width=400]
+
+Each traverser maintains its history through the traversal over the graph -- i.e. its <<path-data-structure,path>>.
+If it is important that the traverser repeat its course, then `cyclic()`-path should be used (*filter*). The step
+analyzes the path of the traverser thus far and if there are any repeats, the traverser is filtered out over the
+traversal computation. If non-cyclic behavior is desired, see <<simplepath-step,`simplePath()`>>.
+
+[gremlin-groovy,modern]
+----
+g.V(1).both().both()
+g.V(1).both().both().cyclicPath()
+g.V(1).both().both().cyclicPath().path()
+----
+
+[[dedup-step]]
+Dedup Step
+~~~~~~~~~~
+
+With `dedup()`-step (*filter*), repeatedly seen objects are removed from the traversal stream. Note that if a
+traverser's bulk is greater than 1, then it is set to 1 before being emitted.
+
+[gremlin-groovy,modern]
+----
+g.V().values('lang')
+g.V().values('lang').dedup()
+g.V(1).repeat(bothE('created').dedup().otherV()).emit().path() <1>
+----
+
+<1> Traverse all `created` edges, but don't touch any edge twice.
+
+If a by-step modulation is provided to `dedup()`, then the object is processed accordingly prior to determining if it
+has been seen or not.
+
+[gremlin-groovy,modern]
+----
+g.V().valueMap(true, 'name')
+g.V().dedup().by(label).values('name')
+----
+
+Finally, if `dedup()` is provided an array of strings, then it will ensure that the de-duplication is not with respect
+to the current traverser object, but to the path history of the traverser.
+
+[gremlin-groovy,modern]
+----
+g.V().as('a').out('created').as('b').in('created').as('c').select('a','b','c')
+g.V().as('a').out('created').as('b').in('created').as('c').dedup('a','b').select('a','b','c') <1>
+----
+
+<1> If the current `a` and `b` combination has been seen previously, then filter the traverser.
+
+[[drop-step]]
+Drop Step
+~~~~~~~~~
+
+The `drop()`-step (*filter*/*sideEffect*) is used to remove element and properties from the graph (i.e. remove). It
+is a filter step because the traversal yields no outgoing objects.
+
+[gremlin-groovy,modern]
+----
+g.V().outE().drop()
+g.E()
+g.V().properties('name').drop()
+g.V().valueMap()
+g.V().drop()
+g.V()
+----
+
+[[explain-step]]
+Explain Step
+~~~~~~~~~~~~
+
+The `explain()`-step (*sideEffect*) will return a `TraversalExplanation`. A traversal explanation details how the
+traversal (prior to `explain()`) will be compiled given the registered <<traversalstrategy,traversal strategies>>.
+A `TraversalExplanation` has a `toString()` representation with 3-columns. The first column is the
+traversal strategy being applied. The second column is the traversal strategy category: [D]ecoration, [O]ptimization,
+[P]rovider optimization, [F]inalization, and [V]erification. Finally, the third column is the state of the traversal
+post strategy application. The final traversal is the resultant execution plan.
+
+[gremlin-groovy,modern]
+----
+g.V().outE().identity().inV().count().is(gt(5)).explain()
+----
+
+For traversal profiling information, please see <<profile-step,`profile()`>>-step.
+
+[[fold-step]]
+Fold Step
+~~~~~~~~~
+
+There are situations when the traversal stream needs a "barrier" to aggregate all the objects and emit a computation
+that is a function of the aggregate. The `fold()`-step (*map*) is one particular instance of this. Please see
+<<unfold-step,`unfold()`>>-step for the inverse functionality.
+
+[gremlin-groovy,modern]
+----
+g.V(1).out('knows').values('name')
+g.V(1).out('knows').values('name').fold() <1>
+g.V(1).out('knows').values('name').fold().next().getClass() <2>
+g.V(1).out('knows').values('name').fold(0) {a,b -> a + b.length()} <3>
+g.V().values('age').fold(0) {a,b -> a + b} <4>
+g.V().values('age').fold(0, sum) <5>
+g.V().values('age').sum() <6>
+----
+
+<1> A parameterless `fold()` will aggregate all the objects into a list and then emit the list.
+<2> A verification of the type of list returned.
+<3> `fold()` can be provided two arguments -- a seed value and a reduce bi-function ("vadas" is 5 characters + "josh" with 4 characters).
+<4> What is the total age of the people in the graph?
+<5> The same as before, but using a built-in bi-function.
+<6> The same as before, but using the <<sum-step,`sum()`-step>>.
+
+[[graph-step]]
+Graph Step
+~~~~~~~~~~
+
+The `V()`-step is usually used to start a `GraphTraversal`, but can also be used mid-traversal.
+
+[gremlin-groovy,modern]
+----
+g.V().has('name', within('marko', 'vadas', 'josh')).as('person').
+ V().has('name', within('lop', 'ripple')).addE('uses').from('person')
+----
+
+NOTE: Whether a mid-traversal `V()` uses an index or not, depends on a) whether suitable index exists and b) if the particular graph system provider implemented this functionality.
+
+[gremlin-groovy,modern]
+----
+g.V().has('name', within('marko', 'vadas', 'josh')).as('person').
+ V().has('name', within('lop', 'ripple')).addE('uses').from('person').toString() <1>
+g.V().has('name', within('marko', 'vadas', 'josh')).as('person').
+ V().has('name', within('lop', 'ripple')).addE('uses').from('person').iterate().toString() <2>
+----
+
+<1> Normally the `V()`-step will iterate over all vertices. However, graph strategies can fold `HasContainer`'s into a `GraphStep` to allow index lookups.
+<2> Whether the graph system provider supports mid-traversal `V()` index lookups or not can easily be determined by inspecting the `toString()` output of the iterated traversal. If `has` conditions were folded into the `V()`-step, an index - if one exists - will be used.
+
+[[group-step]]
+Group Step
+~~~~~~~~~~
+
+As traversers propagate across a graph as defined by a traversal, sideEffect computations are sometimes required.
+That is, the actual path taken or the current location of a traverser is not the ultimate output of the computation,
+but some other representation of the traversal. The `group()`-step (*map*/*sideEffect*) is one such sideEffect that
+organizes the objects according to some function of the object. Then, if required, that organization (a list) is
+reduced. An example is provided below.
+
+[gremlin-groovy,modern]
+----
+g.V().group().by(label) <1>
+g.V().group().by(label).by('name') <2>
+g.V().group().by(label).by(count()) <3>
+----
+
+<1> Group the vertices by their label.
+<2> For each vertex in the group, get their name.
+<3> For each grouping, what is its size?
+
+The three projection parameters available to `group()` via `by()` are:
+
+. Key-projection: What feature of the object to group on (a function that yields the map key)?
+. Value-projection: What feature of the group to store in the key-list?
+. Reduce-projection: What feature of the key-list to ultimately return?
+
+[[groupcount-step]]
+GroupCount Step
+~~~~~~~~~~~~~~~
+
+When it is important to know how many times a particular object has been at a particular part of a traversal,
+`groupCount()`-step (*map*/*sideEffect*) is used.
+
+ "What is the distribution of ages in the graph?"
+
+[gremlin-groovy,modern]
+----
+g.V().hasLabel('person').values('age').groupCount()
+g.V().hasLabel('person').groupCount().by('age') <1>
+----
+
+<1> You can also supply a pre-group projection, where the provided <<by-step,`by()`>>-modulation determines what to
+group the incoming object by.
+
+There is one person that is 32, one person that is 35, one person that is 27, and one person that is 29.
+
+ "Iteratively walk the graph and count the number of times you see the second letter of each name."
+
+image::groupcount-step.png[width=420]
+
+[gremlin-groovy,modern]
+----
+g.V().repeat(both().groupCount('m').by(label)).times(10).cap('m')
+----
+
+The above is interesting in that it demonstrates the use of referencing the internal `Map<Object,Long>` of
+`groupCount()` with a string variable. Given that `groupCount()` is a sideEffect-step, it simply passes the object
+it received to its output. Internal to `groupCount()`, the object's count is incremented.
+
+[[has-step]]
+Has Step
+~~~~~~~~
+
+image::has-step.png[width=670]
+
+It is possible to filter vertices, edges, and vertex properties based on their properties using `has()`-step
+(*filter*). There are numerous variations on `has()` including:
+
+ * `has(key,value)`: Remove the traverser if its element does not have the provided key/value property.
+ * `has(key,predicate)`: Remove the traverser if its element does not have a key value that satisfies the bi-predicate.
+ * `hasLabel(labels...)`: Remove the traverser if its element does not have any of the labels.
+ * `hasId(ids...)`: Remove the traverser if its element does not have any of the ids.
+ * `hasKey(keys...)`: Remove the traverser if its property does not have any of the keys.
+ * `hasValue(values...)`: Remove the traverser if its property does not have any of the values.
+ * `has(key)`: Remove the traverser if its element does not have a value for the key.
+ * `hasNot(key)`: Remove the traverser if its element has a value for the key.
+ * `has(key, traversal)`: Remove the traverser if its object does not yield a result through the traversal off the property value.
+
+[gremlin-groovy,modern]
+----
+g.V().hasLabel('person')
+g.V().hasLabel('person').out().has('name',within('vadas','josh'))
+g.V().hasLabel('person').out().has('name',within('vadas','josh')).
+ outE().hasLabel('created')
+g.V().has('age',inside(20,30)).values('age') <1>
+g.V().has('age',outside(20,30)).values('age') <2>
+g.V().has('name',within('josh','marko')).valueMap() <3>
+g.V().has('name',without('josh','marko')).valueMap() <4>
+g.V().has('name',not(within('josh','marko'))).valueMap() <5>
+----
+
+<1> Find all vertices whose ages are between 20 (inclusive) and 30 (exclusive).
+<2> Find all vertices whose ages are not between 20 (inclusive) and 30 (exclusive).
+<3> Find all vertices whose names are exact matches to any names in the the collection `[josh,marko]`, display all
+the key,value pairs for those verticies.
+<4> Find all vertices whose names are not in the collection `[josh,marko]`, display all the key,value pairs for those vertices.
+<5> Same as the prior example save using `not` on `within` to yield `without`.
+
+TinkerPop does not support a regular expression predicate, although specific graph databases that leverage TinkerPop
+may provide a partial match extension.
+
+[[inject-step]]
+Inject Step
+~~~~~~~~~~~
+
+image::inject-step.png[width=800]
+
+One of the major features of TinkerPop3 is "injectable steps." This makes it possible to insert objects arbitrarily
+into a traversal stream. In general, `inject()`-step (*sideEffect*) exists and a few examples are provided below.
+
+[gremlin-groovy,modern]
+----
+g.V(4).out().values('name').inject('daniel')
+g.V(4).out().values('name').inject('daniel').map {it.get().length()}
+g.V(4).out().values('name').inject('daniel').map {it.get().length()}.path()
+----
+
+In the last example above, note that the path starting with `daniel` is only of length 2. This is because the
+`daniel` string was inserted half-way in the traversal. Finally, a typical use case is provided below -- when the
+start of the traversal is not a graph object.
+
+[gremlin-groovy,modern]
+----
+inject(1,2)
+inject(1,2).map {it.get() + 1}
+inject(1,2).map {it.get() + 1}.map {g.V(it.get()).next()}.values('name')
+----
+
+[[is-step]]
+Is Step
+~~~~~~~
+
+It is possible to filter scalar values using `is()`-step (*filter*).
+
+[gremlin-groovy,modern]
+----
+g.V().values('age').is(32)
+g.V().values('age').is(lte(30))
+g.V().values('age').is(inside(30, 40))
+g.V().where(__.in('created').count().is(1)).values('name') <1>
+g.V().where(__.in('created').count().is(gte(2))).values('name') <2>
+g.V().where(__.in('created').values('age').
+ mean().is(inside(30d, 35d))).values('name') <3>
+----
+
+<1> Find projects having exactly one contributor.
+<2> Find projects having two or more contributors.
+<3> Find projects whose contributors average age is between 30 and 35.
+
+[[limit-step]]
+Limit Step
+~~~~~~~~~~
+
+The `limit()`-step is analogous to <<range-step,`range()`-step>> save that the lower end range is set to 0.
+
+[gremlin-groovy,modern]
+----
+g.V().limit(2)
+g.V().range(0, 2)
+g.V().limit(2).toString()
+----
+
+The `limit()`-step can also be applied with `Scope.local`, in which case it operates on the incoming collection.
+The examples below use the <<the-crew-toy-graph,The Crew>> toy data set.
+
+[gremlin-groovy,theCrew]
+----
+g.V().valueMap().select('location').limit(local,2) <1>
+g.V().valueMap().limit(local, 1) <2>
+----
+
+<1> `List<String>` for each vertex containing the first two locations.
+<2> `Map<String, Object>` for each vertex, but containing only the first property value.
+
+[[local-step]]
+Local Step
+~~~~~~~~~~
+
+image::local-step.png[width=450]
+
+A `GraphTraversal` operates on a continuous stream of objects. In many situations, it is important to operate on a
+single element within that stream. To do such object-local traversal computations, `local()`-step exists (*branch*).
+Note that the examples below use the <<the-crew-toy-graph,The Crew>> toy data set.
+
+[gremlin-groovy,theCrew]
+----
+g.V().as('person').
+ properties('location').order().by('startTime',incr).limit(2).value().as('location').
+ select('person','location').by('name').by() <1>
+g.V().as('person').
+ local(properties('location').order().by('startTime',incr).limit(2)).value().as('location').
+ select('person','location').by('name').by() <2>
+----
+
+<1> Get the first two people and their respective location according to the most historic location start time.
+<2> For every person, get their two most historic locations.
+
+The two traversals above look nearly identical save the inclusion of `local()` which wraps a section of the traversal
+in a object-local traversal. As such, the `order().by()` and the `limit()` refer to a particular object, not to the
+stream as a whole.
+
+WARNING: The anonymous traversal of `local()` processes the current object "locally." In OLAP, where the atomic unit
+of computing is the the vertex and its local "star graph," it is important that the anonymous traversal does not leave
+the confines of the vertex's star graph. In other words, it can not traverse to an adjacent vertex's properties or edges.
+
+[[match-step]]
+Match Step
+~~~~~~~~~~
+
+The `match()`-step (*map*) provides a more link:http://en.wikipedia.org/wiki/Declarative_programming[declarative]
+form of graph querying based on the notion of link:http://en.wikipedia.org/wiki/Pattern_matching[pattern matching].
+With `match()`, the user provides a collection of "traversal fragments," called patterns, that have variables defined
+that must hold true throughout the duration of the `match()`. When a traverser is in `match()`, a registered
+`MatchAlgorithm` analyzes the current state of the traverser (i.e. its history based on its
+<<path-data-structure,path data>>), the runtime statistics of the traversal patterns, and returns a traversal-pattern
+that the traverser should try next. The default `MatchAlgorithm` provided is called `CountMatchAlgorithm` and it
+dynamically revises the pattern execution plan by sorting the patterns according to their filtering capabilities
+(i.e. largest set reduction patterns execute first). For very large graphs, where the developer is uncertain of the
+statistics of the graph (e.g. how many `knows`-edges vs. `worksFor`-edges exist in the graph), it is advantageous to
+use `match()`, as an optimal plan will be determined automatically. Furthermore, some queries are much easier to
+express via `match()` than with single-path traversals.
+
+ "Who created a project named 'lop' that was also created by someone who is 29 years old? Return the two creators."
+
+image::match-step.png[width=500]
+
+[gremlin-groovy,modern]
+----
+g.V().match(
+ __.as('a').out('created').as('b'),
+ __.as('b').has('name', 'lop'),
+ __.as('b').in('created').as('c'),
+ __.as('c').has('age', 29)).
+ select('a','c').by('name')
+----
+
+Note that the above can also be more concisely written as below which demonstrates that standard inner-traversals can
+be arbitrarily defined.
+
+[gremlin-groovy,modern]
+----
+g.V().match(
+ __.as('a').out('created').has('name', 'lop').as('b'),
+ __.as('b').in('created').has('age', 29).as('c')).
+ select('a','c').by('name')
+----
+
+In order to improve readability, `as()`-steps can be given meaningful labels which better reflect your domain. The
+previous query can thus be written in a more expressive way as shown below.
+
+[gremlin-groovy,modern]
+----
+g.V().match(
+ __.as('creators').out('created').has('name', 'lop').as('projects'), <1>
+ __.as('projects').in('created').has('age', 29).as('cocreators')). <2>
+ select('creators','cocreators').by('name') <3>
+----
+
+<1> Find vertices that created something and match them as 'creators', then find out what they created which is
+named 'lop' and match these vertices as 'projects'.
+<2> Using these 'projects' vertices, find out their creators aged 29 and remember these as 'cocreators'.
+<3> Return the name of both 'creators' and 'cocreators'.
+
+[[grateful-dead]]
+.Grateful Dead
+image::grateful-dead-schema.png[width=475]
+
+`MatchStep` brings functionality similar to link:http://en.wikipedia.org/wiki/SPARQL[SPARQL] to Gremlin. Like SPARQL,
+MatchStep conjoins a set of patterns applied to a graph. For example, the following traversal finds exactly those
+songs which Jerry Garcia has both sung and written (using the Grateful Dead graph distributed in the `data/` directory):
+
+[gremlin-groovy]
+----
+graph.io(graphml()).readGraph('data/grateful-dead.xml')
+g = graph.traversal(standard())
+g.V().match(
+ __.as('a').has('name', 'Garcia'),
+ __.as('a').in('writtenBy').as('b'),
+ __.as('a').in('sungBy').as('b')).
+ select('b').values('name')
+----
+
+Among the features which differentiate `match()` from SPARQL are:
+
+[gremlin-groovy,modern]
+----
+g.V().match(
+ __.as('a').out('created').has('name','lop').as('b'), <1>
+ __.as('b').in('created').has('age', 29).as('c'),
+ __.as('c').repeat(out()).times(2)). <2>
+ select('c').out('knows').dedup().values('name') <3>
+----
+
+<1> *Patterns of arbitrary complexity*: `match()` is not restricted to triple patterns or property paths.
+<2> *Recursion support*: `match()` supports the branch-based steps within a pattern, including `repeat()`.
+<3> *Imperative/declarative hybrid*: Before and after a `match()`, it is possible to leverage classic Gremlin traversals.
+
+To extend point #3, it is possible to support going from imperative, to declarative, to imperative, ad infinitum.
+
+[gremlin-groovy,modern]
+----
+g.V().match(
+ __.as('a').out('knows').as('b'),
+ __.as('b').out('created').has('name','lop')).
+ select('b').out('created').
+ match(
+ __.as('x').in('created').as('y'),
+ __.as('y').out('knows').as('z')).
+ select('z').values('name')
+----
+
+IMPORTANT: The `match()`-step is stateless. The variable bindings of the traversal patterns are stored in the path
+history of the traverser. As such, the variables used over all `match()`-steps within a traversal are globally unique.
+A benefit of this is that subsequent `where()`, `select()`, `match()`, etc. steps can leverage the same variables in
+their analysis.
+
+Like all other steps in Gremlin, `match()` is a function and thus, `match()` within `match()` is a natural consequence
+of Gremlin's functional foundation (i.e. recursive matching).
+
+[gremlin-groovy,modern]
+----
+g.V().match(
+ __.as('a').out('knows').as('b'),
+ __.as('b').out('created').has('name','lop'),
+ __.as('b').match(
+ __.as('b').out('created').as('c'),
+ __.as('c').has('name','ripple')).
+ select('c').as('c')).
+ select('a','c').by('name')
+----
+
+If a step-labeled traversal proceeds the `match()`-step and the traverser entering the `match()` is destined to bind
+to a particular variable, then the previous step should be labeled accordingly.
+
+[gremlin-groovy,modern]
+----
+g.V().as('a').out('knows').as('b').
+ match(
+ __.as('b').out('created').as('c'),
+ __.not(__.as('c').in('created').as('a'))).
+ select('a','b','c').by('name')
+----
+
+There are three types of `match()` traversal patterns.
+
+ . `as('a')...as('b')`: both the start and end of the traversal have a declared variable.
+ . `as('a')...`: only the start of the traversal has a declared variable.
+ . `...`: there are no declared variables.
+
+If a variable is at the start of a traversal pattern it *must* exist as a label in the path history of the traverser
+else the traverser can not go down that path. If a variable is at the end of a traversal pattern then if the variable
+exists in the path history of the traverser, the traverser's current location *must* match (i.e. equal) its historic
+location at that same label. However, if the variable does not exist in the path history of the traverser, then the
+current location is labeled as the variable and thus, becomes a bound variable for subsequent traversal patterns. If a
+traversal pattern does not have an end label, then the traverser must simply "survive" the pattern (i.e. not be
+filtered) to continue to the next pattern. If a traversal pattern does not have a start label, then the traverser
+can go down that path at any point, but will only go down that pattern once as a traversal pattern is executed once
+and only once for the history of the traverser. Typically, traversal patterns that do not have a start and end label
+are used in conjunction with `and()`, `or()`, and `where()`. Once the traverser has "survived" all the patterns (or at
+least one for `or()`), `match()`-step analyzes the traverser's path history and emits a `Map<String,Object>` of the
+variable bindings to the next step in the traversal.
+
+[gremlin-groovy,modern]
+----
+g.V().as('a').out().as('b'). <1>
+ match( <2>
+ __.as('a').out().count().as('c'), <3>
+ __.not(__.as('a').in().as('b')), <4>
+ or( <5>
+ __.as('a').out('knows').as('b'),
+ __.as('b').in().count().as('c').and().as('c').is(gt(2)))). <6>
+ dedup('a','c'). <7>
+ select('a','b','c').by('name').by('name').by() <8>
+----
+
+<1> A standard, step-labeled traversal can come prior to `match()`.
+<2> If the traverser's path prior to entering `match()` has requisite label values, then those historic values are bound.
+<3> It is possible to use <<a-note-on-barrier-steps,barrier steps>> though they are computed locally to the pattern (as one would expect).
+<4> It is possible to `not()` a pattern.
+<5> It is possible to nest `and()`- and `or()`-steps for conjunction matching.
+<6> Both infix and prefix conjunction notation is supported.
+<7> It is possible to "distinct" the specified label combination.
+<8> The bound values are of different types -- vertex ("a"), vertex ("b"), long ("c").
+
+[[using-where-with-match]]
+Using Where with Match
+^^^^^^^^^^^^^^^^^^^^^^
+
+Match is typically used in conjunction with both `select()` (demonstrated previously) and `where()` (presented here).
+A `where()`-step allows the user to further constrain the result set provided by `match()`.
+
+[gremlin-groovy,modern]
+----
+g.V().match(
+ __.as('a').out('created').as('b'),
+ __.as('b').in('created').as('c')).
+ where('a', neq('c')).
+ select('a','c').by('name')
+----
+
+The `where()`-step can take either a `P`-predicate (example above) or a `Traversal` (example below). Using
+`MatchPredicateStrategy`, `where()`-clauses are automatically folded into `match()` and thus, subject to the query
+optimizer within `match()`-step.
+
+[gremlin-groovy,modern]
+----
+traversal = g.V().match(
+ __.as('a').has(label,'person'), <1>
+ __.as('a').out('created').as('b'),
+ __.as('b').in('created').as('c')).
+ where(__.as('a').out('knows').as('c')). <2>
+ select('a','c').by('name'); null <3>
+traversal.toString() <4>
+traversal <5> <6>
+traversal.toString() <7>
+----
+
+<1> Any `has()`-step traversal patterns that start with the match-key are pulled out of `match()` to enable the graph
+system to leverage the filter for index lookups.
+<2> A `where()`-step with a traversal containing variable bindings declared in `match()`.
+<3> A useful trick to ensure that the traversal is not iterated by Gremlin Console.
+<4> The string representation of the traversal prior to its strategies being applied.
+<5> The Gremlin Console will automatically iterate anything that is an iterator or is iterable.
+<6> Both marko and josh are co-developers and marko knows josh.
+<7> The string representation of the traversal after the strategies have been applied (and thus, `where()` is folded into `match()`)
+
+IMPORTANT: A `where()`-step is a filter and thus, variables within a `where()` clause are not globally bound to the
+path of the traverser in `match()`. As such, `where()`-steps in `match()` are used for filtering, not binding.
+
+[[max-step]]
+Max Step
+~~~~~~~~
+
+The `max()`-step (*map*) operates on a stream of numbers and determines which is the largest number in the stream.
+
+[gremlin-groovy,modern]
+----
+g.V().values('age').max()
+g.V().repeat(both()).times(3).values('age').max()
+----
+
+IMPORTANT: `max(local)` determines the max of the current, local object (not the objects in the traversal stream).
+This works for `Collection` and `Number`-type objects. For any other object, a max of `Double.NaN` is returned.
+
+[[mean-step]]
+Mean Step
+~~~~~~~~~
+
+The `mean()`-step (*map*) operates on a stream of numbers and determines the average of those numbers.
+
+[gremlin-groovy,modern]
+----
+g.V().values('age').mean()
+g.V().repeat(both()).times(3).values('age').mean() <1>
+g.V().repeat(both()).times(3).values('age').dedup().mean()
+----
+
+<1> Realize that traversers are being bulked by `repeat()`. There may be more of a particular number than another,
+thus altering the average.
+
+IMPORTANT: `mean(local)` determines the mean of the current, local object (not the objects in the traversal stream).
+This works for `Collection` and `Number`-type objects. For any other object, a mean of `Double.NaN` is returned.
+
+[[min-step]]
+Min Step
+~~~~~~~~
+
+The `min()`-step (*map*) operates on a stream of numbers and determines which is the smallest number in the stream.
+
+[gremlin-groovy,modern]
+----
+g.V().values('age').min()
+g.V().repeat(both()).times(3).values('age').min()
+----
+
+IMPORTANT: `min(local)` determines the min of the current, local object (not the objects in the traversal stream).
+This works for `Collection` and `Number`-type objects. For any other object, a min of `Double.NaN` is returned.
+
+[[or-step]]
+Or Step
+~~~~~~~
+
+The `or()`-step ensures that at least one of the provided traversals yield a result (*filter*). Please see
+<<and-step,`and()`>> for and-semantics.
+
+[gremlin-groovy,modern]
+----
+g.V().or(
+ __.outE('created'),
+ __.inE('created').count().is(gt(1))).
+ values('name')
+----
+
+The `or()`-step can take an arbitrary number of traversals. At least one of the traversals must produce at least one
+output for the original traverser to pass to the next step.
+
+An link:http://en.wikipedia.org/wiki/Infix_notation[infix notation] can be used as well. Though, with infix notation,
+only two traversals can be or'd together.
+
+[gremlin-groovy,modern]
+----
+g.V().where(outE('created').or().outE('knows')).values('name')
+----
+
+[[order-step]]
+Order Step
+~~~~~~~~~~
+
+When the objects of the traversal stream need to be sorted, `order()`-step (*map*) can be leveraged.
+
+[gremlin-groovy,modern]
+----
+g.V().values('name').order()
+g.V().values('name').order().by(decr)
+g.V().hasLabel('person').order().by('age', incr).values('name')
+----
+
+One of the most traversed objects in a traversal is an `Element`. An element can have properties associated with it
+(i.e. key/value pairs). In many situations, it is desirable to sort an element traversal stream according to a
+comparison of their properties.
+
+[gremlin-groovy,modern]
+----
+g.V().values('name')
+g.V().order().by('name',incr).values('name')
+g.V().order().by('name',decr).values('name')
+----
+
+The `order()`-step allows the user to provide an arbitrary number of comparators for primary, secondary, etc. sorting.
+In the example below, the primary ordering is based on the outgoing created-edge count. The secondary ordering is
+based on the age of the person.
+
+[gremlin-groovy,modern]
+----
+g.V().hasLabel('person').order().by(outE('created').count(), incr).
+ by('age', incr).values('name')
+g.V().hasLabel('person').order().by(outE('created').count(), incr).
+ by('age', decr).values('name')
+----
+
+Randomizing the order of the traversers at a particular point in the traversal is possible with `Order.shuffle`.
+
+[gremlin-groovy,modern]
+----
+g.V().hasLabel('person').order().by(shuffle)
+g.V().hasLabel('person').order().by(shuffle)
+----
+
+IMPORTANT: `order(local)` orders the current, local object (not the objects in the traversal stream). This works for
+`Collection`- and `Map`-type objects. For any other object, the object is returned unchanged.
+
+[[path-step]]
+Path Step
+~~~~~~~~~
+
+A traverser is transformed as it moves through a series of steps within a traversal. The history of the traverser is
+realized by examining its path with `path()`-step (*map*).
+
+image::path-step.png[width=650]
+
+[gremlin-groovy,modern]
+----
+g.V().out().out().values('name')
+g.V().out().out().values('name').path()
+----
+
+If edges are required in the path, then be sure to traverser those edges explicitly.
+
+[gremlin-groovy,modern]
+----
+g.V().outE().inV().outE().inV().path()
+----
+
+It is possible to post-process the elements of the path in a round-robin fashion via `by()`.
+
+[gremlin-groovy,modern]
+----
+g.V().out().out().path().by('name').by('age')
+----
+
+Finally, because `by()`-based post-processing, nothing prevents triggering yet another traversal. In the traversal
+below, for each element of the path traversed thus far, if its a person (as determined by having an `age`-property),
+then get all of their creations, else if its a creation, get all the people that created it.
+
+[gremlin-groovy,modern]
+----
+g.V().out().out().path().by(
+ choose(hasLabel('person'),
+ out('created').values('name'),
+ __.in('created').values('name')).fold())
+----
+
+WARNING: Generating path information is expensive as the history of the traverser is stored into a Java list. With
+numerous traversers, there are numerous lists. Moreover, in an OLAP <<graphcomputer,`GraphComputer`>> environment
+this becomes exceedingly prohibitive as there are traversers emanating from all vertices in the graph in parallel.
+In OLAP there are optimizations provided for traverser populations, but when paths are calculated (and each traverser
+is unique due to its history), then these optimizations are no longer possible.
+
+[[path-data-structure]]
+Path Data Structure
+^^^^^^^^^^^^^^^^^^^
+
+The `Path` data structure is an ordered list of objects, where each object is associated to a `Set<String>` of
+labels. An example is presented below to demonstrate both the `Path` API as well as how a traversal yields labeled paths.
+
+image::path-data-structure.png[width=350]
+
+[gremlin-groovy,modern]
+----
+path = g.V(1).as('a').has('name').as('b').
+ out('knows').out('created').as('c').
+ has('name','ripple').values('name').as('d').
+ identity().as('e').path().next()
+path.size()
+path.objects()
+path.labels()
+path.a
+path.b
+path.c
+path.d == path.e
+----
+
+[[profile-step]]
+Profile Step
+~~~~~~~~~~~~
+
+The `profile()`-step (*sideEffect*) exists to allow developers to profile their traversals to determine statistical
+information like step runtime, counts, etc.
+
+WARNING: Profiling a Traversal will impede the Traversal's performance. This overhead is mostly excluded from the
+profile results, but durations are not exact. Thus, durations are best considered in relation to each other.
+
+[gremlin-groovy,modern]
+----
+g.V().out('created').repeat(both()).times(3).hasLabel('person').values('age').sum().profile().cap(TraversalMetrics.METRICS_KEY)
+----
+
+The `profile()`-step generates a `TraversalMetrics` sideEffect object that contains the following information:
+
+* `Step`: A step within the traversal being profiled.
+* `Count`: The number of _represented_ traversers that passed through the step.
+* `Traversers`: The number of traversers that passed through the step.
+* `Time (ms)`: The total time the step was actively executing its behavior.
+* `% Dur`: The percentage of total time spent in the step.
+
+image:gremlin-exercise.png[width=120,float=left] It is important to understand the difference between `Count`
+and `Traversers`. Traversers can be merged and as such, when two traversers are "the same" they may be aggregated
+into a single traverser. That new traverser has a `Traverser.bulk()` that is the sum of the two merged traverser
+bulks. On the other hand, the `Count` represents the sum of all `Traverser.bulk()` results and thus, expresses the
+number of "represented" (not enumerated) traversers. `Traversers` will always be less than or equal to `Count`.
+
+For traversal compilation information, please see <<explain-step,`explain()`>>-step.
+
+[[range-step]]
+Range Step
+~~~~~~~~~~
+
+As traversers propagate through the traversal, it is possible to only allow a certain number of them to pass through
+with `range()`-step (*filter*). When the low-end of the range is not met, objects are continued to be iterated. When
+within the low and high range (both inclusive), traversers are emitted. Finally, when above the high range, the
+traversal breaks out of iteration.
+
+[gremlin-groovy,modern]
+----
+g.V().range(0,3)
+g.V().range(1,3)
+g.V().repeat(both()).times(1000000).emit().range(6,10)
+----
+
+The `range()`-step can also be applied with `Scope.local`, in which case it operates on the incoming collection.
+For example, it is possible to produce a `Map<String, String>` for each traversed path, but containing only the second
+property value (the "b" step).
+
+[gremlin-groovy,modern]
+----
+g.V().as('a').out().as('b').in().as('c').select('a','b','c').by('name').range(local,1,2)
+----
+
+The next example uses the <<the-crew-toy-graph,The Crew>> toy data set. It produces a `List<String>` containing the
+second and third location for each vertex.
+
+[gremlin-groovy,theCrew]
+----
+g.V().valueMap().select('location').range(local, 1, 3)
+----
+
+[[repeat-step]]
+Repeat Step
+~~~~~~~~~~~
+
+image::gremlin-fade.png[width=350]
+
+The `repeat()`-step (*branch*) is used for looping over a traversal given some break predicate. Below are some
+examples of `repeat()`-step in action.
+
+[gremlin-groovy,modern]
+----
+g.V(1).repeat(out()).times(2).path().by('name') <1>
+g.V().until(has('name','ripple')).
+ repeat(out()).path().by('name') <2>
+----
+
+<1> do-while semantics stating to do `out()` 2 times.
+<2> while-do semantics stating to break if the traverser is at a vertex named "ripple".
+
+IMPORTANT: There are two modulators for `repeat()`: `until()` and `emit()`. If `until()` comes after `repeat()` it is
+do/while looping. If `until()` comes before `repeat()` it is while/do looping. If `emit()` is placed after `repeat()`,
+it is evaluated on the traversers leaving the repeat-traversal. If `emit()` is placed before `repeat()`, it is
+evaluated on the traversers prior to entering the repeat-traversal.
+
+The `repeat()`-step also supports an "emit predicate", where the predicate for an empty argument `emit()` is
+`true` (i.e. `emit() == emit{true}`). With `emit()`, the traverser is split in two -- the traverser exits the code
+block as well as continues back within the code block (assuming `until()` holds true).
+
+[gremlin-groovy,modern]
+----
+g.V(1).repeat(out()).times(2).emit().path().by('name') <1>
+g.V(1).emit().repeat(out()).times(2).path().by('name') <2>
+----
+
+<1> The `emit()` comes after `repeat()` and thus, emission happens after the `repeat()` traversal is executed. Thus,
+no one vertex paths exist.
+<2> The `emit()` comes before `repeat()` and thus, emission happens prior to the `repeat()` traversal being executed.
+Thus, one vertex paths exist.
+
+The `emit()`-modulator can take an arbitrary predicate.
+
+[gremlin-groovy,modern]
+----
+g.V(1).repeat(out()).times(2).emit(has('lang')).path().by('name')
+----
+
+image::repeat-step.png[width=500]
+
+[gremlin-groovy,modern]
+----
+g.V(1).repeat(out()).times(2).emit().path().by('name')
+----
+
+The first time through the `repeat()`, the vertices lop, vadas, and josh are seen. Given that `loops==1`, the
+traverser repeats. However, because the emit-predicate is declared true, those vertices are emitted. The next time through
+ `repeat()`, the vertices traversed are ripple and lop (Josh's created projects, as lop and vadas have no out edges).
+ Given that `loops==2`, the until-predicate fails and ripple and lop are emitted.
+Therefore, the traverser has seen the vertices: lop, vadas, josh, ripple, and lop.
+
+Finally, note that both `emit()` and `until()` can take a traversal and in such, situations, the predicate is
+determined by `traversal.hasNext()`. A few examples are provided below.
+
+[gremlin-groovy,modern]
+----
+g.V(1).repeat(out()).until(hasLabel('software')).path().by('name') <1>
+g.V(1).emit(hasLabel('person')).repeat(out()).path().by('name') <2>
+g.V(1).repeat(out()).until(outE().count().is(0)).path().by('name') <3>
+----
+
+<1> Starting from vertex 1, keep taking outgoing edges until a software vertex is reached.
+<2> Starting from vertex 1, and in an infinite loop, emit the vertex if it is a person and then traverser the outgoing edges.
+<3> Starting from vertex 1, keep taking outgoing edges until a vertex is reached that has no more outgoing edges.
+
+WARNING: The anonymous traversal of `emit()` and `until()` (not `repeat()`) process their current objects "locally."
+In OLAP, where the atomic unit of computing is the the vertex and its local "star graph," it is important that the
+anonymous traversals do not leave the confines of the vertex's star graph. In other words, they can not traverse to
+an adjacent vertex's properties or edges.
+
+[[sack-step]]
+Sack Step
+~~~~~~~~~
+
+image:gremlin-sacks-running.png[width=175,float=right] A traverser can contain a local data structure called a "sack".
+The `sack()`-step is used to read and write sacks (*sideEffect* or *map*). Each sack of each traverser is created
+when using `GraphTraversal.withSack(initialValueSupplier,splitOperator?,mergeOperator?)`.
+
+* *Initial value supplier*: A `Supplier` providing the initial value of each traverser's sack.
+* *Split operator*: a `UnaryOperator` that clones the traverser's sack when the traverser splits. If no split operator
+is provided, then `UnaryOperator.identity()` is assumed.
+* *Merge operator*: A `BinaryOperator` that unites two traverser's sack when they are merged. If no merge operator is
+provided, then traversers with sacks can not be merged.
+
+Two trivial examples are presented below to demonstrate the *initial value supplier*. In the first example below, a
+traverser is created at each vertex in the graph (`g.V()`), with a 1.0 sack (`withSack(1.0f)`), and then the sack
+value is accessed (`sack()`). In the second example, a random float supplier is used to generate sack values.
+
+[gremlin-groovy,modern]
+----
+g.withSack(1.0f).V().sack()
+rand = new Random()
+g.withSack {rand.nextFloat()}.V().sack()
+----
+
+A more complicated initial value supplier example is presented below where the sack values are used in a running
+computation and then emitted at the end of the traversal. When an edge is traversed, the edge weight is multiplied
+by the sack value (`sack(mult).by('weight')`). Note that the <<by-step,`by()`>>-modulator can be any arbitrary traversal.
+
+[gremlin-groovy,modern]
+----
+g.withSack(1.0f).V().repeat(outE().sack(mult).by('weight').inV()).times(2)
+g.withSack(1.0f).V().repeat(outE().sack(mult).by('weight').inV()).times(2).sack()
+g.withSack(1.0f).V().repeat(outE().sack(mult).by('weight').inV()).times(2).path().
+ by().by('weight')
+----
+
+image:gremlin-sacks-standing.png[width=100,float=left] When complex objects are used (i.e. non-primitives), then a
+*split operator* should be defined to ensure that each traverser gets a clone of its parent's sack. The first example
+does not use a split operator and as such, the same map is propagated to all traversers (a global data structure). The
+second example, demonstrates how `Map.clone()` ensures that each traverser's sack contains a unique, local sack.
+
+[gremlin-groovy,modern]
+----
+g.withSack {[:]}.V().out().out().
+ sack {m,v -> m[v.value('name')] = v.value('lang'); m}.sack() // BAD: single map
+g.withSack {[:]}{it.clone()}.V().out().out().
+ sack {m,v -> m[v.value('name')] = v.value('lang'); m}.sack() // GOOD: cloned map
+----
+
+NOTE: For primitives (i.e. integers, longs, floats, etc.), a split operator is not required as a primitives are
+encoded in the memory address of the sack, not as a reference to an object.
+
+If a *merge operator* is not provided, then traversers with sacks can not be bulked. However, in many situations,
+merging the sacks of two traversers at the same location is algorithmically sound and good to provide so as to gain
+the bulking optimization. In the examples below, the binary merge operator is `Operator.sum`. Thus, when two traverser
+merge, their respective sacks are added together.
+
+[gremlin-groovy,modern]
+----
+g.withSack(1.0f,sum).V(1).local(outE('knows').barrier(normSack).inV()) <1>
+g.withSack(1.0f,sum).V(1).local(outE('knows').barrier(normSack).inV()).sack() <2>
+g.withSack(1.0f,sum).V(1).local(outE('knows').barrier(normSack).inV()).in('knows') <3>
+g.withSack(1.0f,sum).V(1).local(outE('knows').barrier(normSack).inV()).in('knows').sack() <4>
+g.withSack(1.0f,sum).V(1).local(outE('knows').barrier(normSack).inV()).in('knows').barrier().sack() <5>
+g.withBulk(false).withSack(1.0f,sum).V(1).local(outE('knows').barrier(normSack).inV()).in('knows').barrier().sack() <6>
+----
+
+<1> The knows-adjacent vertices of vertex 1 are vertices 2 and 4.
+<2> The `local(...barrier(normSack)...)` ensures that all traversers leaving vertex 1 have an evenly distributed amount of the initial 1.0 "energy" (50-50).
+<3> Going from vertices 2 and 4 yield two traversers at vertex 1.
+<4> Those two traversers each have a sack of 0.5.
+<5> The `barrier()` merges the two traversers at vertex 1 into a single traverser whose sack is 1.0.
+<6> There is now a single traverser with bulk of 2 and sack of 1.0 and thus, setting `withBulk(false)` yields the expected 1.0.
+
+
+[[sample-step]]
+Sample Step
+~~~~~~~~~~~
+
+The `sample()`-step is useful for sampling some number of traversers previous in the traversal.
+
+[gremlin-groovy,modern]
+----
+g.V().outE().sample(1).values('weight')
+g.V().outE().sample(1).by('weight').values('weight')
+g.V().outE().sample(2).by('weight').values('weight')
+----
+
+One of the more interesting use cases for `sample()` is when it is used in conjunction with <<local-step,`local()`>>.
+The combination of the two steps supports the execution of link:http://en.wikipedia.org/wiki/Random_walk[random walks].
+In the example below, the traversal starts are vertex 1 and selects one edge to traverse based on a probability
+distribution generated by the weights of the edges. The output is always a single path as by selecting a single edge,
+the traverser never splits and continues down a single path in the graph.
+
+[gremlin-groovy,modern]
+----
+g.V(1).repeat(local(
+ bothE().sample(1).by('weight').otherV()
+ )).times(5)
+g.V(1).repeat(local(
+ bothE().sample(1).by('weight').otherV()
+ )).times(5).path()
+g.V(1).repeat(local(
+ bothE().sample(1).by('weight').otherV()
+ )).times(10).path()
+----
+
+[[select-step]]
+Select Step
+~~~~~~~~~~~
+
+link:http://en.wikipedia.org/wiki/Functional_programming[Functional languages] make use of function composition and
+lazy evaluation to create complex computations from primitive operations. This is exactly what `Traversal` does. One
+of the differentiating aspects of Gremlin's data flow approach to graph processing is that the flow need not always go
+"forward," but in fact, can go back to a previously seen area of computation. Examples include <<path-step,`path()`>>
+as well as the `select()`-step (*map*). There are two general ways to use `select()`-step.
+
+. Select labeled steps within a path (as defined by `as()` in a traversal).
+. Select objects out of a `Map<String,Object>` flow (i.e. a sub-map).
+
+The first use case is demonstrated via example below.
+
+[gremlin-groovy,modern]
+----
+g.V().as('a').out().as('b').out().as('c') // no select
+g.V().as('a').out().as('b').out().as('c').select('a','b','c')
+g.V().as('a').out().as('b').out().as('c').select('a','b')
+g.V().as('a').out().as('b').out().as('c').select('a','b').by('name')
+g.V().as('a').out().as('b').out().as('c').select('a') <1>
+----
+
+<1> If the selection is one step, no map is returned.
+
+When there is only one label selected, then a single object is returned. This is useful for stepping back in a
+computation and easily moving forward again on the object reverted to.
+
+[gremlin-groovy,modern]
+----
+g.V().out().out()
+g.V().out().out().path()
+g.V().as('x').out().out().select('x')
+g.V().out().as('x').out().select('x')
+g.V().out().out().as('x').select('x') // pointless
+----
+
+NOTE: When executing a traversal with `select()` on a standard traversal engine (i.e. OLTP), `select()` will do its
+best to avoid calculating the path history and instead, will rely on a global data structure for storing the currently
+selected object. As such, if only a subset of the path walked is required, `select()` should be used over the more
+resource intensive <<path-step,`path()`>>-step.
+
+When the set of keys or values (i.e. columns) of a path or map are needed, use `select(keys)` and `select(values)`,
+respectively. This is especially useful when one is only interested in the top N elements in a `groupCount()`
+ranking.
+
+[gremlin-groovy]
+----
+graph.io(graphml()).readGraph('data/grateful-dead.xml')
+g = graph.traversal()
+g.V().hasLabel('song').out('followedBy').groupCount().by('name').
+ order(local).by(valueDecr).limit(local, 5)
+g.V().hasLabel('song').out('followedBy').groupCount().by('name').
+ order(local).by(valueDecr).limit(local, 5).select(keys)
+g.V().hasLabel('song').out('followedBy').groupCount().by('name').
+ order(local).by(valueDecr).limit(local, 5).select(keys).unfold()
+----
+
+Similarly, for extracting the values from a path or map.
+
+[gremlin-groovy]
+----
+graph.io(graphml()).readGraph('data/grateful-dead.xml')
+g = graph.traversal()
+g.V().hasLabel('song').out('sungBy').groupCount().by('name') <1>
+g.V().hasLabel('song').out('sungBy').groupCount().by('name').select(values) <2>
+g.V().hasLabel('song').out('sungBy').groupCount().by('name').select(values).unfold().
+ groupCount().order(local).by(valueDecr).limit(local, 5) <3>
+----
+
+<1> Which artist sung how many songs?
+<2> Get an anonymized set of song repertoire sizes.
+<3> What are the 5 most common song repertoire sizes?
+
+CAUTION: Note that `by()`-modulation is not supported with `select(keys)` and `select(values)`.
+
+[[using-where-with-select]]
+Using Where with Select
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Like <<match-step,`match()`>>-step, it is possible to use `where()`, as where is a filter that processes
+`Map<String,Object>` streams.
+
+[gremlin-groovy,modern]
+----
+g.V().as('a').out('created').in('created').as('b').select('a','b').by('name') <1>
+g.V().as('a').out('created').in('created').as('b').
+ select('a','b').by('name').where('a',neq('b')) <2>
+g.V().as('a').out('created').in('created').as('b').
+ select('a','b'). <3>
+ where('a',neq('b')).
+ where(__.as('a').out('knows').as('b')).
+ select('a','b').by('name')
+----
+
+<1> A standard `select()` that generates a `Map<String,Object>` of variables bindings in the path (i.e. `a` and `b`)
+for the sake of a running example.
+<2> The `select().by('name')` projects each binding vertex to their name property value and `where()` operates to
+ensure respective `a` and `b` strings are not the same.
+<3> The first `select()` projects a vertex binding set. A binding is filtered if `a` vertex equals `b` vertex. A
+binding is filtered if `a` doesn't know `b`. The second and final `select()` projects the name of the vertices.
+
+[[simplepath-step]]
+SimplePath Step
+~~~~~~~~~~~~~~~
+
+image::simplepath-step.png[width=400]
+
+When it is important that a traverser not repeat its path through the graph, `simplePath()`-step should be used
+(*filter*). The <<path-data-structure,path>> information of the traverser is analyzed and if the path has repeated
+objects in it, the traverser is filtered. If cyclic behavior is desired, see <<cyclicpath-step,`cyclicPath()`>>.
+
+[gremlin-groovy,modern]
+----
+g.V(1).both().both()
+g.V(1).both().both().simplePath()
+g.V(1).both().both().simplePath().path()
+----
+
+[[store-step]]
+Store Step
+~~~~~~~~~~
+
+When link:http://en.wikipedia.org/wiki/Lazy_evaluation[lazy] aggregation is needed, `store()`-step (*sideEffect*)
+should be used over <<aggregate-step,`aggregate()`>>. The two steps differ in that `store()` does not block and only
+stores objects in its side-effect collection as they pass through.
+
+[gremlin-groovy,modern]
+----
+g.V().aggregate('x').limit(1).cap('x')
+g.V().store('x').limit(1).cap('x')
+----
+
+It is interesting to note that there are three results in the `store()` side-effect even though the interval
+selection is for 2 objects. Realize that when the third object is on its way to the `range()` filter (i.e. `[0..1]`),
+it passes through `store()` and thus, stored before filtered.
+
+[gremlin-groovy,modern]
+----
+g.E().store('x').by('weight').cap('x')
+----
+
+[[subgraph-step]]
+Subgraph Step
+~~~~~~~~~~~~~
+
+image::subgraph-logo.png[width=380]
+
+Extracting a portion of a graph from a larger one for analysis, visualization or other purposes is a fairly common
+use case for graph analysts and developers. The `subgraph()`-step (*sideEffect*) provides a way to produce an
+link:http://mathworld.wolfram.com/Edge-InducedSubgraph.html[edge-induced subgraph] from virtually any traversal.
+The following example demonstrates how to produce the "knows" subgraph:
+
+[gremlin-groovy,modern]
+----
+subGraph = g.E().hasLabel('knows').subgraph('subGraph').cap('subGraph').next() <1>
+sg = subGraph.traversal(standard())
+sg.E() <2>
+----
+
+<1> As this function produces "edge-induced" subgraphs, `subgraph()` must be called at edge steps.
+<2> The subgraph contains only "knows" edges.
+
+A more common subgraphing use case is to get all of the graph structure surrounding a single vertex:
+
+[gremlin-groovy,modern]
+----
+subGraph = g.V(3).repeat(__.inE().subgraph('subGraph').outV()).times(3).cap('subGraph').next() <1>
+sg = subGraph.traversal(standard())
+sg.E()
+----
+
+<1> Starting at vertex `3`, traverse 3 steps away on in-edges, outputting all of that into the subgraph.
+
+There can be multiple `subgraph()` calls within the same traversal. Each operating against either the same graph
+(i.e. same side-effect key) or different graphs (i.e. different side-effect keys).
+
+[gremlin-groovy,modern]
+----
+t = g.V().outE('knows').subgraph('knowsG').inV().outE('created').subgraph('createdG').
+ inV().inE('created').subgraph('createdG').iterate()
+t.sideEffects.get('knowsG').get().traversal(standard()).E()
+t.sideEffects.get('createdG').get().traversal(standard()).E()
+----
+
+IMPORTANT: The `subgraph()`-step only writes to graphs that support user supplied ids for its elements. Moreover,
+if no graph is specified via `withSideEffect()`, then <<tinkergraph-gremlin,TinkerGraph>> is assumed.
+
+[[sum-step]]
+Sum Step
+~~~~~~~~
+
+The `sum()`-step (*map*) operates on a stream of numbers and sums the numbers together to yield a double. Note that
+the current traverser number is multiplied by the traverser bulk to determine how many such numbers are being
+represented.
+
+[gremlin-groovy,modern]
+----
+g.V().values('age').sum()
+g.V().repeat(both()).times(3).values('age').sum()
+----
+
+IMPORTANT: `sum(local)` determines the sum of the current, local object (not the objects in the traversal stream).
+This works for `Collection`-type objects. For any other object, a sum of `Double.NaN` is returned.
+
+[[tail-step]]
+Tail Step
+~~~~~~~~~
+
+image::tail-step.png[width=530]
+
+The `tail()`-step is analogous to <<limit-step,`limit()`>>-step, except that it emits the last `n`-objects instead of
+the first `n`-objects.
+
+[gremlin-groovy,modern]
+----
+g.V().values('name').order()
+g.V().values('name').order().tail() <1>
+g.V().values('name').order().tail(1) <2>
+g.V().values('name').order().tail(3) <3>
+----
+
+<1> Last name (alphabetically).
+<2> Same as statement 1.
+<3> Last three names.
+
+The `tail()`-step can also be applied with `Scope.local`, in which case it operates on the incoming collection.
+
+[gremlin-groovy,modern]
+----
+g.V().as('a').out().as('a').out().as('a').select('a').by(tail(local)).values('name') <1>
+g.V().as('a').out().as('a').out().as('a').select('a').by(unfold().values('name').fold()).tail(local) <2>
+g.V().as('a').out().as('a').out().as('a').select('a').by(unfold().values('name').fold()).tail(local, 2) <3>
+g.V().valueMap().tail(local) <4>
+----
+
+<1> Only the most recent name from the "a" step (`List<Vertex>` becomes `Vertex`).
+<2> Same result as statement 1 (`List<String>` becomes `String`).
+<3> `List<String>` for each path containing the last two names from the 'a' step.
+<4> `Map<String, Object>` for each vertex, but containing only the last property value.
+
+[[timelimit-step]]
+TimeLimit Step
+~~~~~~~~~~~~~~
+
+In many situations, a graph traversal is not about getting an exact answer as its about getting a relative ranking.
+A classic example is link:http://en.wikipedia.org/wiki/Recommender_system[recommendation]. What is desired is a
+relative ranking of vertices, not their absolute rank. Next, it may be desirable to have the traversal execute for
+no more than 2 milliseconds. In such situations, `timeLimit()`-step (*filter*) can be used.
+
+image::timelimit-step.png[width=400]
+
+NOTE: The method `clock(int runs, Closure code)` is a utility preloaded in the <<gremlin-console,Gremlin Console>>
+that can be used to time execution of a body of code.
+
+[gremlin-groovy,modern]
+----
+g.V().repeat(both().groupCount('m')).times(16).cap('m').order(local).by(valueDecr).next()
+clock(1) {g.V().repeat(both().groupCount('m')).times(16).cap('m').order(local).by(valueDecr).next()}
+g.V().repeat(timeLimit(2).both().groupCount('m')).times(16).cap('m').order(local).by(valueDecr).next()
+clock(1) {g.V().repeat(timeLimit(2).both().groupCount('m')).times(16).cap('m').order(local).by(valueDecr).next()}
+----
+
+In essence, the relative order is respected, even through the number of traversers at each vertex is not. The primary
+benefit being that the calculation is guaranteed to complete at the specified time limit (in milliseconds). Finally,
+note that the internal clock of `timeLimit()`-step starts when the first traverser enters it. When the time limit is
+reached, any `next()` evaluation of the step will yield a `NoSuchElementException` and any `hasNext()` evaluation will
+yield `false`.
+
+[[tree-step]]
+Tree Step
+~~~~~~~~~
+
+From any one element (i.e. vertex or edge), the emanating paths from that element can be aggregated to form a
+link:http://en.wikipedia.org/wiki/Tree_(data_structure)[tree]. Gremlin provides `tree()`-step (*sideEffect*) for such
+this situation.
+
+image::tree-step.png[width=450]
+
+[gremlin-groovy,modern]
+----
+tree = g.V().out().out().tree().next()
+----
+
+It is important to see how the paths of all the emanating traversers are united to form the tree.
+
+image::tree-step2.png[width=500]
+
+The resultant tree data structure can then be manipulated (see `Tree` JavaDoc).
+
+[gremlin-groovy,modern]
+----
+tree = g.V().out().out().tree().by('name').next()
+tree['marko']
+tree['marko']['josh']
+tree.getObjectsAtDepth(3)
+----
+
+[[unfold-step]]
+Unfold Step
+~~~~~~~~~~~
+
+If the object reaching `unfold()` (*flatMap*) is an iterator, iterable, or map, then it is unrolled into a linear
+form. If not, then the object is simply emitted. Please see <<fold-step,`fold()`>> step for the inverse behavior.
+
+[gremlin-groovy,modern]
+----
+g.V(1).out().fold().inject('gremlin',[1.23,2.34])
+g.V(1).out().fold().inject('gremlin',[1.23,2.34]).unfold()
+----
+
+Note that `unfold()` does not recursively unroll iterators. Instead, `repeat()` can be used to for recursive unrolling.
+
+[gremlin-groovy,modern]
+----
+inject(1,[2,3,[4,5,[6]]])
+inject(1,[2,3,[4,5,[6]]]).unfold()
+inject(1,[2,3,[4,5,[6]]]).repeat(unfold()).until(count(local).is(1)).unfold()
+----
+
+[[union-step]]
+Union Step
+~~~~~~~~~~
+
+image::union-step.png[width=650]
+
+The `union()`-step (*branch*) supports the merging of the results of an arbitrary number of traversals. When a
+traverser reaches a `union()`-step, it is copied to each of its internal steps. The traversers emitted from `union()`
+are the outputs of the respective internal traversals.
+
+[gremlin-groovy,modern]
+----
+g.V(4).union(
+ __.in().values('age'),
+ out().values('lang'))
+g.V(4).union(
+ __.in().values('age'),
+ out().values('lang')).path()
+----
+
+[[valuemap-step]]
+ValueMap Step
+~~~~~~~~~~~~~
+
+The `valueMap()`-step yields a Map representation of the properties of an element.
+
+[gremlin-groovy,modern]
+----
+g.V().valueMap()
+g.V().valueMap('age')
+g.V().valueMap('age','blah')
+g.E().valueMap()
+----
+
+It is important to note that the map of a vertex maintains a list of values for each key. The map of an edge or
+vertex-property represents a single property (not a list). The reason is that vertices in TinkerPop3 leverage
+<<vertex-properties,vertex properties>> which are support multiple values per key. Using the <<the-crew-toy-graph,
+"The Crew">> toy graph, the point is made explicit.
+
+[gremlin-groovy,theCrew]
+----
+g.V().valueMap()
+g.V().has('name','marko').properties('location')
+g.V().has('name','marko').properties('location').valueMap()
+----
+
+If the `id`, `label`, `key`, and `value` of the `Element` is desired, then a boolean triggers its insertion into the
+returned map.
+
+[gremlin-groovy,theCrew]
+----
+g.V().hasLabel('person').valueMap(true)
+g.V().hasLabel('person').valueMap(true,'name')
+g.V().hasLabel('person').properties('location').valueMap(true)
+----
+
+[[vertex-steps]]
+Vertex Steps
+~~~~~~~~~~~~
+
+image::vertex-steps.png[width=350]
+
+The vertex steps (*flatMap*) are fundamental to the Gremlin language. Via these steps, its possible to "move" on the
+graph -- i.e. traverse.
+
+* `out(string...)`: Move to the outgoing adjacent vertices given the edge labels.
+* `in(string...)`: Move to the incoming adjacent vertices given the edge labels.
+* `both(string...)`: Move to both the incoming and outgoing adjacent vertices given the edge labels.
+* `outE(string...)`: Move to the outgoing incident edges given the edge labels.
+* `inE(string...)`: Move to the incoming incident edges given the edge labels.
+* `bothE(string...)`: Move to both the incoming and outgoing incident edges given the edge labels.
+* `outV()`: Move to the outgoing vertex.
+* `inV()`: Move to the incoming vertex.
+* `bothV()`: Move to both vertices.
+* `otherV()` : Move to the vertex that was not the vertex that was moved from.
+
+[gremlin-groovy,modern]
+----
+g.V(4)
+g.V(4).outE() <1>
+g.V(4).inE('knows') <2>
+g.V(4).inE('created') <3>
+g.V(4).bothE('knows','created','blah')
+g.V(4).bothE('knows','created','blah').otherV()
+g.V(4).both('knows','created','blah')
+g.V(4).outE().inV() <4>
+g.V(4).out() <5>
+g.V(4).inE().outV()
+g.V(4).inE().bothV()
+----
+
+<1> All outgoing edges.
+<2> All incoming knows-edges.
+<3> All incoming created-edges.
+<4> Moving forward touching edges and vertices.
+<5> Moving forward only touching vertices.
+
+[[where-step]]
+Where Step
+~~~~~~~~~~
+
+The `where()`-step filters the current object based on either the object itself (`Scope.local`) or the path history
+of the object (`Scope.global`) (*filter*). This step is typically used in conjuction with either
+<<match-step,`match()`>>-step or <<select-step,`select()`>>-step, but can be used in isolation.
+
+[gremlin-groovy,modern]
+----
+g.V(1).as('a').out('created').in('created').where(neq('a')) <1>
+g.withSideEffect('a',['josh','peter']).V(1).out('created').in('created').values('name').where(within('a')) <2>
+g.V(1).out('created').in('created').where(out('created').count().is(gt(1))).values('name') <3>
+----
+
+<1> Who are marko's collaborators, where marko can not be his own collaborator? (predicate)
+<2> Of the co-creators of marko, only keep those whose name is josh or peter. (using a sideEffect)
+<3> Which of marko's collaborators have worked on more than 1 project? (using a traversal)
+
+IMPORTANT: Please see <<using-where-with-match,`match().where()`>> and <<using-where-with-select,`select().where()`>>
+for how `where()` can be used in conjunction with `Map<String,Object>` projecting steps -- i.e. `Scope.local`.
+
+A few more examples of filtering an arbitrary object based on a anonymous traversal is provided below.
+
+[gremlin-groovy,modern]
+----
+g.V().where(out('created')).values('name') <1>
+g.V().out('knows').where(out('created')).values('name') <2>
+g.V().where(out('created').count().is(gte(2))).values('name') <3>
+g.V().where(out('knows').where(out('created'))).values('name') <4>
+g.V().where(__.not(out('created'))).where(__.in('knows')).values('name') <5>
+g.V().where(__.not(out('created')).and().in('knows')).values('name') <6>
+----
+
+<1> What are the names of the people who have created a project?
+<2> What are the names of the people that are known by someone one and have created a project?
+<3> What are the names of the people how have created two or more projects?
+<4> What are the names of the people who know someone that has created a project? (This only works in OLTP -- see the `WARNING` below)
+<5> What are the names of the people who have not created anything, but are known by someone?
+<6> The concatenation of `where()`-steps is the same as a single `where()`-step with an and'd clause.
+
+WARNING: The anonymous traversal of `where()` processes the current object "locally". In OLAP, where the atomic unit
+of computing is the the vertex and its local "star graph," it is important that the anonymous traversal does not leave
+the confines of the vertex's star graph. In other words, it can not traverse to an adjacent vertex's properties or
+edges. Note that is only a temporary limitation that will be addressed in a future version of TinkerPop3 (see
+link:https://issues.apache.org/jira/browse/TINKERPOP3-693[JIRA#693]).
+
+[[a-note-on-predicates]]
+A Note on Predicates
+--------------------
+
+A `P` is a predicate of the form `Function<Object,Boolean>`. That is, given some object, return true or false. The
+provided predicates are outlined in the table below and are used in various steps such as <<has-step,`has()`>>-step,
+<<where-step,`where()`>>-step, <<is-step,`is()`>>-step, etc.
+
+[width="100%",cols="3,15",options="header"]
+|=========================================================
+| Predicate | Description
+| `eq(object)` | Is the incoming object equal to the provided object?
+| `neq(object)` | Is the incoming object not equal to the provided object?
+| `lt(number)` | Is the incoming number less than the provided number?
+| `lte(number)` | Is the incoming number less than or equal to the provided number?
+| `gt(number)` | Is the incoming number greater than the provided number?
+| `gte(number)` | Is the incoming number greater than or equal to the provided number?
+| `inside(number,number)` | Is the incoming number greater than the first provided number and less than the second?
+| `outside(number,number)` | Is the incoming number less than the first provided number and greater than the second?
+| `between(number,number)` | Is the incoming number greater than or equal to the first provided number and less than the second?
+| `within(objects...)` | Is the incoming object in the array of provided objects?
+| `without(objects...)` | Is the incoming object not in the array of the provided objects?
+|=========================================================
+
+[gremlin-groovy]
+----
+eq(2)
+not(neq(2)) <1>
+not(within('a','b','c'))
+not(within('a','b','c')).test('d') <2>
+not(within('a','b','c')).test('a')
+within(1,2,3).and(not(eq(2))).test(3) <3>
+inside(1,4).or(eq(5)).test(3) <4>
+inside(1,4).or(eq(5)).test(5)
+between(1,2) <5>
+not(between(1,2))
+----
+
+<1> The `not()` of a `P`-predicate is another `P`-predicate.
+<2> `P`-predicates are arguments to various steps which internally `test()` the incoming value.
+<3> `P`-predicates can be and'd together.
+<4> `P`-predicates can be or' together.
+<5> `and()` is a `P`-predicate and thus, a `P`-predicate can be composed of multiple `P`-predicates.
+
+Finally, note that <<where-step,`where()`>>-step takes a `P<String>`. The provided string value refers to a variable
+binding, not to the explicit string value.
+
+[gremlin-groovy,modern]
+----
+g.V().as('a').both().both().as('b').count()
+g.V().as('a').both().both().as('b').where('a',neq('b')).count()
+----
+
+NOTE: It is possible for graph system providers and users to extend `P` and provide new predicates. For instance, a
+`regex(pattern)` could be a graph system specific `P`.
+
+[[a-note-on-barrier-steps]]
+A Note on Barrier Steps
+-----------------------
+
+image:barrier.png[width=165,float=right] Gremlin is primarily a
+link:http://en.wikipedia.org/wiki/Lazy_evaluation[lazy], stream processing language. This means that Gremlin fully
+processes (to the best of its abilities) any traversers currently in the traversal pipeline before getting more data
+from the start/head of the traversal. However, there are numerous situations in which a completely lazy computation
+is not possible (or impractical). When a computation is not lazy, a "barrier step" exists. There are thre
<TRUNCATED>
[13/22] incubator-tinkerpop git commit: Made subdirectories for
various "books" in the docs.
Posted by sp...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/gremlin-applications.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/gremlin-applications.asciidoc b/docs/src/gremlin-applications.asciidoc
deleted file mode 100644
index 7ef7a71..0000000
--- a/docs/src/gremlin-applications.asciidoc
+++ /dev/null
@@ -1,1760 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-[[gremlin-applications]]
-Gremlin Applications
-====================
-
-Gremlin applications represent tools that are built on top of the core APIs to help expose common functionality to
-users when working with graphs. There are two key applications:
-
-. Gremlin Console - A link:http://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop[REPL] environment for
-interactive development and analysis
-. Gremlin Server - A server that hosts script engines thus enabling remote Gremlin execution
-
-image:gremlin-lab-coat.png[width=310,float=left] Gremlin is designed to be extensible, making it possible for users
-and graph system/language providers to customize it to their needs. Such extensibility is also found in the Gremlin
-Console and Server, where a universal plugin system makes it possible to extend their capabilities. One of the
-important aspects of the plugin system is the ability to help the user install the plugins through the command line
-thus automating the process of gathering dependencies and other error prone activities.
-
-The process of plugin installation is handled by link:http://groovy.codehaus.org/Grape[Grape], which helps resolve
-dependencies into the classpath. It is therefore important to ensure that Grape is properly configured in order to
-use the automated capabilities of plugin installation. Grape is configured by `~/.groovy/grapeConfig.xml` and
-generally speaking, if that file is not present, the default settings will suffice. However, they will not suffice
-if a required dependency is not in one of the default configured repositories. Please see the
-link:http://groovy.codehaus.org/Grape[Custom Ivy Settings] section of the Grape documentation for more details on
-the defaults. TinkerPop recommends the following configuration in that file:
-
-[source,xml]
-<ivysettings>
- <settings defaultResolver="downloadGrapes"/>
- <resolvers>
- <chain name="downloadGrapes">
- <filesystem name="cachedGrapes">
- <ivy pattern="${user.home}/.groovy/grapes/[organisation]/[module]/ivy-[revision].xml"/>
- <artifact pattern="${user.home}/.groovy/grapes/[organisation]/[module]/[type]s/[artifact]-[revision].[ext]"/>
- </filesystem>
- <ibiblio name="codehaus" root="http://repository.codehaus.org/" m2compatible="true"/>
- <ibiblio name="central" root="http://central.maven.org/maven2/" m2compatible="true"/>
- <ibiblio name="java.net2" root="http://download.java.net/maven/2/" m2compatible="true"/>
- <ibiblio name="hyracs-releases" root="http://obelix.ics.uci.edu/nexus/content/groups/hyracks-public-releases/" m2compatible="true"/>
- </chain>
- </resolvers>
-</ivysettings>
-
-Note that if the intention is to work with TinkerPop snapshots then the file should also include:
-
-[source,xml]
-<ibiblio name="apache-snapshots" root="http://repository.apache.org/snapshots/" m2compatible="true"/>
-
-Additionally, the Graph configuration can also be modified to include the local system's Maven `.m2` directory by including:
-
-[source,xml]
-<ibiblio name="local" root="file:${user.home}/.m2/repository/" m2compatible="true"/>
-
-This configuration is useful during development (i.e. if one is working with locally built artifacts) of TinkerPop
-Plugins. Consider adding the "local" reference first in the set of `<ibilio>` resolvers, as putting it after
-"apache-snapshots" will likely resolve dependencies from that repository before looking locally. If it does that,
-then it's possible that the artifact from the newer local build will not be used.
-
-CAUTION: If building TinkerPop from source, be sure to clear TinkerPop-related jars from the `~/.groovy/grapes`
-directory as they can become stale on some systems and not re-import properly from the local `.m2` after fresh rebuilds.
-
-[[gremlin-console]]
-Gremlin Console
----------------
-
-image:gremlin-console.png[width=325,float=right] The Gremlin Console is an interactive terminal or
-link:http://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop[REPL] that can be used to traverse graphs
-and interact with the data that they contain. It represents the most common method for performing ad-hoc graph
-analysis, small to medium sized data loading projects and other exploratory functions. The Gremlin Console is
-highly extensible, featuring a rich plugin system that allows new tools, commands,
-link:http://en.wikipedia.org/wiki/Domain-specific_language[DSLs], etc. to be exposed to users.
-
-To start the Gremlin Console, run `gremlin.sh` or `gremlin.bat`:
-
-[source,text]
-----
-$ bin/gremlin.sh
-
- \,,,/
- (o o)
------oOOo-(3)-oOOo-----
-plugin loaded: tinkerpop.server
-plugin loaded: tinkerpop.utilities
-plugin loaded: tinkerpop.tinkergraph
-gremlin>
-----
-
-NOTE: If the above plugins are not loaded then they will need to be enabled or else certain examples will not work.
-If using the standard Gremlin Console distribution, then the plugins should be enabled by default. See below for
-more information on the `:plugin use` command to manually enable plugins. These plugins, with the exception of
-`tinkerpop.tinkergraph`, cannot be removed from the Console as they are a part of the `gremlin-console.jar` itself.
-These plugins can only be deactivated.
-
-The Gremlin Console is loaded and ready for commands. Recall that the console hosts the Gremlin-Groovy language.
-Please review link:http://groovy.codehaus.org/[Groovy] for help on Groovy-related constructs. In short, Groovy is a
-superset of Java. What works in Java, works in Groovy. However, Groovy provides many shorthands to make it easier
-to interact with the Java API. Moreoever, Gremlin provides many neat shorthands to make it easier to express paths
-through a property graph.
-
-[gremlin-groovy]
-----
-i = 'goodbye'
-j = 'self'
-i + " " + j
-"${i} ${j}"
-----
-
-The "toy" graph provides a way to get started with Gremlin quickly.
-
-[gremlin-groovy]
-----
-g = TinkerFactory.createModern().traversal(standard())
-g.V()
-g.V().values('name')
-g.V().has('name','marko').out('knows').values('name')
-----
-
-TIP: When using Gremlin-Groovy in a Groovy class file, add `static { GremlinLoader.load() }` to the head of the file.
-
-Console Commands
-~~~~~~~~~~~~~~~~
-
-In addition to the standard commands of the link:http://groovy.codehaus.org/Groovy+Shell[Groovy Shell], Gremlin adds
-some other useful operations. The following table outlines the most commonly used commands:
-
-[width="100%",cols="3,^2,10",options="header"]
-|=========================================================
-|Command |Alias |Description
-|:help |:? |Displays list of commands and descriptions. When followed by a command name, it will display more specific help on that particular item.
-|:exit |:x |Ends the Console session.
-|import |:i |Import a class into the Console session.
-|:clear |:c |Sometimes the Console can get into a state where the command buffer no longer understands input (e.g. a misplaced `(` or `}`). Use this command to clear that buffer.
-|:load |:l |Load a file or URL into the command buffer for execution.
-|:install |:+ |Imports a maven library and its dependencies into the Console.
-|:uninstall |:- |Removes a maven library and its dependencies. A restart of the console is required for removal to fully take effect.
-|:plugin |:pin |Plugin management functions to list, activate and deactivate available plugins.
-|:remote |:rem |Configures a "remote" context where Gremlin or results of Gremlin will be processed via usage of `:submit`.
-|:submit |:> |Submit Gremlin to the currently active context defined by `:remote`.
-|=========================================================
-
-Gremlin Console adds a special `max-iteration` preference that can be configured with the standard `:set` command
-from the Groovy Shell. Use this setting to control the maximum number of results that the Console will display.
-Consider the following usage:
-
-[gremlin-groovy]
-----
-:set max-iteration 10
-(0..200)
-:set max-iteration 5
-(0..200)
-----
-
-If this setting is not present, the console will default the maximum to 100 results.
-
-Dependencies and Plugin Usage
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The Gremlin Console can dynamically load external code libraries and make them available to the user. Furthermore,
-those dependencies may contain Gremlin plugins which can expand the language, provide useful functions, etc. These
-important console features are managed by the `:install` and `:plugin` commands.
-
-The following Gremlin Console session demonstrates the basics of these features:
-
-[source,groovy]
-----
-gremlin> :plugin list <1>
-==>tinkerpop.server[active]
-==>tinkerpop.gephi
-==>tinkerpop.utilities[active]
-==>tinkerpop.sugar
-==>tinkerpop.tinkergraph[active]
-gremlin> :plugin use tinkerpop.sugar <2>
-==>tinkerpop.sugar activated
-gremlin> :install org.apache.tinkerpop neo4j-gremlin x.y.z <3>
-==>loaded: [org.apache.tinkerpop, neo4j-gremlin, x.y.z]
-gremlin> :plugin list <4>
-==>tinkerpop.server[active]
-==>tinkerpop.gephi
-==>tinkerpop.utilities[active]
-==>tinkerpop.sugar
-==>tinkerpop.tinkergraph[active]
-==>tinkerpop.neo4j
-gremlin> :plugin use tinkerpop.neo4j <5>
-==>tinkerpop.neo4j activated
-gremlin> :plugin list <6>
-==>tinkerpop.server[active]
-==>tinkerpop.gephi
-==>tinkerpop.sugar[active]
-==>tinkerpop.utilities[active]
-==>tinkerpop.neo4j[active]
-==>tinkerpop.tinkergraph[active]
-----
-
-<1> Show a list of "available" plugins. The list of "available" plugins is determined by the classes available on
-the Console classpath. Plugins need to be "active" for their features to be available.
-<2> To make a plugin "active" execute the `:plugin use` command and specify the name of the plugin to enable.
-<3> Sometimes there are external dependencies that would be useful within the Console. To bring those in, execute
-`:install` and specify the Maven coordinates for the dependency.
-<4> Note that there is a "tinkerpop.neo4j" plugin available, but it is not yet "active".
-<5> Again, to use the "tinkerpop.neo4j" plugin, it must be made "active" with `:plugin use`.
-<6> Now when the plugin list is displayed, the "tinkerpop.neo4j" plugin is displayed as "active".
-
-CAUTION: Plugins must be compatible with the version of the Gremlin Console (or Gremlin Server) being used. Attempts
-to use incompatible versions cannot be guaranteed to work. Moreover, be prepared for dependency conflicts in
-third-party plugins, that may only be resolved via manual jar removal from the `ext/{plugin}` directory.
-
-TIP: It is possible to manage plugin activation and deactivation by manually editing the `ext/plugins.txt` file which
-contains the class names of the "active" plugins. It is also possible to clear dependencies added by `:install` by
-deleting them from the `ext` directory.
-
-Script Executor
-~~~~~~~~~~~~~~~
-
-For automated tasks and batch executions of Gremlin, it can be useful to execute Gremlin scripts from the command
-line. Consider the following file named `gremlin.groovy`:
-
-[source,groovy]
-----
-import org.apache.tinkerpop.gremlin.tinkergraph.structure.*
-graph = TinkerFactory.createModern()
-g = graph.traversal()
-g.V().each { println it }
-----
-
-This script creates the toy graph and then iterates through all its vertices printing each to the system out. Note
-that under this approach, "imports" need to be explicitly defined (except for "core" TinkerPop classes). In addition,
-plugins and other dependencies should already be "installed" via console commands which cannot be used with this mode
-of execution. To execute this script from the command line, `gremlin.sh` has the `-e` option used as follows:
-
-[source,bash]
-----
-$ bin/gremlin.sh -e gremlin.groovy
-v[1]
-v[2]
-v[3]
-v[4]
-v[5]
-v[6]
-----
-
-It is also possible to pass arguments to scripts. Any parameters following the file name specification are treated
-as arguments to the script. They are collected into a list and passed in as a variable called "args". The following
-Gremlin script is exactly like the previous one, but it makes use of the "args" option to filter the vertices printed
-to system out:
-
-[source,groovy]
-----
-import org.apache.tinkerpop.gremlin.tinkergraph.structure.*
-graph = TinkerFactory.createModern()
-g = graph.traversal()
-g.V().has('name',args[0]).each { println it }
-----
-
-When executed from the command line a parameter can be supplied:
-
-[source,bash]
-----
-$ bin/gremlin.sh -e gremlin.groovy marko
-v[1]
-$ bin/gremlin.sh -e gremlin.groovy vadas
-v[2]
-----
-
-[[gremlin-server]]
-Gremlin Server
---------------
-
-image:gremlin-server.png[width=400,float=right] Gremlin Server provides a way to remotely execute Gremlin scripts
-against one or more `Graph` instances hosted within it. The benefits of using Gremlin Server include:
-
-* Allows any Gremlin Structure-enabled graph to exist as a standalone server, which in turn enables the ability for
-multiple clients to communicate with the same graph database.
-* Enables execution of ad-hoc queries through remotely submitted Gremlin scripts.
-* Allows for the hosting of Gremlin-based DSLs (Domain Specific Language) that expand the Gremlin language to match
-the language of the application domain, which will help support common graph use cases such as searching, ranking,
-and recommendation.
-* Provides a method for Non-JVM languages (e.g. Python, Javascript, etc.) to communicate with the TinkerPop stack.
-* Exposes numerous methods for extension and customization to include serialization options, remote commands, etc.
-
-NOTE: Gremlin Server is the replacement for link:http://rexster.tinkerpop.com[Rexster].
-
-By default, communication with Gremlin Server occurs over link:http://en.wikipedia.org/wiki/WebSocket[WebSockets] and
-exposes a custom sub-protocol for interacting with the server.
-
-[[connecting-via-console]]
-Connecting via Console
-~~~~~~~~~~~~~~~~~~~~~~
-
-The most direct way to get started with Gremlin Server is to issue it some remote Gremlin scripts from the Gremlin
-Console. To do that, first start Gremlin Server:
-
-[source,text]
-----
-$ bin/gremlin-server.sh conf/gremlin-server-modern.yaml
-[INFO] GremlinServer -
- \,,,/
- (o o)
------oOOo-(3)-oOOo-----
-
-[INFO] GremlinServer - Configuring Gremlin Server from conf/gremlin-server-modern.yaml
-[INFO] MetricManager - Configured Metrics Slf4jReporter configured with interval=180000ms and loggerName=org.apache.tinkerpop.gremlin.server.Settings$Slf4jReporterMetrics
-[INFO] Graphs - Graph [graph] was successfully configured via [conf/tinkergraph-empty.properties].
-[INFO] ServerGremlinExecutor - Initialized Gremlin thread pool. Threads in pool named with pattern gremlin-*
-[INFO] ScriptEngines - Loaded gremlin-groovy ScriptEngine
-[INFO] GremlinExecutor - Initialized gremlin-groovy ScriptEngine with scripts/generate-modern.groovy
-[INFO] ServerGremlinExecutor - Initialized GremlinExecutor and configured ScriptEngines.
-[INFO] ServerGremlinExecutor - A GraphTraversalSource is now bound to [g] with graphtraversalsource[tinkergraph[vertices:0 edges:0], standard]
-[INFO] GremlinServer - Executing start up LifeCycleHook
-[INFO] Logger$info - Loading 'modern' graph data.
-[INFO] AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo with org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0
-[INFO] AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo-stringd with org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0
-[INFO] GremlinServer$1 - Gremlin Server configured with worker thread pool of 1, gremlin pool of 8 and boss thread pool of 1.
-[INFO] GremlinServer$1 - Channel started at port 8182.
-----
-
-Gremlin Server is configured by the provided link:http://www.yaml.org/[YAML] file `conf/gremlin-server-modern.yaml`.
-That file tells Gremlin Server many things such as:
-
-* The host and port to serve on
-* Thread pool sizes
-* Where to report metrics gathered by the server
-* The serializers to make available
-* The Gremlin `ScriptEngine` instances to expose and external dependencies to inject into them
-* `Graph` instances to expose
-
-The log messages that printed above show a number of things, but most importantly, there is a `Graph` instance named
-`graph` that is exposed in Gremlin Server. This graph is an in-memory TinkerGraph and was empty at the start of the
-server. An initialization script at `scripts/generate-modern.groovy` was executed during startup. It's contents are
-as follows:
-
-[source,groovy]
-----
-include::{basedir}/gremlin-server/scripts/generate-modern.groovy[]
-----
-
-The script above initializes a `Map` and assigns two key/values to it. The first, assigned to "hook", defines a
-`LifeCycleHook` for Gremlin Server. The "hook" provides a way to tie script code into the Gremlin Server startup and
-shutdown sequences. The `LifeCycleHook` has two methods that can be implemented: `onStartUp` and `onShutDown`.
-These events are called once at Gremlin Server start and once at Gremlin Server stop. This is an important point
-because code outside of the "hook" is executed for each `ScriptEngine` creation (multiple may be created when
-"sessions" are enabled) and therefore the `LifeCycleHook` provides a way to ensure that a script is only executed a
-single time. In this case, the startup hook loads the "modern" graph into the empty TinkerGraph instance, preparing
-it for use. The second key/value pair assigned to the `Map`, named "g", defines a `TraversalSource` from the `Graph`
-bound to the "graph" variable in the YAML configuration file. This variable `g`, as well as any other variable
-assigned to the `Map`, will be made available as variables for future remote script executions. In more general
-terms, any key/value pairs assigned to a `Map` returned from the init script will become variables that are global
-to all requests. In addition, any functions that are defined will be cached for future use.
-
-With Gremlin Server running it is now possible to issue some scripts to it for processing. Start Gremlin Console as follows:
-
-[source,text]
-----
-$ bin/gremlin.sh
-
- \,,,/
- (o o)
------oOOo-(3)-oOOo-----
-gremlin>
-----
-
-The console has the notion of a "remote", which represents a place a script will be sent from the console to be
-evaluated elsewhere in some other context (e.g. Gremlin Server, Hadoop, etc.). To create a remote in the console,
-do the following:
-
-[gremlin-groovy]
-----
-:remote connect tinkerpop.server conf/remote.yaml
-----
-
-The `:remote` command shown above displays the current status of the remote connection. This command can also be
-used to configure a new connection and change other related settings. To actually send a script to the server a
-different command is required:
-
-[gremlin-groovy]
-----
-:> g.V().values('name')
-:> g.V().has('name','marko').out('created').values('name')
-:> g.E().label().groupCount()
-result
-:remote close
-----
-
-The `:>` command, which is a shorthand for `:submit`, sends the script to the server to execute there. Results are
-wrapped in an `Result` object which is a just a holder for each individual result. The `class` shows the data type
-for the containing value. Note that the last script sent was supposed to return a `Map`, but its `class` is
-`java.lang.String`. By default, the connection is configured to only return text results. In other words,
-Gremlin Server is using `toString` to serialize all results back to the console. This enables virtually any
-object on the server to be returned to the console, but it doesn't allow the opportunity to work with this data
-in any way in the console itself. A different configuration of the `:remote` is required to get the results back
-as "objects":
-
-[gremlin-groovy]
-----
-:remote connect tinkerpop.server conf/remote-objects.yaml <1>
-:remote list <2>
-:> g.E().label().groupCount() <3>
-m = result[0].object <4>
-m.sort {it.value}
-script = """
- matthias = graph.addVertex('name','matthias')
- matthias.addEdge('co-creator',g.V().has('name','marko').next())
- """
-:> @script <5>
-:> g.V().has('name','matthias').out('co-creator').values('name')
-:remote close
-----
-
-<1> This configuration file specifies that results should be deserialized back into an `Object` in the console with
-the caveat being that the server and console both know how to serialize and deserialize the result to be returned.
-<2> There are now two configured remote connections. The one marked by an asterisk is the one that was just created
-and denotes the current one that `:sumbit` will react to.
-<3> When the script is executed again, the `class` is no longer shown to be a `java.lang.String`. It is instead a `java.util.HashMap`.
-<4> The last result of a remote script is always stored in the reserved variable `result`, which allows access to
-the `Result` and by virtue of that, the `Map` itself.
-<5> If the submission requires multiple-lines to express, then a multi-line string can be created. The `:>` command
-realizes that the user is referencing a variable via `@` and submits the string script.
-
-TIP: In Groovy, `""" text """` is a convenient way to create a multi-line string and works well in concert with
-`:> @variable`. Note that this model of submitting a string variable works for all `:>` based plugins, not just Gremlin Server.
-
-WARNING: Not all values that can be returned from a Gremlin script end up being serializable. For example,
-submitting `:> graph` will return a `Graph` instance and in most cases those are not serializable by Gremlin Server
-and will return a serialization error. It should be noted that `TinkerGraph`, as a convenience for shipping around
-small sub-graphs, is serializable from Gremlin Server.
-
-The Gremlin Server `:remote` command has the following configuration options:
-
-[width="100%",cols="3,10a",options="header"]
-|=========================================================
-|Command |Description
-|alias |
-[width="100%",cols="3,10",options="header"]
-!=========================================================
-!Option !Description
-! _pairs_ !A set of key/value alias/binding pairs to apply to requests.
-!`reset` !Clears any aliases that were supplied in previous configurations of the remote.
-!`show` !Shows the current set of aliases which is returned as a `Map`
-!=========================================================
-|timeout |Specifies the length of time in milliseconds that the remote will wait for a response from the server.
-|=========================================================
-
-The `alias` configuration command for the Gremlin Server `:remote` can be useful in situations where there are
-multiple `Graph` or `TraversalSource` instances on the server, as it becomes possible to rename them from the client
-for purposes of execution within the context of a script. Therefore, it becomes possible to submit commands this way:
-
-[gremlin-groovy]
-----
-:remote connect tinkerpop.server conf/remote-objects.yaml
-:remote config alias x g
-:> x.E().label().groupCount()
-----
-
-Connecting via Java
-~~~~~~~~~~~~~~~~~~~
-
-[source,xml]
-----
-<dependency>
- <groupId>org.apache.tinkerpop</groupId>
- <artifactId>gremlin-driver</artifactId>
- <version>x.y.z</version>
-</dependency>
-----
-
-image:gremlin-java.png[width=175,float=left] TinkerPop3 comes equipped with a reference client for Java-based
-applications. It is referred to as Gremlin Driver, which enables applications to send requests to Gremlin Server
-and get back results.
-
-Gremlin code is sent to the server from a `Client` instance. A `Client` is created as follows:
-
-[source,java]
-----
-Cluster cluster = Cluster.open(); <1>
-Client client = cluster.connect(); <2>
-----
-
-<1> Opens a reference to `localhost` - note that there are many configuration options available in defining a `Cluster` object.
-<2> Creates a `Client` given the configuration options of the `Cluster`.
-
-Once a `Client` instance is ready, it is possible to issue some Gremlin:
-
-[source,java]
-----
-ResultSet results = client.submit("[1,2,3,4]"); <1>
-results.stream().map(i -> i.get(Integer.class) * 2); <2>
-
-CompletableFuture<List<Result>> results = client.submit("[1,2,3,4]").all(); <3>
-
-CompletableFuture<ResultSet> future = client.submitAsync("[1,2,3,4]"); <4>
-
-Map<String,Object> params = new HashMap<>()
-params.put("x",4)
-client.submit("[1,2,3,x]", params); <5>
-----
-
-<1> Submits a script that simply returns a `List` of integers. This method blocks until the request is written to
-the server and a `ResultSet` is constructed.
-<2> Even though the `ResultSet` is constructed, it does not mean that the server has sent back the results (or even
-evaluated the script potentially). The `ResultSet` is just a holder that is awaiting the results from the server.
-In this case, they are streamed from the server as they arrive.
-<3> Submit a script, get a `ResultSet`, then return a `CompletableFuture` that will be called when all results have been returned.
-<4> Submit a script asynchronously without waiting for the request to be written to the server.
-<5> Parameterized request are considered the most efficient way to send Gremlin to the server as they can be cached,
-which will boost performance and reduce resources required on the server.
-
-Aliases
-^^^^^^^
-
-Scripts submitted to Gremlin Server automatically have the globally configured `Graph` and `TraversalSource` instances
-made available to them. Therefore, if Gremlin Server configures two `TraversalSource` instances called "g1" and "g2"
-a script can simply reference them directly as:
-
-[source,java]
-client.submit("g1.V()")
-client.submit("g2.V()")
-
-While this is an acceptable way to submit scripts, it has the downside of forcing the client to encode the server-side
-variable name directly into the script being sent. If the server configuration ever changed such that "g1" became
-"g100", the client-side code might have to see a significant amount of change. Decoupling the script code from the
-server configuration can be managed by the `alias` method on `Client` as follows:
-
-[source,java]
-Client g1Client = client.alias("g1")
-Client g2Client = client.alias("g2")
-g1Client.submit("g.V()")
-g2Client.submit("g.V()")
-
-The above code demonstrates how the `alias` method can be used such that the script need only contain a reference
-to "g" and "g1" and "g2" are automatically rebound into "g" on the server-side.
-
-Serialization
-^^^^^^^^^^^^^
-
-When using Gryo serialization (the default serializer for the driver), it is important that the client and server
-have the same serializers configured or else one or the other will experience serialization exceptions and fail to
-always communicate. Discrepancy in serializer registration between client and server can happen fairly easily as
-graphs will automatically include serializers on the server-side, thus leaving the client to be configured manually.
-This can be done manually as follows:
-
-[source,java]
-GryoMapper kryo = GryoMapper.build().addRegistry(TitanIoRegistry.INSTANCE).create();
-MessageSerializer serializer = new GryoMessageSerializerV1d0(kryo);
-Cluster cluster = Cluster.build()
- .serializer(serializer)
- .create();
-Client client = cluster.connect().init();
-
-The above code demonstrates using the `TitanIoRegistry` which is an `IoRegistry` instance. It tells the serializer
-what classes (from Titan in this case) to auto-register during serialization. Gremlin Server roughly uses this same
-approach when it configures it's serializers, so using this same model will ensure compatibility when making requests.
-
-Connecting via REST
-~~~~~~~~~~~~~~~~~~~
-
-image:gremlin-rexster.png[width=225,float=left] While the default behavior for Gremlin Server is to provide a
-WebSockets-based connection, it can also be configured to support link:http://en.wikipedia.org/wiki/Representational_state_transfer[REST].
-The REST endpoint provides for a communication protocol familiar to most developers, with a wide support of
-programming languages, tools and libraries for accessing it. As a result, REST provides a fast way to get started
-with Gremlin Server. It also may represent an easier upgrade path from link:http://rexster.tinkerpop.com/[Rexster]
-as the API for the endpoint is very similar to Rexster's link:https://github.org/apache/tinkerpop/rexster/wiki/Gremlin-Extension[Gremlin Extension].
-
-Gremlin Server provides for a single REST endpoint - a Gremlin evaluator - which allows the submission of a Gremlin
-script as a request. For each request, it returns a response containing the serialized results of that script.
-To enable this endpoint, Gremlin Server needs to be configured with the `HttpChannelizer`, which replaces the default
-`WebSocketChannelizer`, in the configuration file:
-
-[source,yaml]
-channelizer: org.apache.tinkerpop.gremlin.server.channel.HttpChannelizer
-
-This setting is already configured in the `gremlin-server-rest-modern.yaml` file that is packaged with the Gremlin
-Server distribution. To utilize it, start Gremlin Server as follows:
-
-[source,text]
-bin/gremlin-server.sh conf/gremlin-server-rest-modern.yaml
-
-Once the server has started, issue a request. Here's an example with link:http://curl.haxx.se/[cURL]:
-
-[source,text]
-$ curl "http://localhost:8182?gremlin=100-1"
-
-which returns:
-
-[source,js]
-{
- "result":{"data":99,"meta":{}},
- "requestId":"0581cdba-b152-45c4-80fa-3d36a6eecf1c",
- "status":{"code":200,"attributes":{},"message":""}
-}
-
-The above example showed a `GET` operation, but the preferred method for this endpoint is `POST`:
-
-[source,text]
-curl -X POST -d "{\"gremlin\":\"100-1\"}" "http://localhost:8182"
-
-which returns:
-
-[source,js]
-{
- "result":{"data":99,"meta":{}},
- "requestId":"ef2fe16c-441d-4e13-9ddb-3c7b5dfb10ba",
- "status":{"code":200,"attributes":{},"message":""}
-}
-
-It is also preferred that Gremlin scripts be parameterized when possible via `bindings`:
-
-[source,text]
-curl -X POST -d "{\"gremlin\":\"100-x\", \"bindings\":{\"x\":1}}" "http://localhost:8182"
-
-The `bindings` argument is a `Map` of variables where the keys become available as variables in the Gremlin script.
-Note that parameterization of requests is critical to performance, as repeated script compilation can be avoided on
-each request.
-
-NOTE: It is possible to pass bindings via `GET` based requests. Query string arguments prefixed with "bindings." will
-be treated as parameters, where that prefix will be removed and the value following the period will become the
-parameter name. In other words, `bindings.x` will create a parameter named "x" that can be referenced in the submitted
-Gremlin script. The caveat is that these arguments will always be treated as `String` values. To ensure that data
-types are preserved or to pass complex objects such as lists or maps, use `POST` which will at least support the
-allowed JSON data types.
-
-Finally, as Gremlin Server can host multiple `ScriptEngine` instances (e.g. `gremlin-groovy`, `nashorn`), it is
-possible to define the language to utilize to process the request:
-
-[source,text]
-curl -X POST -d "{\"gremlin\":\"100-x\", \"language\":\"gremlin-groovy\", \"bindings\":{\"x\":1}}" "http://localhost:8182"
-
-By default this value is set to `gremlin-groovy`. If using a `GET` operation, this value can be set as a query
-string argument with by setting the `language` key.
-
-CAUTION: Consider the size of the result of a submitted script being returned from the REST endpoint. A script
-that iterates thousands of results will serialize each of those in memory into a single JSON result set. It is
-quite possible that such a script will generate `OutOfMemoryError` exceptions on the server. Consider the default
-WebSockets configuration, which supports streaming, if that type of use case is required.
-
-Configuring
-~~~~~~~~~~~
-
-As mentioned earlier, Gremlin Server is configured though a YAML file. By default, Gremlin Server will look for a
-file called `config/gremlin-server.yaml` to configure itself on startup. To override this default, supply the file
-to use to `bin/gremlin-server.sh` as in:
-
-[source,text]
-----
-bin/gremlin-server.sh conf/gremlin-server-min.yaml
-----
-
-The `gremlin-server.sh` file also serves a second purpose. It can be used to "install" dependencies to the Gremlin
-Server path. For example, to be able to configure and use other `Graph` implementations, the dependencies must be
-made available to Gremlin Server. To do this, use the `-i` switch and supply the Maven coordinates for the dependency
-to "install". For example, to use Neo4j in Gremlin Server:
-
-[source,text]
-----
-bin/gremlin-server.sh -i org.apache.tinkerpop neo4j-gremlin x.y.z
-----
-
-This command will "grab" the appropriate dependencies and copy them to the `ext` directory of Gremlin Server, which
-will then allow them to be "used" the next time the server is started. To uninstall dependencies, simply delete them
-from the `ext` directory.
-
-The following table describes the various configuration options that Gremlin Server expects:
-
-[width="100%",cols="3,10,^2",options="header"]
-|=========================================================
-|Key |Description |Default
-|authentication.className |The fully qualified classname of an `Authenticator` implementation to use. If this setting is not present, then authentication is effectively disabled. |`AllowAllAuthenticator`
-|authentication.config |A `Map` of configuration settings to be passes to the `Authenticator` when it is constructed. The settings available are dependent on the implementation. |_none_
-|channelizer |The fully qualified classname of the `Channelizer` implementation to use. A `Channelizer` is a "channel initializer" which Gremlin Server uses to define the type of processing pipeline to use. By allowing different `Channelizer` implementations, Gremlin Server can support different communication protocols (e.g. Websockets, Java NIO, etc.). |WebSocketChannelizer
-|graphs |A `Map` of `Graph` configuration files where the key of the `Map` becomes the name to which the `Graph` will be bound and the value is the file name of a `Graph` configuration file. |_none_
-|gremlinPool |The number of "Gremlin" threads available to execute actual scripts in a `ScriptEngine`. This pool represents the workers available to handle blocking operations in Gremlin Server. |8
-|host |The name of the host to bind the server to. |localhost
-|useEpollEventLoop |try to use epoll event loops (works only on Linux os) instead of netty NIO. |false
-|maxAccumulationBufferComponents |Maximum number of request components that can be aggregated for a message. |1024
-|maxChunkSize |The maximum length of the content or each chunk. If the content length exceeds this value, the transfer encoding of the decoded request will be converted to 'chunked' and the content will be split into multiple `HttpContent` objects. If the transfer encoding of the HTTP request is 'chunked' already, each chunk will be split into smaller chunks if the length of the chunk exceeds this value. |8192
-|maxContentLength |The maximum length of the aggregated content for a message. Works in concert with `maxChunkSize` where chunked requests are accumulated back into a single message. A request exceeding this size will return a `413 - Request Entity Too Large` status code. A response exceeding this size will raise an internal exception. |65536
-|maxHeaderSize |The maximum length of all headers. |8192
-|maxInitialLineLength |The maximum length of the initial line (e.g. "GET / HTTP/1.0") processed in a request, which essentially controls the maximum length of the submitted URI. |4096
-|metrics.consoleReporter.enabled |Turns on console reporting of metrics. |false
-|metrics.consoleReporter.interval |Time in milliseconds between reports of metrics to console. |180000
-|metrics.csvReporter.enabled |Turns on CSV reporting of metrics. |false
-|metrics.csvReporter.fileName |The file to write metrics to. |_none_
-|metrics.csvReporter.interval |Time in milliseconds between reports of metrics to file. |180000
-|metrics.gangliaReporter.addressingMode |Set to `MULTICAST` or `UNICAST`. |_none_
-|metrics.gangliaReporter.enabled |Turns on Ganglia reporting of metrics. |false
-|metrics.gangliaReporter.host |Define the Ganglia host to report Metrics to. |localhost
-|metrics.gangliaReporter.interval |Time in milliseconds between reports of metrics for Ganglia. |180000
-|metrics.gangliaReporter.port |Define the Ganglia port to report Metrics to. |8649
-|metrics.graphiteReporter.enabled |Turns on Graphite reporting of metrics. |false
-|metrics.graphiteReporter.host |Define the Graphite host to report Metrics to. |localhost
-|metrics.graphiteReporter.interval |Time in milliseconds between reports of metrics for Graphite. |180000
-|metrics.graphiteReporter.port |Define the Graphite port to report Metrics to. |2003
-|metrics.graphiteReporter.prefix |Define a "prefix" to append to metrics keys reported to Graphite. |_none_
-|metrics.jmxReporter.enabled |Turns on JMX reporting of metrics. |false
-|metrics.slf4jReporter.enabled |Turns on SLF4j reporting of metrics. |false
-|metrics.slf4jReporter.interval |Time in milliseconds between reports of metrics to SLF4j. |180000
-|plugins |A list of plugins that should be activated on server startup in the available script engines. It assumes that the plugins are in Gremlin Server's classpath. |_none_
-|port |The port to bind the server to. |8182
-|processors |A `List` of `Map` settings, where each `Map` represents a `OpProcessor` implementation to use along with its configuration. |_none_
-|processors[X].className |The full class name of the `OpProcessor` implementation. |_none_
-|processors[X].config |A `Map` containing `OpProcessor` specific configurations. |_none_
-|resultIterationBatchSize |Defines the size in which the result of a request is "batched" back to the client. In other words, if set to `1`, then a result that had ten items in it would get each result sent back individually. If set to `2` the same ten results would come back in five batches of two each. |64
-|scriptEngines |A `Map` of `ScriptEngine` implementations to expose through Gremlin Server, where the key is the name given by the `ScriptEngine` implementation. The key must match the name exactly for the `ScriptEngine` to be constructed. The value paired with this key is itself a `Map` of configuration for that `ScriptEngine`. |_none_
-|scriptEngines.<name>.imports |A comma separated list of classes/packages to make available to the `ScriptEngine`. |_none_
-|scriptEngines.<name>.staticImports |A comma separated list of "static" imports to make available to the `ScriptEngine`. |_none_
-|scriptEngines.<name>.scripts |A comma separated list of script files to execute on `ScriptEngine` initialization. `Graph` and `TraversalSource` instance references produced from scripts will be stored globally in Gremlin Server, therefore it is possible to use initialization scripts to add Traversal Strategies or create entirely new `Graph` instances all together. Instantiating a `LifeCycleHook` in a script provides a way to execute scripts when Gremlin Server starts and stops.|_none_
-|scriptEngines.<name>.config |A `Map` of configuration settings for the `ScriptEngine`. These settings are dependent on the `ScriptEngine` implementation being used. |_none_
-|scriptEvaluationTimeout |The amount of time in milliseconds before a script evaluation times out. The notion of "script evaluation" refers to the time it takes for the `ScriptEngine` to do its work and *not* any additional time it takes for the result of the evaluation to be iterated and serialized. |30000
-|serializers |A `List` of `Map` settings, where each `Map` represents a `MessageSerializer` implementation to use along with its configuration. |_none_
-|serializers[X].className |The full class name of the `MessageSerializer` implementation. |_none_
-|serializers[X].config |A `Map` containing `MessageSerializer` specific configurations. |_none_
-|serializedResponseTimeout |The amount of time in milliseconds before a response serialization times out. The notion of "response serialization" refers to the time it takes for Gremlin Server to iterate an entire result after the script is evaluated in the `ScriptEngine`. |30000
-|ssl.enabled |Determines if SSL is turned on or not. |false
-|ssl.keyCertChainFile |The X.509 certificate chain file in PEM format. If this value is not present and `ssl.enabled` is `true` a self-signed certificate will be used (not suitable for production). |_none_
-|ssl.keyFile |The `PKCS#8` private key file in PEM format. If this value is not present and `ssl.enabled` is `true` a self-signed certificate will be used (not suitable for production). |_none_
-|ssl.keyPassword |The password of the `keyFile` if it's not password-protected |_none_
-|ssl.trustCertChainFile |Trusted certificates for verifying the remote endpoint's certificate. The file should contain an X.509 certificate chain in PEM format. A system default will be used if this setting is not present. |_none_
-|threadPoolBoss |The number of threads available to Gremlin Server for accepting connections. Should always be set to `1`. |1
-|threadPoolWorker |The number of threads available to Gremlin Server for processing non-blocking reads and writes. |1
-|writeBufferHighWaterMark | If the number of bytes in the network send buffer exceeds this value then the channel is no longer writeable, accepting no additional writes until buffer is drained and the `writeBufferLowWaterMark` is met. |65536
-|writeBufferLowWaterMark | Once the number of bytes queued in the network send buffer exceeds the `writeBufferHighWaterMark`, the channel will not become writeable again until the buffer is drained and it drops below this value. |65536
-|=========================================================
-
-NOTE: Configuration of link:http://ganglia.sourceforge.net/[Ganglia] requires an additional library that is not
-packaged with Gremlin Server due to its LGPL licensing that conflicts with the TinkerPop's Apache 2.0 License. To
-run Gremlin Server with Ganglia monitoring, download the `org.acplt:oncrpc` jar from
-link:http://repo1.maven.org/maven2/org/acplt/oncrpc/1.0.7/[here] and copy it to the Gremlin Server `/lib` directory
-before starting the server.
-
-Security
-^^^^^^^^
-
-image:gremlin-server-secure.png[width=175,float=right] Gremlin Server provides for several features that aid in the
-security of the graphs that it exposes. It has built in SSL support and a pluggable authentication framework using
-link:https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer[SASL] (Simple Authentication and
-Security Layer). SSL options are described in the configuration settings table above, so this section will focus on
-authentication.
-
-By default, Gremlin Server is configured to allow all requests to be processed (i.e. no authentication). To enable
-authentication, Gremlin Server must be configured with an `Authenticator` implementation in its YAML file. Gremlin
-Server comes packaged with an implementation called `SimpleAuthenticator`. The `SimpleAuthenticator` implements the
-`PLAIN` SASL mechanism (i.e. plain text) to authenticate a request. It validates username/password pairs against a
-graph database, which must be provided to it as part of the configuration.
-
-[source,yaml]
-authentication: {
- className: org.apache.tinkerpop.gremlin.server.auth.SimpleAuthenticator,
- config: {
- credentialsDb: conf/credential-graph.properties}}
-
-Quick Start
-+++++++++++
-
-A quick way to get started with the `SimpleAuthenticator` is to use TinkerGraph for the "credentials graph" and the
-"sample" credential graph that is packaged with the server. Recall that TinkerGraph is an in-memory graph and
-therefore always starts as "empty" when opened by `GraphFactory`. To allow TinkerGraph to be used in this "getting
-started" capacity, Gremlin Server allows for a TinkerGraph-only configuration option called `credentialsDbLocation`.
-The following snippet comes from the `conf/gremlin-server-secure.yaml` file packaged with the server:
-
-[source,yaml]
-authentication: {
- className: org.apache.tinkerpop.gremlin.server.auth.SimpleAuthenticator,
- config: {
- credentialsDb: conf/tinkergraph-empty.properties,
- credentialsDbLocation: data/credentials.kryo}}
-
-This added configuration tells Gremlin Server to look for a gryo file at that location containing the data for the
-graph which it loads via standard `io` methods. The limitation is that this read is only performed at the
-initialization of the server so therefore credentials remain static for the life of the server. In this case,
-`data/credentials.kryo` contains a single user named "stephen" with the imaginative password of "password".
-
-[source,text]
-----
-$ bin/gremlin-server.sh conf/gremlin-server-secure.yaml
-[INFO] GremlinServer -
- \,,,/
- (o o)
------oOOo-(3)-oOOo-----
-
-[INFO] GremlinServer - Configuring Gremlin Server from conf/gremlin-server-secure.yaml
-...
-[WARN] AbstractChannelizer - Enabling SSL with self-signed certificate (NOT SUITABLE FOR PRODUCTION)
-[INFO] AbstractChannelizer - SSL enabled
-[INFO] SimpleAuthenticator - Initializing authentication with the org.apache.tinkerpop.gremlin.server.auth.SimpleAuthenticator
-[INFO] SimpleAuthenticator - CredentialGraph initialized at CredentialGraph{graph=tinkergraph[vertices:1 edges:0]}
-[INFO] GremlinServer$1 - Gremlin Server configured with worker thread pool of 1, gremlin pool of 8 and boss thread pool of 1.
-[INFO] GremlinServer$1 - Channel started at port 8182.
-----
-
-In addition to configuring the authenticator, `gremlin-server-secure.yaml` also enables SSL with a self-signed
-certificate. As SSL is enabled on the server it must also be enabled on the client when connecting. To connect to
-Gremlin Server with `gremlin-driver`, set the `credentials` and `enableSsl` when constructing the `Cluster`.
-
-[source,java]
-Cluster cluster = Cluster.build().credentials("stephen", "password")
- .enableSsl(true).create();
-
-If connecting with Gremlin Console, which utilizes `gremlin-driver` for remote script execution, use the provided
-`config/remote-secure.yaml` file when defining the remote. That file contains configuration for the username and
-password as well as enablement of SSL from the client side.
-
-Similarly, Gremlin Server can be configured for REST and security.
-
-[source,text]
-----
-$ bin/gremlin-server.sh conf/gremlin-server-rest-secure.yaml
-[INFO] GremlinServer -
- \,,,/
- (o o)
------oOOo-(3)-oOOo-----
-
-[INFO] GremlinServer - Configuring Gremlin Server from conf/gremlin-server-secure.yaml
-...
-[WARN] AbstractChannelizer - Enabling SSL with self-signed certificate (NOT SUITABLE FOR PRODUCTION)
-[INFO] AbstractChannelizer - SSL enabled
-[INFO] SimpleAuthenticator - Initializing authentication with the org.apache.tinkerpop.gremlin.server.auth.SimpleAuthenticator
-[INFO] SimpleAuthenticator - CredentialGraph initialized at CredentialGraph{graph=tinkergraph[vertices:1 edges:0]}
-[INFO] GremlinServer$1 - Gremlin Server configured with worker thread pool of 1, gremlin pool of 8 and boss thread pool of 1.
-[INFO] GremlinServer$1 - Channel started at port 8182.
-----
-
-Once the server has started, issue a request passing the credentials with an `Authentication` header, as described in link:http://tools.ietf.org/html/rfc2617#section-2[RFC2617]. Here's a HTTP Basic authentication example with cURL:
-
-[source,text]
-curl -X POST --insecure -u stephen:password -d "{\"gremlin\":\"100-1\"}" "https://localhost:8182"
-
-[[credentials-dsl]]
-Credentials Graph DSL
-+++++++++++++++++++++
-
-The "credentials graph", which has been mentioned in previous sections, is used by Gremlin Server to hold the list of
-users who can authenticate to the server. It is possible to use virtually any `Graph` instance for this task as long
-as it complies to a defined schema. The credentials graph stores users as vertices with the `label` of "user". Each
-"user" vertex has two properties: `username` and `password`. Naturally, these are both `String` values. The password
-must not be stored in plain text and should be hashed.
-
-IMPORTANT: Be sure to define an index on the `username` property, as this will be used for lookups. If supported by
-the `Graph`, consider specifying a unique constraint as well.
-
-To aid with the management of a credentials graph, Gremlin Server provides a Gremlin Console plugin which can be
-used to add and remove users so as to ensure that the schema is adhered to, thus ensuring compatibility with Gremlin
- Server. In addition, as it is a plugin, it works naturally in the Gremlin Console as an extension of its
- capabilities (though one could use it programmatically, if desired). This plugin is distributed with the Gremlin
- Console so it does not have to be "installed". It does however need to be activated:
-
-[source,groovy]
-gremlin> :plugin use tinkerpop.credentials
-==>tinkerpop.credentials activated
-
-Please see the example usage as follows:
-
-[gremlin-groovy]
-----
-graph = TinkerGraph.open()
-graph.createIndex("username",Vertex.class)
-credentials = credentials(graph)
-credentials.createUser("stephen","password")
-credentials.createUser("daniel","better-password")
-credentials.createUser("marko","rainbow-dash")
-credentials.findUser("marko").properties()
-credentials.countUsers()
-credentials.removeUser("daniel")
-credentials.countUsers()
-----
-
-[[script-execution]]
-Script Execution
-++++++++++++++++
-
-It is important to remember that Gremlin Server exposes a `ScriptEngine` instance that allows for remote execution
-of arbitrary code on the server. Obviously, this situation can represent a security risk or, more minimally, provide
-ways for "bad" scripts to be inadvertently executed. A simple example of a "valid" Gremlin script that would cause
-some problems would be, `while(true) {}`, which would consume a thread in the Gremlin pool indefinitely, thus
-preventing it from serving other requests. Sending enough of these kinds of scripts would eventually consume all
-available threads and Gremlin Server would stop responding.
-
-Gremlin Server (more specifically the `GremlinGroovyScriptEngine`) provides methods to protect itself from these
-kinds of troublesome scripts. A user can configure the script engine with different `CompilerCustomizerProvider`
-implementations. Consider the basic configuration from the Gremlin Server YAML file:
-
-[source,yaml]
-scriptEngines: {
- gremlin-groovy: {
- imports: [java.lang.Math],
- staticImports: [java.lang.Math.PI],
- scripts: [scripts/empty-sample.groovy]}}
-
-This configuration can be extended to include a `config` key as follows:
-
-[source,yaml]
-scriptEngines: {
- gremlin-groovy: {
- imports: [java.lang.Math],
- staticImports: [java.lang.Math.PI],
- scripts: [scripts/empty-sample.groovy],
- config: {
- compilerCustomizerProviders: {
- "org.apache.tinkerpop.gremlin.groovy.jsr223.customizer.TimedInterruptCustomizerProvider":[10000] }}}
-
-This configuration sets up the script engine with a `CompilerCustomizerProvider` implementation. The
-`TimedInterruptCustomizerProvider` injects checks that ensure that loops (like `while`) can only execute for `10000`
-milliseconds. With this configuration in place, a remote execution as follows, now times out rather than consuming
-the thread continuously:
-
-[source,groovy]
-gremlin> :remote connect tinkerpop.server conf/remote.yaml
-==>Connected - localhost/127.0.0.1:8182
-gremlin> :> while(true) { }
-Execution timed out after 10000 units. Start time: Fri Jul 24 11:04:52 EDT 2015
-
-There are a number of pre-packaged `CustomizerProvider` implementations:
-
-[width="100%",cols="3,10a",options="header"]
-|=========================================================
-|Customizer |Description
-|`CompileStaticCustomizerProvider` |Applies `CompileStatic` annotations to incoming scripts thus removing dynamic dispatch. More information about static compilation can be found in the link:http://docs.groovy-lang.org/latest/html/documentation/#_static_compilation[Groovy Documentation]. It is possible to configure this `CustomizerProvider` by specifying a comma separated list of link:http://docs.groovy-lang.org/latest/html/documentation/#Typecheckingextensions-Workingwithextensions[type checking extensions] that can have the effect of securing calls to various methods.
-|`ThreadInterruptCustomizerProvider` |Injects checks for thread interruption, thus allowing the thread to potentially respect calls to `Thread.interrupt()`
-|`TimedInterruptCustomizerProvider` |Injects checks into loops to interrupt them if they exceed the configured timeout in milliseconds.
-|`TypeCheckedCustomizerProvider` |Similar to the above mentioned, `CompileStaticCustomizerProvider`, the `TypeCheckedCustomizerProvider` injects `TypeChecked` annotations to incoming scripts. More information on the nature of this annotation can be found in the link:http://docs.groovy-lang.org/latest/html/documentation/#_the_code_typechecked_code_annotation[Groovy Documentation]. It too takes a comma separated list of link:http://docs.groovy-lang.org/latest/html/documentation/#Typecheckingextensions-Workingwithextensions[type checking extensions].
-|=========================================================
-
-To provide some basic out-of-the-box protections against troublesome scripts, the following configuration can be used:
-
-[source,yaml]
-scriptEngines: {
- gremlin-groovy: {
- imports: [java.lang.Math],
- staticImports: [java.lang.Math.PI],
- scripts: [scripts/empty-sample.groovy],
- config: {
- compilerCustomizerProviders: {
- "org.apache.tinkerpop.gremlin.groovy.jsr223.customizer.ThreadInterruptCustomizerProvider":[],
- "org.apache.tinkerpop.gremlin.groovy.jsr223.customizer.TimedInterruptCustomizerProvider":[10000],
- "org.apache.tinkerpop.gremlin.groovy.jsr223.customizer.CompileStaticCustomizerProvider":["org.apache.tinkerpop.gremlin.groovy.jsr223.customizer.SimpleSandboxExtension"]}}}}
-
-NOTE: The above configuration could also use the `TypeCheckedCustomizerProvider` in place of the
-`CompileStaticCustomizerProvider`. The differences between `TypeChecked` and `CompileStatic` are beyond the scope of
-this documentation. Consult the latest link:http://docs.groovy-lang.org/latest/html/documentation/#_typing[Groovy Documentation]
-for information on the differences. It is important to understand the impact that these configuration will have on
-submitted scripts before enabling this feature.
-
-This configuration uses the `SimpleSandboxExtension`, which blacklists calls to methods on the `System` class,
-thereby preventing someone from remotely killing the server:
-
-[source,groovy]
-----
-gremlin> :> System.exit(0)
-Script8.groovy: 1: [Static type checking] - Not authorized to call this method: java.lang.System#exit(int)
- @ line 1, column 1.
- System.exit(0)
- ^
-
-1 error
-----
-
-The `SimpleSandboxExtension` is by no means a "complete" implementation protecting against all manner of nefarious
-scripts, but it does provide an example for how such a capability might be implemented. A full implementation would
-likely represent domain specific white-listing of methods (and possibly variables) available for execution in the
-script engine.
-
-A final thought on the topic of `CompilerCustomizerProvider` implementations is that they are not just for
-"security" (though they are demonstrated in that capacity here). They can be used for a variety of features that
-can fine tune the Groovy compilation process. Read more about compilation customization in the
-link:http://docs.groovy-lang.org/latest/html/documentation/#compilation-customizers[Groovy Documentation].
-
-NOTE: The import of classes to the script engine is handled by the `ImportCustomizerProvider`. As the concept of
-"imports" is a first-class citizen (i.e. has its own configuration options), it is not recommended that the
-`ImportCustomizerProvider` be used as a configuration option to `compilerCustomizerProviders`.
-
-Serialization
-^^^^^^^^^^^^^
-
-Gremlin Server can accept requests and return results using different serialization formats. The format of the
-serialization is configured by the `serializers` setting described in the table above. Note that some serializers
-have additional configuration options as defined by the `serializers[X].config` setting. The `config` setting is a
-`Map` where the keys and values get passed to the serializer at its initialization. The available and/or expected
- keys are dependent on the serializer being used. Gremlin Server comes packaged with two different serializers:
- GraphSON and Gryo.
-
-GraphSON
-++++++++
-
-The GraphSON serializer produces human readable output in JSON format and is a good configuration choice for those
-trying to use TinkerPop from non-JVM languages. JSON obviously has wide support across virtually all major
-programming languages and can be consumed by a wide variety of tools.
-
-[source,yaml]
- - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0 }
-
-The above configuration represents the default serialization under the `application/json` MIME type and produces JSON
-consistent with standard JSON data types. It has the following configuration option:
-
-[width="100%",cols="3,10,^2",options="header"]
-|=========================================================
-|Key |Description |Default
-|useMapperFromGraph |Specifies the name of the `Graph` (from the `graphs` `Map` in the configuration file) from which to plugin any custom serializers that are tied to it. |_none_
-|=========================================================
-
-[source,yaml]
- - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0 }
-
-When the standard JSON data types are not enough (e.g. need to identify the difference between `double` and `float`
-data types), the above configuration will embed types into the JSON itself. The type embedding uses standard Java
-type names, so interpretation from non-JVM languages will be required. It has the MIME type of
-`application/vnd.gremlin-v1.0+json` and the following configuration options:
-
-[width="100%",cols="3,10,^2",options="header"]
-|=========================================================
-|Key |Description |Default
-|useMapperFromGraph |Specifies the name of the `Graph` (from the `graphs` `Map` in the configuration file) from which to plugin any custom serializers that are tied to it. |_none_
-|=========================================================
-
-Gryo
-++++
-
-The Gryo serializer utilizes Kryo-based serialization which produces a binary output. This format is best consumed
-by JVM-based languages.
-
-[source,yaml]
- - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerGremlinV1d0 }
-
-It has the MIME type of `application/vnd.gremlin-v1.0+gryo` and the following configuration options:
-
-[width="100%",cols="3,10,^2",options="header"]
-|=========================================================
-|Key |Description |Default
-|bufferSize |The maximum size of the Kryo buffer for use on a single object being serialized. Increasing this value will correct `KryoException` errors that complain of "Buffer too small". |_4096_
-|serializeResultToString |When set to `true`, results are serialized by first calling `toString()` on each object in the result list resulting in an extended MIME Type of `application/vnd.gremlin-v1.0+gryo-stringd`. When set to `false` Kryo-based serialization is applied. |_false_
-|useMapperFromGraph |Specifies the name of the `Graph` (from the `graphs` `Map` in the configuration file) from which to plugin any custom serializers that are tied to it. |_none_
-|ioRegistries |A list of `IoRegistry` implementations to be applied to the serializer.
-|custom |A list of classes with custom kryo `Serializer` implementations related to them in the form of `<class>;<serializer-class>`. |_none_
-|=========================================================
-
-As described above, there are multiple ways in which to register serializers for Kryo-based serialization. These
-configurations can be used in conjunction with one another where there is a specific ordering to how the configurations
-are applied. The `userMapperFromGraph` setting is applied first, followed by any `ioRegistries` and finalized by the
-`custom` setting.
-
-Best Practices
-~~~~~~~~~~~~~~
-
-The following sections define best practices for working with Gremlin Server.
-
-Tuning
-^^^^^^
-
-image:gremlin-handdrawn.png[width=120,float=right] Tuning Gremlin Server for a particular environment may require some simple trial-and-error, but the following represent some basic guidelines that might be useful:
-
-* When configuring the size of `threadPoolWorker` start with the default of `1` and increment by one as needed to a maximum of `2*number of cores`.
-* The "right" size of the `gremlinPool` setting is somewhat dependent on the type of scripts that will be processed
-by Gremlin Server. As requests arrive to Gremlin Server they are decoded and queued to be processed by threads in
-this pool. When this pool is exhausted of threads, Gremlin Server will continue to accept incoming requests, but
-the queue will continue to grow. If left to grow too large, the server will begin to slow. When tuning around
-this setting, consider whether the bulk of the scripts being processed will be "fast" or "slow", where "fast"
-generally means being measured in the low hundreds of milliseconds and "slow" means anything longer than that.
-** If the bulk of the scripts being processed are expected to be "fast", then a good starting point for this setting is `2*threadPoolWorker`.
-** If the bulk of the scripts being processed are expected to be "slow", then a good starting point for this setting is `4*threadPoolWorker`.
-* Scripts that are "slow" can really hurt Gremlin Server if they are not properly accounted for. `ScriptEngine`
-evaluations are blocking operations that aren't easily interrupted, so once a "slow" script is being evaluated in
-the context of a `ScriptEngine` it must finish its work. Lots of "slow" scripts will eventually consume the
-`gremlinPool` preventing other scripts from getting processed from the queue.
-** To limit the impact of this problem consider properly setting the `scriptEvaluationTimeout` and the `serializedResponseTimeout` to something "sane".
-** Test the traversals being sent to Gremlin Server and determine the maximum time they take to evaluate and iterate
-over results, then set these configurations accordingly.
-** Note that `scriptEvaluationTimeout` does not interrupt the evaluation on timeout. It merely allows Gremlin Server
-to "ignore" the result of that evaluation, which means the thread in the `gremlinPool` will still be consumed after
-the timeout.
-** The `serializedResponseTimeout` will kill the result iteration process and prevent additional processing. In most
-situations, the iteration and serialization process is the more costly step in this process as an errant script that
-returns a million or more results could send Gremlin Server into a long streaming cycle. Script evaluation on the
-other hand is usually very fast, occurring on the order of milliseconds, but that is entirely dependent on the
-contents of the script itself.
-
-[[parameterized-scripts]]
-Parameterized Scripts
-^^^^^^^^^^^^^^^^^^^^^
-
-image:gremlin-parameterized.png[width=150,float=left] Use script parameterization. Period. Gremlin Server caches
-all scripts that are passed to it. The cache is keyed based on the a hash of the script. Therefore `g.V(1)` and
-`g.V(2)` will be recognized as two separate scripts in the cache. If that script is parameterized to `g.V(x)`
-where `x` is passed as a parameter from the client, there will be no additional compilation cost for future requests
-on that script. Compilation of a script should be considered "expensive" and avoided when possible.
-
-Cache Management
-^^^^^^^^^^^^^^^^
-
-If Gremlin Server processes a large number of unique scripts, the cache will grow beyond the memory available to
-Gremlin Server and an `OutOfMemoryError` will loom. Script parameterization goes a long way to solving this problem
-and running out of memory should not be an issue for those cases. If it is a problem or if there is no script
-parameterization due to a given use case (perhaps using with use of <<sessions,sessions>>), it is possible to better
-control the nature of the script cache from the client side, by issuing scripts with a parameter to help define how
-the garbage collector should treat the references.
-
-The parameter is called `#jsr223.groovy.engine.keep.globals` and has four options:
-
-* `hard` - available in the cache for the life of the JVM (default when not specified).
-* `soft` - retained until memory is "low" and should be reclaimed before an `OutOfMemoryError` is thrown.
-* `weak` - garbage collected even when memory is abundant.
-* `phantom` - removed immediately after being evaluated by the `ScriptEngine`.
-
-By specifying an option other than `hard`, an `OutOfMemoryError` in Gremlin Server should be avoided. Of course,
-this approach will come with the downside that compiled scripts could be garbage collected and thus removed from the
-cache, forcing Gremlin Server to recompile later.
-
-[[sessions]]
-Considering Sessions
-^^^^^^^^^^^^^^^^^^^^
-
-The preferred approach for issuing requests to Gremlin Server is to do so in a sessionless manner. The concept of
-"sessionless" refers to a request that is completely encapsulated within a single transaction, such that the script
-in the request starts with a new transaction and ends with closed transaction. Sessionless requests have automatic
-transaction management handled by Gremlin Server, thus automatically opening and closing transactions as previously
-described. The downside to the sessionless approach is that the entire script to be executed must be known at the
-time of submission so that it can all be executed at once. This requirement makes it difficult for some use cases
-where more control over the transaction is desired.
-
-For such use cases, Gremlin Server supports sessions. With sessions, the user is in complete control of the start
-and end of the transaction. This feature comes with some additional expense to consider:
-
-* Initialization scripts will be executed for each session created so any expense related to them will be established
-each time a session is constructed.
-* There will be one script cache per session, which obviously increases memory requirements. The cache is not shared,
-so as to ensure that a session has isolation from other session environments. As a result, if the same script is
-executed in each session the same compilation cost will be paid for each session it is executed in.
-* Each session will require its own thread pool with a single thread in it - this ensures that transactional
-boundaries are managed properly from one request to the next.
-* If there are multiple Gremlin Server instances, communication from the client to the server must be bound to the
-server that the session was initialized in. Gremlin Server does not share session state as the transactional context
-of a `Graph` is bound to the thread it was initialized in.
-
-A session is a "heavier" approach to the simple "request/response" approach of sessionless requests, but is sometimes
-necessary for a given use case.
-
-Developing a Driver
-~~~~~~~~~~~~~~~~~~~
-
-image::gremlin-server-protocol.png[width=325]
-
-One of the roles for Gremlin Server is to provide a bridge from TinkerPop to non-JVM languages (e.g. Go, Python,
-etc.). Developers can build language bindings (or driver) that provide a way to submit Gremlin scripts to Gremlin
-Server and get back results. Given the exstensible nature of Gremlin Server, it is difficult to provide an
-authoritative guide to developing a driver. It is however possible to describe the core communication protocal
-using the standard out-of-the-box configuration which should provide enough information to develop a driver for a
-specific language.
-
-image::gremlin-server-flow.png[width=300,float=right]
-
-Gremlin Server is distributed with a configuration that utilizes link:http://en.wikipedia.org/wiki/WebSocket[WebSockets]
-with a custom sub-protocol. Under this configuration, Gremlin Server accepts requests containing a Gremlin script,
-evaluates that script and then streams back the results. The notion of "streaming" is depicted in the diagram to the right.
-
-The diagram shows an incoming request to process the Gremlin script of `g.V`. Gremlin Server evaluates that script,
-getting an `Iterator` of vertices as a result, and steps through each `Vertex` within it. The vertices are batched
-together given the `resultIterationBatchSize` configuration. In this case, that value must be `2` given that each
-"response" contains two vertices. Each response is serialized given the requested serializer type (JSON is likely
-best for non-JVM languages) and written back to the requesting client immediately. Gremlin Server does not wait for
-the entire result to be iterated, before sending back a response. It will send the responses as they are realized.
-
-This approach allows for the processing of large result sets without having to serialize the entire result into memory
-for the response. It places a bit of a burden on the developer of the driver however, because it becomes necessary to
-provide a way to reconstruct the entire result on the client side from all of the individual responses that Gremlin
-Server returns for a single request. Again, this description of Gremlin Server's "flow" is related to the
-out-of-the-box configuration. It is quite possible to construct other flows, that might be more amenable to a
-particular language or style of processing.
-
-To formulate a request to Gremlin Server, a `RequestMessage` needs to be constructed. The `RequestMessage` is a
-generalized representation of a request that carries a set of "standard" values in addition to optional ones that are
-dependent on the operation being performed. A `RequestMessage` has these fields:
-
-[width="100%",cols="3,10",options="header"]
-|=========================================================
-|Key |Description
-|requestId |A link:http://en.wikipedia.org/wiki/Globally_unique_identifier[UUID] representing the unique identification for the request.
-|op |The name of the "operation" to execute based on the available `OpProcessor` configured in the Gremlin Server. To evaluate a script, use `eval`.
-|processor |The name of the `OpProcessor` to utilize. The default `OpProcessor` for evaluating scripts is unamed and therefore script evaluation purposes, this value can be an empty string.
-|args |A `Map` of arbitrary parameters to pass to Gremlin Server. The requirements for the contents of this `Map` are dependent on the `op` selected.
-|=========================================================
-
-This message can be serialized in any fashion that is supported by Gremlin Server. New serialization methods can
-be plugged in by implementing a `ServiceLoader` enabled `MessageSerializer`, however Gremlin Server provides for
-JSON serialization by default which will be good enough for purposes of most developers building drivers.
-A `RequestMessage` to evaluate a script with variable bindings looks like this in JSON:
-
-[source,js]
-----
-{ "requestId":"1d6d02bd-8e56-421d-9438-3bd6d0079ff1",
- "op":"eval",
- "processor":"",
- "args":{"gremlin":"g.traversal().V(x).out()",
- "bindings":{"x":1},
- "language":"gremlin-groovy"}}
-----
-
-The above JSON represents the "body" of the request to send to Gremlin Server. When sending this "body" over
-websockets Gremlin Server can accept a packet frame using a "text" (1) or a "binary" (2) opcode. Using "text"
-is a bit more limited in that Gremlin Server will always process the body of that request as JSON. Generally speaking
-"text" is just for testing purposes.
-
-The preferred method for sending requests to Gremlin Server is to use the "binary" opcode. In this case, a "header"
-will need be sent in addition to to the "body". The "header" basically consists of a "mime type" so that Gremlin
-Server knows how to deserialize the `RequestMessage`. So, the actual byte array sent to Gremlin Server would be
-formatted as follows:
-
-image::gremlin-server-request.png[]
-
-The first byte represents the length of the "mime type" string value that follows. Given the default configuration of
-Gremlin Server, this value should be set to `application/json`. The "payload" represents the JSON message above
-encoded as bytes.
-
-NOTE: Gremlin Server will only accept masked packets as it pertains to websocket packet header construction.
-
-When Gremlin Server receives that request, it will decode it given the "mime type", pass it to the requested
-`OpProcessor` which will execute the `op` defined in the message. In this case, it will evaluate the script
-`g.traversal().V(x).out()` using the `bindings` supplied in the `args` and stream back the results in a series of
-`ResponseMessages`. A `ResponseMessage` looks like this:
-
-[width="100%",cols="3,10",options="header"]
-|=========================================================
-|Key |Description
-|requestId |The identifier of the `RequestMessage` that generated this `ResponseMessage`.
-|status | The `status` contains a `Map` of three keys: `code` which refers to a `ResultCode` that is somewhat analogous to an link:http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html[HTTP status code], `attributes` that represent a `Map` of protocol-level information, and `message` which is just a human-readable `String` usually associated with errors.
-|result | The `result` contains a `Map` of two keys: `data` which refers to the actual data returned from the server (the type of data is determined by the operation requested) and `meta` which is a `Map` of meta-data related to the response.
-|=========================================================
-
-In this case the `ResponseMessage` returned to the client would look something like this:
-
-[source,js]
-----
-{"result":{"data":[{"id": 2,"label": "person","type": "vertex","properties": [
- {"id": 2, "value": "vadas", "label": "name"},
- {"id": 3, "value": 27, "label": "age"}]},
- ], "meta":{}},
- "requestId":"1d6d02bd-8e56-421d-9438-3bd6d0079ff1",
- "status":{"code":206,"attributes":{},"message":""}}
-----
-
-Gremlin Server is capable of streaming results such that additional responses will arrive over the websocket until
-the iteration of the result on the server is complete. Each successful incremental message will have a `ResultCode`
-of `206`. Termination of the stream will be marked by a final `200` status code. Note that all messages without a
-`206` represent terminating conditions for a request. The following table details the various status codes that
-Gremlin Server will send:
-
-[width="100%",cols="2,2,9",options="header"]
-|=========================================================
-|Code |Name |Description
-|200 |SUCCESS |The server successfully processed a request to completion - there are no messages remaining in this stream.
-|204 |NO CONTENT |The server processed the request but there is no result to return (e.g. an `Iterator` with no elements).
-|206 |PARTIAL CONTENT |The server successfully returned some content, but there is more in the stream to arrive - wait for a `SUCCESS` to signify the end of the stream.
-|401 |UNAUTHORIZED |The request attempted to access resources that the requesting user did not have access to.
-|407 |AUTHENTICATE |A challenge from the server for the client to authenticate its request.
-|498 |MALFORMED REQUEST | The request message was not properly formatted which means it could not be parsed at all or the "op" code was not recognized such that Gremlin Server could properly route it for processing. Check the message format and retry the request.
-|499 |INVALID REQUEST ARGUMENTS |The request message was parseable, but the arguments supplied in the message were in conflict or incomplete. Check the message format and retry the request.
-|500 |SERVER ERROR |A general server error occurred that prevented the request from being processed.
-|597 |SCRIPT EVALUATION ERROR |The script submitted for processing evaluated in the `ScriptEngine` with errors and could not be processed. Check the script submitted for syntax errors or other problems and then resubmit.
-|598 |SERVER TIMEOUT |The server exceeded one of the timeout settings for the request and could therefore only partially responded or did not respond at all.
-|599 |SERVER SERIALIZATION ERROR |The server was not capable of serializing an object that was returned from the script supplied on the request. Either transform the object into something Gremlin Server can process within the script or install mapper serialization classes to Gremlin Server.
-|=========================================================
-
-`SUCCESS` and `NO CONTENT` messages are terminating messages that indicate that a request was properly handled on the
-server and that there are no additional messages streaming in for that request. When developing a driver, it is
-important to note the slight differences in semantics for these result codes when it comes to sessionless versus
-in-session requests. For a sessionless request, which operates under automatic transaction management, Gremlin Server
-will only send one of these message types after result iteration and transaction `commit()`. In other words, the
-driver could potentially expect to receive a number of "successful" `PARTIAL CONTENT` messages before ultimately
-ending in failure on `commit()`. For in-session requests, the client is responsible for managing the transaction
-and therefore, a first request could receive multiple "success" related messages, only to fail on a future request
-that finally issues the `commit()`.
-
-OpProcessors Arguments
-^^^^^^^^^^^^^^^^^^^^^^
-
-The following sections define a non-exhaustive list of available operations and arguments for embedded `OpProcessors`
-(i.e. ones packaged with Gremlin Server).
-
-Common
-++++++
-
-All `OpProcessor` instances support these arguments.
-
-[width="100%",cols="2,2,9",options="header"]
-|=========================================================
-|Key |Type |Description
-|batchSize |Int |When the result is an iterator this value defines the number of iterations each `ResponseMessage` should contain - overrides the `resultIterationBatchSize` server setting.
-|=========================================================
-
-Standard OpProcessor
-++++++++++++++++++++
-
-The "standard" `OpProcessor` handles requests for the primary function of Gremlin Server - executing Gremlin.
-Requests made to this `OpProcessor` are "sessionless" in the sense that a request must encapsulate the entirety
-of a transaction. There is no state maintained between requests. A transaction is started when the script is first
-evaluated and is committed when the script completes (or rolled back if an error occurred).
-
-[width="100%",cols="3,10a",options="header"]
-|=========================================================
-|Key |Description
-|processor |As this is the default `OpProcessor` this value can be set to an empty string
-|op |[width="100%",cols="3,10",options="header"]
-!=========================================================
-!Key !Description
-!`authentication` !A request that contains the response to a server challenge for authentication.
-!`eval` !Evaluate a Gremlin script provided as a `String`
-!=========================================================
-|=========================================================
-
-'`authentication` operation arguments'
-[width="100%",cols="2,2,9",options="header"]
-|=========================================================
-|Key |Type |Description
-|sasl |byte[] | *Required* The response to the server authentication challenge. This value is dependent on the SASL authentication mechanism required by the server.
-|=========================================================
-
-'`eval` operation arguments'
-[width="100%",cols="2,2,9",options="header"]
-|=========================================================
-|Key |Type |Description
-|gremlin |String | *Required* The Gremlin script to evaluate
-|bindings |Map |A map of key/value pairs to apply as variables in the context of the Gremlin script
-|language |String |The flavor used (e.g. `gremlin-groovy`)
-|aliases |Map |A map of key/value pairs that allow globally bound `Graph` and `TraversalSource` objects to
-be aliased to different variable names for purposes of the current request. The value represents the name the
-global variable and its key represents the new binding name as it will be referenced in the Gremlin query. For
-example, if the Gremlin Server defines two `TraversalSource` instances named `g1` and `g2`, it would be possible
-to send an alias pair with key of "g" and value of "g2" and thus allow the script to refer to "g2" simply as "g".
-|=========================================================
-
-Session OpProcessor
-+++++++++++++++++++
-
-The "session" `OpProcessor` handles requests for the primary function of Gremlin Server - executing Gremlin. It is
-like the "standard" `OpProcessor`, but instead maintains state between sessions and leaves all transaction management
-up to the calling client. It is important that clients that open sessions, commit or roll them back, however Gremlin
-Server will try to clean up such things when a session is killed that has been abandoned. It is important to consider
-that a session can only be maintained with a single machine. In the event that multiple Gremlin Server are deployed,
-session state is not shared among them.
-
-[width="100%",cols="3,10a",options="header"]
-|=========================================================
-|Key |Description
-|processor |This value should be set to `session`
-|op |
-[cols="3,10",options="header"]
-!=========================================================
-!Key !Description
-!`authentication` !A request that contains the response to a server challenge for authentication
-!`eval` !Evaluate a Gremlin script provided as a `String`
-!`close` !Close the specified session and rollback any open transactions.
-|=========================================================
-
-'`authentication` operation arguments'
-[width="100%",cols="2,2,9",options="header"]
-|=========================================================
-|Key |Type |Description
-|sasl |byte[] | *Required* The response to the server authentication challenge. This value is dependent on the SASL authentication mechanism required by the server.
-|=========================================================
-
-'`eval` operation arguments'
-[width="100%",options="header"]
-|=========================================================
-|Key |Type |Description
-|gremlin |String | *Required* The Gremlin script to evaluate
-|session |String | *Required* The session identifier for the current session - typically this value should be a UUID (the session will be created if it doesn't exist)
-|bindings |Map |A map of key/value pairs to apply as variables in the context of the Gremlin script
-|language |String |The flavor used (e.g. `gremlin-groovy`)
-|=========================================================
-
-'`close` operation arguments'
-[width="100%",cols="2,2,9",options="header"]
-|=========================================================
-|Key |Type |Description
-|session |String | *Required* The session identifier for the session to close.
-|=========================================================
-
-Authentication
-^^^^^^^^^^^^^^
-
-Gremlin Server supports link:https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer[SASL-based]
-authentication. A SASL implementation provides a series of challenges and responses that a driver must comply with
-in order to authenticate. By default, Gremlin Server only supports the "PLAIN" SASL mechanism, which is a cleartext
-password system. When authentication is enabled, an incoming request is intercepted before it is evaluated by the
-`ScriptEngine`. The request is saved on the server and a `AUTHENTICATE` challenge response (status code `407`) is
-returned to the client.
-
-The client will detect the `AUTHENTICATE` and respond with an `authentication` for the `op` and an `arg` named `sasl`
-that contains the password. The password should be either, an encoded sequence of UTF-8 bytes, delimited by 0
-(US-ASCII NUL), where the form is : `<NUL>username<NUL>password`, or a Base64 encoded string of the former (which
-in this instance would be `AHVzZXJuYW1lAHBhc3N3b3Jk`). Should Gremlin Server be able to authenticate with the
-provided credentials, the server will return the results of the original request as it normally does without
-authentication. If it cannot authenticate given the challenge response from the client, it will return `UNAUTHORIZED`
-(status code `401`).
-
-NOTE: Gremlin Server does not support the "authorization identity" as described in link:https://tools.ietf.org/html/rfc4616[RFC4616].
-
-[[gremlin-plugins]]
-Gremlin Plugins
----------------
-
-image:gremlin-plugin.png[width=125]
-
-Plugins provide a way to expand the features of Gremlin Console and Gremlin Server. The first step to developing a
-plugin is to implement the `GremlinPlugin` interface:
-
-[source,java]
-----
-include::{basedir}/gremlin-groovy/src/main/java/org/apache/tinkerpop/gremlin/groovy/plugin/GremlinPlugin.java[]
-----
-
-The most simple plugin and the one most commonly implemented will likely be one that just provides a list of classes
-to import to the Gremlin Console. This type of plugin is the easiest way for implementers of the TinkerPop Structure
-and Process APIs to make their implementations available to users. The TinkerGraph implementation has just such a plugin:
-
-[source,java]
-----
-include::{basedir}/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/groovy/plugin/TinkerGraphGremlinPlugin.java[]
-----
-
-Note that the plugin provides a unique name for the plugin which follows a namespaced pattern as _namespace_._plugin-name_
-(e.g. "tinkerpop.hadoop" - "tinkerpop" is the reserved namespace for TinkerPop maintained plugins). To make TinkerGraph
-classes available to the Console, the `PluginAcceptor` is given a `Set` of imports to provide to the plugin host. The
-`PluginAcceptor` essentially behaves as an abstraction to the "host" that is handling the `GremlinPlugin`. `GremlinPlugin`
-implementations maybe hosted by the Console as well as the `ScriptEngine` in Gremlin Server. Obviously, registering
-new commands and other operations that are specific to the Groovy Shell don't make sense there. Write the code for
-the plugin defensively by checking the `GremlinPlugin.env` key in the `PluginAcceptor.environment()` to understand
-which environment the plugin is being used in.
-
-There is one other step to follow to ensure that the `GremlinPlugin` is visible to its hosts. `GremlinPlugin`
-implementations are loaded via link:http://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html[ServiceLoader]
-and therefore need a resource file added to the jar file where the plugin exists. Add a file called
-`org.apache.tinkerpop.gremlin.groovy.plugin.GremlinPlugin` to `META-INF.services`. In the case of the TinkerGraph
-plugin above, that file will have this line in it:
-
-[source,java]
-----
-include::
<TRUNCATED>
[20/22] incubator-tinkerpop git commit: minor fix in the AsciiDocs
post-processor
Posted by sp...@apache.org.
minor fix in the AsciiDocs post-processor
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/ead508d1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/ead508d1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/ead508d1
Branch: refs/heads/master
Commit: ead508d15539f17140efbca1a823eaa1dd5e94e6
Parents: cc0c2ba
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Fri Nov 20 22:12:55 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Fri Nov 20 22:12:55 2015 +0100
----------------------------------------------------------------------
docs/postprocessor/postprocess.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/ead508d1/docs/postprocessor/postprocess.sh
----------------------------------------------------------------------
diff --git a/docs/postprocessor/postprocess.sh b/docs/postprocessor/postprocess.sh
index cef9b93..8d1fd92 100755
--- a/docs/postprocessor/postprocess.sh
+++ b/docs/postprocessor/postprocess.sh
@@ -26,7 +26,7 @@ if [ -d "target/docs" ]; then
find target/docs -name index.html | while read file ; do
awk -f "docs/postprocessor/processor.awk" "${file}" \
| perl -0777 -pe 's/<span class="comment">\/\*\n \*\/<\/span>//igs' \
- | sed "s/x\.y\.z/${TP_VERSION}/" \
+ | sed "s/x\.y\.z/${TP_VERSION}/g" \
> "${file}.tmp" && mv "${file}.tmp" "${file}"
done
fi
[04/22] incubator-tinkerpop git commit: Made subdirectories for
various "books" in the docs.
Posted by sp...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/the-traversal.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/the-traversal.asciidoc b/docs/src/the-traversal.asciidoc
deleted file mode 100644
index 0b82823..0000000
--- a/docs/src/the-traversal.asciidoc
+++ /dev/null
@@ -1,2378 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-[[traversal]]
-The Traversal
-=============
-
-image::gremlin-running.png[width=125]
-
-At the most general level there is `Traversal<S,E>` which implements `Iterator<E>`, where the `S` stands for start and
-the `E` stands for end. A traversal is composed of four primary components:
-
- . `Step<S,E>`: an individual function applied to `S` to yield `E`. Steps are chained within a traversal.
- . `TraversalStrategy`: interceptor methods to alter the execution of the traversal (e.g. query re-writing).
- . `TraversalSideEffects`: key/value pairs that can be used to store global information about the traversal.
- . `Traverser<T>`: the object propagating through the `Traversal` currently representing an object of type `T`.
-
-The classic notion of a graph traversal is provided by `GraphTraversal<S,E>` which extends `Traversal<S,E>`.
-`GraphTraversal` provides an interpretation of the graph data in terms of vertices, edges, etc. and thus, a graph
-traversal link:http://en.wikipedia.org/wiki/Domain-specific_language[DSL].
-
-IMPORTANT: The underlying `Step` implementations provided by TinkerPop should encompass most of the functionality
-required by a DSL author. It is important that DSL authors leverage the provided steps as then the common optimization
-and decoration strategies can reason on the underlying traversal sequence. If new steps are introduced, then common
-traversal strategies may not function properly.
-
-[[graph-traversal-steps]]
-Graph Traversal Steps
----------------------
-
-image::step-types.png[width=650]
-
-A `GraphTraversal<S,E>` is spawned from a `GraphTraversalSource`. It can also be spawned anonymously (i.e. empty)
-via `__`. A graph traversal is composed of an ordered list of steps. All the steps provided by `GraphTraversal`
-inherit from the more general forms diagrammed above. A list of all the steps (and their descriptions) are provided
-in the TinkerPop3 link:http://tinkerpop.incubator.apache.org/javadocs/3.0.2-incubating/core/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversal.html[GraphTraversal JavaDoc].
-The following subsections will demonstrate the GraphTraversal steps using the <<gremlin-console,Gremlin Console>>.
-
-NOTE: To reduce the verbosity of the expression, it is good to
-`import static org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__.*`. This way, instead of doing `__.inE()`
-for an anonymous traversal, it is possible to simply write `inE()`. Be aware of language-specific reserved keywords
-when using anonymous traversals. For example, `in` and `as` are reserved keywords in Groovy, therefore you must use
-the verbose syntax `__.in()` and `__.as()` to avoid collisions.
-
-[[lambda-steps]]
-Lambda Steps
-~~~~~~~~~~~~
-
-CAUTION: Lambda steps are presented for educational purposes as they represent the foundational constructs of the
-Gremlin language. In practice, lambda steps should be avoided and traversal verification strategies exist to disallow t
-heir use unless explicitly "turned off." For more information on the problems with lambdas, please read
-<<a-note-on-lambdas,A Note on Lambdas>>.
-
-There are four generic steps by which all other specific steps described later extend.
-
-[width="100%",cols="10,12",options="header"]
-|=========================================================
-| Step| Description
-| `map(Function<Traverser<S>, E>)` | map the traverser to some object of type `E` for the next step to process.
-| `flatMap(Function<Traverser<S>, Iterator<E>>)` | map the traverser to an iterator of `E` objects that are streamed to the next step.
-| `filter(Predicate<Traverser<S>>)` | map the traverser to either true or false, where false will not pass the traverser to the next step.
-| `sideEffect(Consumer<Traverser<S>>)` | perform some operation on the traverser and pass it to the next step.
-| `branch(Function<Traverser<S>,M>)` | split the traverser to all the traversals indexed by the `M` token.
-|=========================================================
-
-The `Traverser<S>` object provides access to:
-
- . The current traversed `S` object -- `Traverser.get()`.
- . The current path traversed by the traverser -- `Traverser.path()`.
- .. A helper shorthand to get a particular path-history object -- `Traverser.path(String) == Traverser.path().get(String)`.
- . The number of times the traverser has gone through the current loop -- `Traverser.loops()`.
- . The number of objects represented by this traverser -- `Traverser.bulk()`.
- . The local data structure associated with this traverser -- `Traverser.sack()`.
- . The side-effects associated with the traversal -- `Traverser.sideEffects()`.
- .. A helper shorthand to get a particular side-effect -- `Traverser.sideEffect(String) == Traverser.sideEffects().get(String)`.
-
-image:map-lambda.png[width=150,float=right]
-[gremlin-groovy,modern]
-----
-g.V(1).out().values('name') <1>
-g.V(1).out().map {it.get().value('name')} <2>
-----
-
-<1> An outgoing traversal from vertex 1 to the name values of the adjacent vertices.
-<2> The same operation, but using a lambda to access the name property values.
-
-image:filter-lambda.png[width=160,float=right]
-[gremlin-groovy,modern]
-----
-g.V().filter {it.get().label() == 'person'} <1>
-g.V().hasLabel('person') <2>
-----
-
-<1> A filter that only allows the vertex to pass if it has an age-property.
-<2> The more specific `has()`-step is implemented as a `filter()` with respective predicate.
-
-
-image:side-effect-lambda.png[width=175,float=right]
-[gremlin-groovy,modern]
-----
-g.V().hasLabel('person').sideEffect(System.out.&println) <1>
-----
-
-<1> Whatever enters `sideEffect()` is passed to the next step, but some intervening process can occur.
-
-image:branch-lambda.png[width=180,float=right]
-[gremlin-groovy,modern]
-----
-g.V().branch(values('name')).
- option('marko', values('age')).
- option(none, values('name')) <1>
-g.V().choose(has('name','marko'),
- values('age'),
- values('name')) <2>
-----
-
-<1> If the vertex is "marko", get his age, else get the name of the vertex.
-<2> The more specific boolean-based `choose()`-step is implemented as a `branch()`.
-
-[[addedge-step]]
-AddEdge Step
-~~~~~~~~~~~~
-
-link:http://en.wikipedia.org/wiki/Automated_reasoning[Reasoning] is the process of making explicit what is implicit
-in the data. What is explicit in a graph are the objects of the graph -- i.e. vertices and edges. What is implicit
-in the graph is the traversal. In other words, traversals expose meaning where the meaning is determined by the
-traversal definition. For example, take the concept of a "co-developer." Two people are co-developers if they have
-worked on the same project together. This concept can be represented as a traversal and thus, the concept of
-"co-developers" can be derived. Moreover, what was once implicit can be made explicit via the `addE()`-step
-(*map*/*sideEffect*).
-
-image::addedge-step.png[width=450]
-
-[gremlin-groovy,modern]
-----
-g.V(1).as('a').out('created').in('created').where(neq('a')).
- addE('co-developer').from('a').property('year',2009) <1>
-g.V(3,4,5).aggregate('x').has('name','josh').as('a').
- select('x').unfold().hasLabel('software').addE('createdBy').to('a') <2>
-g.V().as('a').out('created').addE('createdBy').to('a').property('acl','public') <3>
-g.V(1).as('a').out('knows').
- addE('livesNear').from('a').property('year',2009).
- inV().inE('livesNear').values('year') <4>
-g.V().match(
- __.as('a').out('knows').as('b'),
- __.as('a').out('created').as('c'),
- __.as('b').out('created').as('c')).
- addE('friendlyCollaborator').from('a').to('b').
- property(id,13).property('project',select('c').values('name')) <5>
-g.E(13).valueMap()
-----
-
-<1> Add a co-developer edge with a year-property between marko and his collaborators.
-<2> Add incoming createdBy edges from the josh-vertex to the lop- and ripple-vertices.
-<3> Add an inverse createdBy edge for all created edges.
-<4> The newly created edge is a traversable object.
-<5> Two arbitrary bindings in a traversal can be joined `from()`->`to()`, where `id` can be provided for graphs that
-supports user provided ids.
-
-[[addvertex-step]]
-AddVertex Step
-~~~~~~~~~~~~~~
-
-The `addV()`-step is used to add vertices to the graph (*map*/*sideEffect*). For every incoming object, a vertex is
-created. Moreover, `GraphTraversalSource` maintains an `addV()` method.
-
-[gremlin-groovy,modern]
-----
-g.addV('person').property('name','stephen')
-g.V().values('name')
-g.V().outE('knows').addV().property('name','nothing')
-g.V().has('name','nothing')
-g.V().has('name','nothing').bothE()
-----
-
-[[addproperty-step]]
-AddProperty Step
-~~~~~~~~~~~~~~~~
-
-The `property()`-step is used to add properties to the elements of the graph (*sideEffect*). Unlike `addV()` and
-`addE()`, `property()` is a full sideEffect step in that it does not return the property it created, but the element
-that streamed into it. Moreover, if `property()` follows an `addV()` or `addE()`, then it is "folded" into the
-previous step to enable vertex and edge creation with all its properties in one creation operation.
-
-[gremlin-groovy,modern]
-----
-g.V(1).property('country','usa')
-g.V(1).property('city','santa fe').property('state','new mexico').valueMap()
-g.V(1).property(list,'age',35) <1>
-g.V(1).valueMap()
-g.V(1).property('friendWeight',outE('knows').values('weight').sum(),'acl','private') <2>
-g.V(1).properties('friendWeight').valueMap() <3>
-----
-
-<1> For vertices, a cardinality can be provided for <<vertex properties,vertex-properties>>.
-<2> It is possible to select the property value (as well as key) via a traversal.
-<3> For vertices, the `property()`-step can add meta-properties.
-
-
-[[aggregate-step]]
-Aggregate Step
-~~~~~~~~~~~~~~
-
-image::aggregate-step.png[width=800]
-
-The `aggregate()`-step (*sideEffect*) is used to aggregate all the objects at a particular point of traversal into a
-`Collection`. The step uses link:http://en.wikipedia.org/wiki/Eager_evaluation[eager evaluation] in that no objects
-continue on until all previous objects have been fully aggregated (as opposed to <<store-step,`store()`>> which
-link:http://en.wikipedia.org/wiki/Lazy_evaluation[lazily] fills a collection). The eager evaluation nature is crucial
-in situations where everything at a particular point is required for future computation. An example is provided below.
-
-[gremlin-groovy,modern]
-----
-g.V(1).out('created') <1>
-g.V(1).out('created').aggregate('x') <2>
-g.V(1).out('created').aggregate('x').in('created') <3>
-g.V(1).out('created').aggregate('x').in('created').out('created') <4>
-g.V(1).out('created').aggregate('x').in('created').out('created').
- where(without('x')).values('name') <5>
-----
-
-<1> What has marko created?
-<2> Aggregate all his creations.
-<3> Who are marko's collaborators?
-<4> What have marko's collaborators created?
-<5> What have marko's collaborators created that he hasn't created?
-
-In link:http://en.wikipedia.org/wiki/Recommender_system[recommendation systems], the above pattern is used:
-
- "What has userA liked? Who else has liked those things? What have they liked that userA hasn't already liked?"
-
-Finally, `aggregate()`-step can be modulated via `by()`-projection.
-
-[gremlin-groovy,modern]
-----
-g.V().out('knows').aggregate('x').cap('x')
-g.V().out('knows').aggregate('x').by('name').cap('x')
-----
-
-[[and-step]]
-And Step
-~~~~~~~~
-
-The `and()`-step ensures that all provided traversals yield a result (*filter*). Please see <<or-step,`or()`>> for or-semantics.
-
-[gremlin-groovy,modern]
-----
-g.V().and(
- outE('knows'),
- values('age').is(lt(30))).
- values('name')
-----
-
-The `and()`-step can take an arbitrary number of traversals. All traversals must produce at least one output for the
-original traverser to pass to the next step.
-
-An link:http://en.wikipedia.org/wiki/Infix_notation[infix notation] can be used as well. Though, with infix notation,
-only two traversals can be and'd together.
-
-[gremlin-groovy,modern]
-----
-g.V().where(outE('created').and().outE('knows')).values('name')
-----
-
-[[as-step]]
-As Step
-~~~~~~~
-
-The `as()`-step is not a real step, but a "step modulator" similar to <<by-step,`by()`>> and <<option-step,`option()`>>.
-With `as()`, it is possible to provide a label to the step that can later be accessed by steps and data structures
-that make use of such labels -- e.g., <<select-step,`select()`>>, <<match-step,`match()`>>, and path.
-
-[gremlin-groovy,modern]
-----
-g.V().as('a').out('created').as('b').select('a','b') <1>
-g.V().as('a').out('created').as('b').select('a','b').by('name') <2>
-----
-
-<1> Select the objects labeled "a" and "b" from the path.
-<2> Select the objects labeled "a" and "b" from the path and, for each object, project its name value.
-
-A step can have any number of labels associated with it. This is useful for referencing the same step multiple times in a future step.
-
-[gremlin-groovy,modern]
-----
-g.V().hasLabel('software').as('a','b','c').
- select('a','b','c').
- by('name').
- by('lang').
- by(__.in('created').values('name').fold())
-----
-
-[[barrier-step]]
-Barrier Step
-~~~~~~~~~~~~
-
-The `barrier()`-step (*barrier*) turns the the lazy traversal pipeline into a bulk-synchronous pipeline. This step is
-useful in the following situations:
-
- * When everything prior to `barrier()` needs to be executed before moving onto the steps after the `barrier()` (i.e. ordering).
- * When "stalling" the traversal may lead to a "bulking optimization" in traversals that repeatedly touch many of the same elements (i.e. optimizing).
-
-[gremlin-groovy,modern]
-----
-g.V().sideEffect{println "first: ${it}"}.sideEffect{println "second: ${it}"}.iterate()
-g.V().sideEffect{println "first: ${it}"}.barrier().sideEffect{println "second: ${it}"}.iterate()
-----
-
-The theory behind a "bulking optimization" is simple. If there are one million traversers at vertex 1, then there is
-no need to calculate one million `both()`-computations. Instead, represent those one million traversers as a single
-traverser with a `Traverser.bulk()` equal to one million and execute `both()` once. A bulking optimization example is
-made more salient on a larger graph. Therefore, the example below leverages the <<grateful-dead,Grateful Dead graph>>.
-
-[gremlin-groovy]
-----
-graph = TinkerGraph.open()
-graph.io(graphml()).readGraph('data/grateful-dead.xml')
-g = graph.traversal(standard())
-clockWithResult(1){g.V().both().both().both().count().next()} <1>
-clockWithResult(1){g.V().repeat(both()).times(3).count().next()} <2>
-clockWithResult(1){g.V().both().barrier().both().barrier().both().barrier().count().next()} <3>
-----
-
-<1> A non-bulking traversal where each traverser is processed.
-<2> Each traverser entering `repeat()` has its recursion bulked.
-<3> A bulking traversal where implicit traversers are not processed.
-
-If `barrier()` is provided an integer argument, then the barrier will only hold `n`-number of unique traversers in its
-barrier before draining the aggregated traversers to the next step. This is useful in the aforementioned bulking
-optimization scenario, but reduces the risk of an out-of-memory exception.
-
-The non-default `LazyBarrierStrategy` inserts `barrier()`-steps in a traversal where appropriate in order to gain the
-"bulking optimization."
-
-[gremlin-groovy]
-----
-graph = TinkerGraph.open()
-graph.io(graphml()).readGraph('data/grateful-dead.xml')
-g = graph.traversal(GraphTraversalSource.build().with(LazyBarrierStrategy.instance()).engine(StandardTraversalEngine.build()))
-clockWithResult(1){g.V().both().both().both().count().next()}
-g.V().both().both().both().count().iterate().toString() <1>
-----
-
-<1> With `LazyBarrierStrategy` activated, `barrier()` steps are automatically inserted where appropriate.
-
-[[by-step]]
-By Step
-~~~~~~~
-
-The `by()`-step is not an actual step, but instead is a "step-modulator" similar to <<as-step,`as()`>> and
-<<option-step,`option()`>>. If a step is able to accept traversals, functions, comparators, etc. then `by()` is the
-means by which they are added. The general pattern is `step().by()...by()`. Some steps can only accept one `by()`
-while others can take an arbitrary amount.
-
-[gremlin-groovy,modern]
-----
-g.V().group().by(bothE().count()) <1>
-g.V().group().by(bothE().count()).by('name') <2>
-g.V().group().by(bothE().count()).by(count()) <3>
-----
-
-<1> `by(outE().count())` will group the elements by their edge count (*traversal*).
-<2> `by('name')` will process the grouped elements by their name (*element property projection*).
-<3> `by(count())` will count the number of elements in each group (*traversal*).
-
-[cap-step]]
-Cap Step
-~~~~~~~~
-
-The `cap()`-step (*barrier*) iterates the traversal up to itself and emits the sideEffect referenced by the provided
-key. If multiple keys are provided, then a `Map<String,Object>` of sideEffects is emitted.
-
-[gremlin-groovy,modern]
-----
-g.V().groupCount('a').by(label).cap('a') <1>
-g.V().groupCount('a').by(label).groupCount('b').by(outE().count()).cap('a','b') <2>
-----
-
-<1> Group and count verticies by their label. Emit the side effect labeled 'a', which is the group count by label.
-<2> Same as statement 1, but also emit the side effect labeled 'b' which groups vertices by the number of out edges.
-
-[[coalesce-step]]
-Coalesce Step
-~~~~~~~~~~~~~
-
-The `coalesce()`-step evaluates the provided traversals in order and returns the first traversal that emits at
-least one element.
-
-[gremlin-groovy,modern]
-----
-g.V(1).coalesce(outE('knows'), outE('created')).inV().path().by('name').by(label)
-g.V(1).coalesce(outE('created'), outE('knows')).inV().path().by('name').by(label)
-g.V(1).next().property('nickname', 'okram')
-g.V().hasLabel('person').coalesce(values('nickname'), values('name'))
-----
-
-[[count-step]]
-Count Step
-~~~~~~~~~~
-
-image::count-step.png[width=195]
-
-The `count()`-step (*map*) counts the total number of represented traversers in the streams (i.e. the bulk count).
-
-[gremlin-groovy,modern]
-----
-g.V().count()
-g.V().hasLabel('person').count()
-g.V().hasLabel('person').outE('created').count().path() <1>
-g.V().hasLabel('person').outE('created').count().map {it.get() * 10}.path() <2>
-----
-
-<1> `count()`-step is a <<a-note-on-barrier-steps,reducing barrier step>> meaning that all of the previous traversers are folded into a new traverser.
-<2> The path of the traverser emanating from `count()` starts at `count()`.
-
-IMPORTANT: `count(local)` counts the current, local object (not the objects in the traversal stream). This works for
-`Collection`- and `Map`-type objects. For any other object, a count of 1 is returned.
-
-[[choose-step]]
-Choose Step
-~~~~~~~~~~~
-
-image::choose-step.png[width=700]
-
-The `choose()`-step (*branch*) routes the current traverser to a particular traversal branch option. With `choose()`,
-it is possible to implement if/else-based semantics as well as more complicated selections.
-
-[gremlin-groovy,modern]
-----
-g.V().hasLabel('person').
- choose(values('age').is(lte(30)),
- __.in(),
- __.out()).values('name') <1>
-g.V().hasLabel('person').
- choose(values('age')).
- option(27, __.in()).
- option(32, __.out()).values('name') <2>
-----
-
-<1> If the traversal yields an element, then do `in`, else do `out` (i.e. true/false-based option selection).
-<2> Use the result of the traversal as a key to the map of traversal options (i.e. value-based option selection).
-
-However, note that `choose()` can have an arbitrary number of options and moreover, can take an anonymous traversal as its choice function.
-
-[gremlin-groovy,modern]
-----
-g.V().hasLabel('person').
- choose(values('name')).
- option('marko', values('age')).
- option('josh', values('name')).
- option('vadas', valueMap()).
- option('peter', label())
-----
-
-The `choose()`-step can leverage the `Pick.none` option match. For anything that does not match a specified option, the `none`-option is taken.
-
-[gremlin-groovy,modern]
-----
-g.V().hasLabel('person').
- choose(values('name')).
- option('marko', values('age')).
- option(none, values('name'))
-----
-
-[[coin-step]]
-Coin Step
-~~~~~~~~~
-
-To randomly filter out a traverser, use the `coin()`-step (*filter*). The provided double argument biases the "coin toss."
-
-[gremlin-groovy,modern]
-----
-g.V().coin(0.5)
-g.V().coin(0.0)
-g.V().coin(1.0)
-----
-
-[[constant-step]]
-Constant Step
-~~~~~~~~~~~~~
-
-To specify a constant value for a traverser, use the `constant()`-step (*map*). This is often useful with conditional
-steps like <<choose-step,`choose()`-step>> or <<coalesce-step,`coalesce()`-step>>.
-
-[gremlin-groovy,modern]
-----
-g.V().choose(__.hasLabel('person'),
- __.values('name'),
- __.constant('inhuman')) <1>
-g.V().coalesce(
- __.hasLabel('person').values('name'),
- __.constant('inhuman')) <2>
-----
-
-<1> Show the names of people, but show "inhuman" for other vertices.
-<2> Same as statement 1 (unless there is a person vertex with no name).
-
-[[cyclicpath-step]]
-CyclicPath Step
-~~~~~~~~~~~~~~~
-
-image::cyclicpath-step.png[width=400]
-
-Each traverser maintains its history through the traversal over the graph -- i.e. its <<path-data-structure,path>>.
-If it is important that the traverser repeat its course, then `cyclic()`-path should be used (*filter*). The step
-analyzes the path of the traverser thus far and if there are any repeats, the traverser is filtered out over the
-traversal computation. If non-cyclic behavior is desired, see <<simplepath-step,`simplePath()`>>.
-
-[gremlin-groovy,modern]
-----
-g.V(1).both().both()
-g.V(1).both().both().cyclicPath()
-g.V(1).both().both().cyclicPath().path()
-----
-
-[[dedup-step]]
-Dedup Step
-~~~~~~~~~~
-
-With `dedup()`-step (*filter*), repeatedly seen objects are removed from the traversal stream. Note that if a
-traverser's bulk is greater than 1, then it is set to 1 before being emitted.
-
-[gremlin-groovy,modern]
-----
-g.V().values('lang')
-g.V().values('lang').dedup()
-g.V(1).repeat(bothE('created').dedup().otherV()).emit().path() <1>
-----
-
-<1> Traverse all `created` edges, but don't touch any edge twice.
-
-If a by-step modulation is provided to `dedup()`, then the object is processed accordingly prior to determining if it
-has been seen or not.
-
-[gremlin-groovy,modern]
-----
-g.V().valueMap(true, 'name')
-g.V().dedup().by(label).values('name')
-----
-
-Finally, if `dedup()` is provided an array of strings, then it will ensure that the de-duplication is not with respect
-to the current traverser object, but to the path history of the traverser.
-
-[gremlin-groovy,modern]
-----
-g.V().as('a').out('created').as('b').in('created').as('c').select('a','b','c')
-g.V().as('a').out('created').as('b').in('created').as('c').dedup('a','b').select('a','b','c') <1>
-----
-
-<1> If the current `a` and `b` combination has been seen previously, then filter the traverser.
-
-[[drop-step]]
-Drop Step
-~~~~~~~~~
-
-The `drop()`-step (*filter*/*sideEffect*) is used to remove element and properties from the graph (i.e. remove). It
-is a filter step because the traversal yields no outgoing objects.
-
-[gremlin-groovy,modern]
-----
-g.V().outE().drop()
-g.E()
-g.V().properties('name').drop()
-g.V().valueMap()
-g.V().drop()
-g.V()
-----
-
-[[explain-step]]
-Explain Step
-~~~~~~~~~~~~
-
-The `explain()`-step (*sideEffect*) will return a `TraversalExplanation`. A traversal explanation details how the
-traversal (prior to `explain()`) will be compiled given the registered <<traversalstrategy,traversal strategies>>.
-A `TraversalExplanation` has a `toString()` representation with 3-columns. The first column is the
-traversal strategy being applied. The second column is the traversal strategy category: [D]ecoration, [O]ptimization,
-[P]rovider optimization, [F]inalization, and [V]erification. Finally, the third column is the state of the traversal
-post strategy application. The final traversal is the resultant execution plan.
-
-[gremlin-groovy,modern]
-----
-g.V().outE().identity().inV().count().is(gt(5)).explain()
-----
-
-For traversal profiling information, please see <<profile-step,`profile()`>>-step.
-
-[[fold-step]]
-Fold Step
-~~~~~~~~~
-
-There are situations when the traversal stream needs a "barrier" to aggregate all the objects and emit a computation
-that is a function of the aggregate. The `fold()`-step (*map*) is one particular instance of this. Please see
-<<unfold-step,`unfold()`>>-step for the inverse functionality.
-
-[gremlin-groovy,modern]
-----
-g.V(1).out('knows').values('name')
-g.V(1).out('knows').values('name').fold() <1>
-g.V(1).out('knows').values('name').fold().next().getClass() <2>
-g.V(1).out('knows').values('name').fold(0) {a,b -> a + b.length()} <3>
-g.V().values('age').fold(0) {a,b -> a + b} <4>
-g.V().values('age').fold(0, sum) <5>
-g.V().values('age').sum() <6>
-----
-
-<1> A parameterless `fold()` will aggregate all the objects into a list and then emit the list.
-<2> A verification of the type of list returned.
-<3> `fold()` can be provided two arguments -- a seed value and a reduce bi-function ("vadas" is 5 characters + "josh" with 4 characters).
-<4> What is the total age of the people in the graph?
-<5> The same as before, but using a built-in bi-function.
-<6> The same as before, but using the <<sum-step,`sum()`-step>>.
-
-[[graph-step]]
-Graph Step
-~~~~~~~~~~
-
-The `V()`-step is usually used to start a `GraphTraversal`, but can also be used mid-traversal.
-
-[gremlin-groovy,modern]
-----
-g.V().has('name', within('marko', 'vadas', 'josh')).as('person').
- V().has('name', within('lop', 'ripple')).addE('uses').from('person')
-----
-
-NOTE: Whether a mid-traversal `V()` uses an index or not, depends on a) whether suitable index exists and b) if the particular graph system provider implemented this functionality.
-
-[gremlin-groovy,modern]
-----
-g.V().has('name', within('marko', 'vadas', 'josh')).as('person').
- V().has('name', within('lop', 'ripple')).addE('uses').from('person').toString() <1>
-g.V().has('name', within('marko', 'vadas', 'josh')).as('person').
- V().has('name', within('lop', 'ripple')).addE('uses').from('person').iterate().toString() <2>
-----
-
-<1> Normally the `V()`-step will iterate over all vertices. However, graph strategies can fold `HasContainer`'s into a `GraphStep` to allow index lookups.
-<2> Whether the graph system provider supports mid-traversal `V()` index lookups or not can easily be determined by inspecting the `toString()` output of the iterated traversal. If `has` conditions were folded into the `V()`-step, an index - if one exists - will be used.
-
-[[group-step]]
-Group Step
-~~~~~~~~~~
-
-As traversers propagate across a graph as defined by a traversal, sideEffect computations are sometimes required.
-That is, the actual path taken or the current location of a traverser is not the ultimate output of the computation,
-but some other representation of the traversal. The `group()`-step (*map*/*sideEffect*) is one such sideEffect that
-organizes the objects according to some function of the object. Then, if required, that organization (a list) is
-reduced. An example is provided below.
-
-[gremlin-groovy,modern]
-----
-g.V().group().by(label) <1>
-g.V().group().by(label).by('name') <2>
-g.V().group().by(label).by(count()) <3>
-----
-
-<1> Group the vertices by their label.
-<2> For each vertex in the group, get their name.
-<3> For each grouping, what is its size?
-
-The three projection parameters available to `group()` via `by()` are:
-
-. Key-projection: What feature of the object to group on (a function that yields the map key)?
-. Value-projection: What feature of the group to store in the key-list?
-. Reduce-projection: What feature of the key-list to ultimately return?
-
-[[groupcount-step]]
-GroupCount Step
-~~~~~~~~~~~~~~~
-
-When it is important to know how many times a particular object has been at a particular part of a traversal,
-`groupCount()`-step (*map*/*sideEffect*) is used.
-
- "What is the distribution of ages in the graph?"
-
-[gremlin-groovy,modern]
-----
-g.V().hasLabel('person').values('age').groupCount()
-g.V().hasLabel('person').groupCount().by('age') <1>
-----
-
-<1> You can also supply a pre-group projection, where the provided <<by-step,`by()`>>-modulation determines what to
-group the incoming object by.
-
-There is one person that is 32, one person that is 35, one person that is 27, and one person that is 29.
-
- "Iteratively walk the graph and count the number of times you see the second letter of each name."
-
-image::groupcount-step.png[width=420]
-
-[gremlin-groovy,modern]
-----
-g.V().repeat(both().groupCount('m').by(label)).times(10).cap('m')
-----
-
-The above is interesting in that it demonstrates the use of referencing the internal `Map<Object,Long>` of
-`groupCount()` with a string variable. Given that `groupCount()` is a sideEffect-step, it simply passes the object
-it received to its output. Internal to `groupCount()`, the object's count is incremented.
-
-[[has-step]]
-Has Step
-~~~~~~~~
-
-image::has-step.png[width=670]
-
-It is possible to filter vertices, edges, and vertex properties based on their properties using `has()`-step
-(*filter*). There are numerous variations on `has()` including:
-
- * `has(key,value)`: Remove the traverser if its element does not have the provided key/value property.
- * `has(key,predicate)`: Remove the traverser if its element does not have a key value that satisfies the bi-predicate.
- * `hasLabel(labels...)`: Remove the traverser if its element does not have any of the labels.
- * `hasId(ids...)`: Remove the traverser if its element does not have any of the ids.
- * `hasKey(keys...)`: Remove the traverser if its property does not have any of the keys.
- * `hasValue(values...)`: Remove the traverser if its property does not have any of the values.
- * `has(key)`: Remove the traverser if its element does not have a value for the key.
- * `hasNot(key)`: Remove the traverser if its element has a value for the key.
- * `has(key, traversal)`: Remove the traverser if its object does not yield a result through the traversal off the property value.
-
-[gremlin-groovy,modern]
-----
-g.V().hasLabel('person')
-g.V().hasLabel('person').out().has('name',within('vadas','josh'))
-g.V().hasLabel('person').out().has('name',within('vadas','josh')).
- outE().hasLabel('created')
-g.V().has('age',inside(20,30)).values('age') <1>
-g.V().has('age',outside(20,30)).values('age') <2>
-g.V().has('name',within('josh','marko')).valueMap() <3>
-g.V().has('name',without('josh','marko')).valueMap() <4>
-g.V().has('name',not(within('josh','marko'))).valueMap() <5>
-----
-
-<1> Find all vertices whose ages are between 20 (inclusive) and 30 (exclusive).
-<2> Find all vertices whose ages are not between 20 (inclusive) and 30 (exclusive).
-<3> Find all vertices whose names are exact matches to any names in the the collection `[josh,marko]`, display all
-the key,value pairs for those verticies.
-<4> Find all vertices whose names are not in the collection `[josh,marko]`, display all the key,value pairs for those vertices.
-<5> Same as the prior example save using `not` on `within` to yield `without`.
-
-TinkerPop does not support a regular expression predicate, although specific graph databases that leverage TinkerPop
-may provide a partial match extension.
-
-[[inject-step]]
-Inject Step
-~~~~~~~~~~~
-
-image::inject-step.png[width=800]
-
-One of the major features of TinkerPop3 is "injectable steps." This makes it possible to insert objects arbitrarily
-into a traversal stream. In general, `inject()`-step (*sideEffect*) exists and a few examples are provided below.
-
-[gremlin-groovy,modern]
-----
-g.V(4).out().values('name').inject('daniel')
-g.V(4).out().values('name').inject('daniel').map {it.get().length()}
-g.V(4).out().values('name').inject('daniel').map {it.get().length()}.path()
-----
-
-In the last example above, note that the path starting with `daniel` is only of length 2. This is because the
-`daniel` string was inserted half-way in the traversal. Finally, a typical use case is provided below -- when the
-start of the traversal is not a graph object.
-
-[gremlin-groovy,modern]
-----
-inject(1,2)
-inject(1,2).map {it.get() + 1}
-inject(1,2).map {it.get() + 1}.map {g.V(it.get()).next()}.values('name')
-----
-
-[[is-step]]
-Is Step
-~~~~~~~
-
-It is possible to filter scalar values using `is()`-step (*filter*).
-
-[gremlin-groovy,modern]
-----
-g.V().values('age').is(32)
-g.V().values('age').is(lte(30))
-g.V().values('age').is(inside(30, 40))
-g.V().where(__.in('created').count().is(1)).values('name') <1>
-g.V().where(__.in('created').count().is(gte(2))).values('name') <2>
-g.V().where(__.in('created').values('age').
- mean().is(inside(30d, 35d))).values('name') <3>
-----
-
-<1> Find projects having exactly one contributor.
-<2> Find projects having two or more contributors.
-<3> Find projects whose contributors average age is between 30 and 35.
-
-[[limit-step]]
-Limit Step
-~~~~~~~~~~
-
-The `limit()`-step is analogous to <<range-step,`range()`-step>> save that the lower end range is set to 0.
-
-[gremlin-groovy,modern]
-----
-g.V().limit(2)
-g.V().range(0, 2)
-g.V().limit(2).toString()
-----
-
-The `limit()`-step can also be applied with `Scope.local`, in which case it operates on the incoming collection.
-The examples below use the <<the-crew-toy-graph,The Crew>> toy data set.
-
-[gremlin-groovy,theCrew]
-----
-g.V().valueMap().select('location').limit(local,2) <1>
-g.V().valueMap().limit(local, 1) <2>
-----
-
-<1> `List<String>` for each vertex containing the first two locations.
-<2> `Map<String, Object>` for each vertex, but containing only the first property value.
-
-[[local-step]]
-Local Step
-~~~~~~~~~~
-
-image::local-step.png[width=450]
-
-A `GraphTraversal` operates on a continuous stream of objects. In many situations, it is important to operate on a
-single element within that stream. To do such object-local traversal computations, `local()`-step exists (*branch*).
-Note that the examples below use the <<the-crew-toy-graph,The Crew>> toy data set.
-
-[gremlin-groovy,theCrew]
-----
-g.V().as('person').
- properties('location').order().by('startTime',incr).limit(2).value().as('location').
- select('person','location').by('name').by() <1>
-g.V().as('person').
- local(properties('location').order().by('startTime',incr).limit(2)).value().as('location').
- select('person','location').by('name').by() <2>
-----
-
-<1> Get the first two people and their respective location according to the most historic location start time.
-<2> For every person, get their two most historic locations.
-
-The two traversals above look nearly identical save the inclusion of `local()` which wraps a section of the traversal
-in a object-local traversal. As such, the `order().by()` and the `limit()` refer to a particular object, not to the
-stream as a whole.
-
-WARNING: The anonymous traversal of `local()` processes the current object "locally." In OLAP, where the atomic unit
-of computing is the the vertex and its local "star graph," it is important that the anonymous traversal does not leave
-the confines of the vertex's star graph. In other words, it can not traverse to an adjacent vertex's properties or edges.
-
-[[match-step]]
-Match Step
-~~~~~~~~~~
-
-The `match()`-step (*map*) provides a more link:http://en.wikipedia.org/wiki/Declarative_programming[declarative]
-form of graph querying based on the notion of link:http://en.wikipedia.org/wiki/Pattern_matching[pattern matching].
-With `match()`, the user provides a collection of "traversal fragments," called patterns, that have variables defined
-that must hold true throughout the duration of the `match()`. When a traverser is in `match()`, a registered
-`MatchAlgorithm` analyzes the current state of the traverser (i.e. its history based on its
-<<path-data-structure,path data>>), the runtime statistics of the traversal patterns, and returns a traversal-pattern
-that the traverser should try next. The default `MatchAlgorithm` provided is called `CountMatchAlgorithm` and it
-dynamically revises the pattern execution plan by sorting the patterns according to their filtering capabilities
-(i.e. largest set reduction patterns execute first). For very large graphs, where the developer is uncertain of the
-statistics of the graph (e.g. how many `knows`-edges vs. `worksFor`-edges exist in the graph), it is advantageous to
-use `match()`, as an optimal plan will be determined automatically. Furthermore, some queries are much easier to
-express via `match()` than with single-path traversals.
-
- "Who created a project named 'lop' that was also created by someone who is 29 years old? Return the two creators."
-
-image::match-step.png[width=500]
-
-[gremlin-groovy,modern]
-----
-g.V().match(
- __.as('a').out('created').as('b'),
- __.as('b').has('name', 'lop'),
- __.as('b').in('created').as('c'),
- __.as('c').has('age', 29)).
- select('a','c').by('name')
-----
-
-Note that the above can also be more concisely written as below which demonstrates that standard inner-traversals can
-be arbitrarily defined.
-
-[gremlin-groovy,modern]
-----
-g.V().match(
- __.as('a').out('created').has('name', 'lop').as('b'),
- __.as('b').in('created').has('age', 29).as('c')).
- select('a','c').by('name')
-----
-
-In order to improve readability, `as()`-steps can be given meaningful labels which better reflect your domain. The
-previous query can thus be written in a more expressive way as shown below.
-
-[gremlin-groovy,modern]
-----
-g.V().match(
- __.as('creators').out('created').has('name', 'lop').as('projects'), <1>
- __.as('projects').in('created').has('age', 29).as('cocreators')). <2>
- select('creators','cocreators').by('name') <3>
-----
-
-<1> Find vertices that created something and match them as 'creators', then find out what they created which is
-named 'lop' and match these vertices as 'projects'.
-<2> Using these 'projects' vertices, find out their creators aged 29 and remember these as 'cocreators'.
-<3> Return the name of both 'creators' and 'cocreators'.
-
-[[grateful-dead]]
-.Grateful Dead
-image::grateful-dead-schema.png[width=475]
-
-`MatchStep` brings functionality similar to link:http://en.wikipedia.org/wiki/SPARQL[SPARQL] to Gremlin. Like SPARQL,
-MatchStep conjoins a set of patterns applied to a graph. For example, the following traversal finds exactly those
-songs which Jerry Garcia has both sung and written (using the Grateful Dead graph distributed in the `data/` directory):
-
-[gremlin-groovy]
-----
-graph.io(graphml()).readGraph('data/grateful-dead.xml')
-g = graph.traversal(standard())
-g.V().match(
- __.as('a').has('name', 'Garcia'),
- __.as('a').in('writtenBy').as('b'),
- __.as('a').in('sungBy').as('b')).
- select('b').values('name')
-----
-
-Among the features which differentiate `match()` from SPARQL are:
-
-[gremlin-groovy,modern]
-----
-g.V().match(
- __.as('a').out('created').has('name','lop').as('b'), <1>
- __.as('b').in('created').has('age', 29).as('c'),
- __.as('c').repeat(out()).times(2)). <2>
- select('c').out('knows').dedup().values('name') <3>
-----
-
-<1> *Patterns of arbitrary complexity*: `match()` is not restricted to triple patterns or property paths.
-<2> *Recursion support*: `match()` supports the branch-based steps within a pattern, including `repeat()`.
-<3> *Imperative/declarative hybrid*: Before and after a `match()`, it is possible to leverage classic Gremlin traversals.
-
-To extend point #3, it is possible to support going from imperative, to declarative, to imperative, ad infinitum.
-
-[gremlin-groovy,modern]
-----
-g.V().match(
- __.as('a').out('knows').as('b'),
- __.as('b').out('created').has('name','lop')).
- select('b').out('created').
- match(
- __.as('x').in('created').as('y'),
- __.as('y').out('knows').as('z')).
- select('z').values('name')
-----
-
-IMPORTANT: The `match()`-step is stateless. The variable bindings of the traversal patterns are stored in the path
-history of the traverser. As such, the variables used over all `match()`-steps within a traversal are globally unique.
-A benefit of this is that subsequent `where()`, `select()`, `match()`, etc. steps can leverage the same variables in
-their analysis.
-
-Like all other steps in Gremlin, `match()` is a function and thus, `match()` within `match()` is a natural consequence
-of Gremlin's functional foundation (i.e. recursive matching).
-
-[gremlin-groovy,modern]
-----
-g.V().match(
- __.as('a').out('knows').as('b'),
- __.as('b').out('created').has('name','lop'),
- __.as('b').match(
- __.as('b').out('created').as('c'),
- __.as('c').has('name','ripple')).
- select('c').as('c')).
- select('a','c').by('name')
-----
-
-If a step-labeled traversal proceeds the `match()`-step and the traverser entering the `match()` is destined to bind
-to a particular variable, then the previous step should be labeled accordingly.
-
-[gremlin-groovy,modern]
-----
-g.V().as('a').out('knows').as('b').
- match(
- __.as('b').out('created').as('c'),
- __.not(__.as('c').in('created').as('a'))).
- select('a','b','c').by('name')
-----
-
-There are three types of `match()` traversal patterns.
-
- . `as('a')...as('b')`: both the start and end of the traversal have a declared variable.
- . `as('a')...`: only the start of the traversal has a declared variable.
- . `...`: there are no declared variables.
-
-If a variable is at the start of a traversal pattern it *must* exist as a label in the path history of the traverser
-else the traverser can not go down that path. If a variable is at the end of a traversal pattern then if the variable
-exists in the path history of the traverser, the traverser's current location *must* match (i.e. equal) its historic
-location at that same label. However, if the variable does not exist in the path history of the traverser, then the
-current location is labeled as the variable and thus, becomes a bound variable for subsequent traversal patterns. If a
-traversal pattern does not have an end label, then the traverser must simply "survive" the pattern (i.e. not be
-filtered) to continue to the next pattern. If a traversal pattern does not have a start label, then the traverser
-can go down that path at any point, but will only go down that pattern once as a traversal pattern is executed once
-and only once for the history of the traverser. Typically, traversal patterns that do not have a start and end label
-are used in conjunction with `and()`, `or()`, and `where()`. Once the traverser has "survived" all the patterns (or at
-least one for `or()`), `match()`-step analyzes the traverser's path history and emits a `Map<String,Object>` of the
-variable bindings to the next step in the traversal.
-
-[gremlin-groovy,modern]
-----
-g.V().as('a').out().as('b'). <1>
- match( <2>
- __.as('a').out().count().as('c'), <3>
- __.not(__.as('a').in().as('b')), <4>
- or( <5>
- __.as('a').out('knows').as('b'),
- __.as('b').in().count().as('c').and().as('c').is(gt(2)))). <6>
- dedup('a','c'). <7>
- select('a','b','c').by('name').by('name').by() <8>
-----
-
-<1> A standard, step-labeled traversal can come prior to `match()`.
-<2> If the traverser's path prior to entering `match()` has requisite label values, then those historic values are bound.
-<3> It is possible to use <<a-note-on-barrier-steps,barrier steps>> though they are computed locally to the pattern (as one would expect).
-<4> It is possible to `not()` a pattern.
-<5> It is possible to nest `and()`- and `or()`-steps for conjunction matching.
-<6> Both infix and prefix conjunction notation is supported.
-<7> It is possible to "distinct" the specified label combination.
-<8> The bound values are of different types -- vertex ("a"), vertex ("b"), long ("c").
-
-[[using-where-with-match]]
-Using Where with Match
-^^^^^^^^^^^^^^^^^^^^^^
-
-Match is typically used in conjunction with both `select()` (demonstrated previously) and `where()` (presented here).
-A `where()`-step allows the user to further constrain the result set provided by `match()`.
-
-[gremlin-groovy,modern]
-----
-g.V().match(
- __.as('a').out('created').as('b'),
- __.as('b').in('created').as('c')).
- where('a', neq('c')).
- select('a','c').by('name')
-----
-
-The `where()`-step can take either a `P`-predicate (example above) or a `Traversal` (example below). Using
-`MatchPredicateStrategy`, `where()`-clauses are automatically folded into `match()` and thus, subject to the query
-optimizer within `match()`-step.
-
-[gremlin-groovy,modern]
-----
-traversal = g.V().match(
- __.as('a').has(label,'person'), <1>
- __.as('a').out('created').as('b'),
- __.as('b').in('created').as('c')).
- where(__.as('a').out('knows').as('c')). <2>
- select('a','c').by('name'); null <3>
-traversal.toString() <4>
-traversal <5> <6>
-traversal.toString() <7>
-----
-
-<1> Any `has()`-step traversal patterns that start with the match-key are pulled out of `match()` to enable the graph
-system to leverage the filter for index lookups.
-<2> A `where()`-step with a traversal containing variable bindings declared in `match()`.
-<3> A useful trick to ensure that the traversal is not iterated by Gremlin Console.
-<4> The string representation of the traversal prior to its strategies being applied.
-<5> The Gremlin Console will automatically iterate anything that is an iterator or is iterable.
-<6> Both marko and josh are co-developers and marko knows josh.
-<7> The string representation of the traversal after the strategies have been applied (and thus, `where()` is folded into `match()`)
-
-IMPORTANT: A `where()`-step is a filter and thus, variables within a `where()` clause are not globally bound to the
-path of the traverser in `match()`. As such, `where()`-steps in `match()` are used for filtering, not binding.
-
-[[max-step]]
-Max Step
-~~~~~~~~
-
-The `max()`-step (*map*) operates on a stream of numbers and determines which is the largest number in the stream.
-
-[gremlin-groovy,modern]
-----
-g.V().values('age').max()
-g.V().repeat(both()).times(3).values('age').max()
-----
-
-IMPORTANT: `max(local)` determines the max of the current, local object (not the objects in the traversal stream).
-This works for `Collection` and `Number`-type objects. For any other object, a max of `Double.NaN` is returned.
-
-[[mean-step]]
-Mean Step
-~~~~~~~~~
-
-The `mean()`-step (*map*) operates on a stream of numbers and determines the average of those numbers.
-
-[gremlin-groovy,modern]
-----
-g.V().values('age').mean()
-g.V().repeat(both()).times(3).values('age').mean() <1>
-g.V().repeat(both()).times(3).values('age').dedup().mean()
-----
-
-<1> Realize that traversers are being bulked by `repeat()`. There may be more of a particular number than another,
-thus altering the average.
-
-IMPORTANT: `mean(local)` determines the mean of the current, local object (not the objects in the traversal stream).
-This works for `Collection` and `Number`-type objects. For any other object, a mean of `Double.NaN` is returned.
-
-[[min-step]]
-Min Step
-~~~~~~~~
-
-The `min()`-step (*map*) operates on a stream of numbers and determines which is the smallest number in the stream.
-
-[gremlin-groovy,modern]
-----
-g.V().values('age').min()
-g.V().repeat(both()).times(3).values('age').min()
-----
-
-IMPORTANT: `min(local)` determines the min of the current, local object (not the objects in the traversal stream).
-This works for `Collection` and `Number`-type objects. For any other object, a min of `Double.NaN` is returned.
-
-[[or-step]]
-Or Step
-~~~~~~~
-
-The `or()`-step ensures that at least one of the provided traversals yield a result (*filter*). Please see
-<<and-step,`and()`>> for and-semantics.
-
-[gremlin-groovy,modern]
-----
-g.V().or(
- __.outE('created'),
- __.inE('created').count().is(gt(1))).
- values('name')
-----
-
-The `or()`-step can take an arbitrary number of traversals. At least one of the traversals must produce at least one
-output for the original traverser to pass to the next step.
-
-An link:http://en.wikipedia.org/wiki/Infix_notation[infix notation] can be used as well. Though, with infix notation,
-only two traversals can be or'd together.
-
-[gremlin-groovy,modern]
-----
-g.V().where(outE('created').or().outE('knows')).values('name')
-----
-
-[[order-step]]
-Order Step
-~~~~~~~~~~
-
-When the objects of the traversal stream need to be sorted, `order()`-step (*map*) can be leveraged.
-
-[gremlin-groovy,modern]
-----
-g.V().values('name').order()
-g.V().values('name').order().by(decr)
-g.V().hasLabel('person').order().by('age', incr).values('name')
-----
-
-One of the most traversed objects in a traversal is an `Element`. An element can have properties associated with it
-(i.e. key/value pairs). In many situations, it is desirable to sort an element traversal stream according to a
-comparison of their properties.
-
-[gremlin-groovy,modern]
-----
-g.V().values('name')
-g.V().order().by('name',incr).values('name')
-g.V().order().by('name',decr).values('name')
-----
-
-The `order()`-step allows the user to provide an arbitrary number of comparators for primary, secondary, etc. sorting.
-In the example below, the primary ordering is based on the outgoing created-edge count. The secondary ordering is
-based on the age of the person.
-
-[gremlin-groovy,modern]
-----
-g.V().hasLabel('person').order().by(outE('created').count(), incr).
- by('age', incr).values('name')
-g.V().hasLabel('person').order().by(outE('created').count(), incr).
- by('age', decr).values('name')
-----
-
-Randomizing the order of the traversers at a particular point in the traversal is possible with `Order.shuffle`.
-
-[gremlin-groovy,modern]
-----
-g.V().hasLabel('person').order().by(shuffle)
-g.V().hasLabel('person').order().by(shuffle)
-----
-
-IMPORTANT: `order(local)` orders the current, local object (not the objects in the traversal stream). This works for
-`Collection`- and `Map`-type objects. For any other object, the object is returned unchanged.
-
-[[path-step]]
-Path Step
-~~~~~~~~~
-
-A traverser is transformed as it moves through a series of steps within a traversal. The history of the traverser is
-realized by examining its path with `path()`-step (*map*).
-
-image::path-step.png[width=650]
-
-[gremlin-groovy,modern]
-----
-g.V().out().out().values('name')
-g.V().out().out().values('name').path()
-----
-
-If edges are required in the path, then be sure to traverser those edges explicitly.
-
-[gremlin-groovy,modern]
-----
-g.V().outE().inV().outE().inV().path()
-----
-
-It is possible to post-process the elements of the path in a round-robin fashion via `by()`.
-
-[gremlin-groovy,modern]
-----
-g.V().out().out().path().by('name').by('age')
-----
-
-Finally, because `by()`-based post-processing, nothing prevents triggering yet another traversal. In the traversal
-below, for each element of the path traversed thus far, if its a person (as determined by having an `age`-property),
-then get all of their creations, else if its a creation, get all the people that created it.
-
-[gremlin-groovy,modern]
-----
-g.V().out().out().path().by(
- choose(hasLabel('person'),
- out('created').values('name'),
- __.in('created').values('name')).fold())
-----
-
-WARNING: Generating path information is expensive as the history of the traverser is stored into a Java list. With
-numerous traversers, there are numerous lists. Moreover, in an OLAP <<graphcomputer,`GraphComputer`>> environment
-this becomes exceedingly prohibitive as there are traversers emanating from all vertices in the graph in parallel.
-In OLAP there are optimizations provided for traverser populations, but when paths are calculated (and each traverser
-is unique due to its history), then these optimizations are no longer possible.
-
-[[path-data-structure]]
-Path Data Structure
-^^^^^^^^^^^^^^^^^^^
-
-The `Path` data structure is an ordered list of objects, where each object is associated to a `Set<String>` of
-labels. An example is presented below to demonstrate both the `Path` API as well as how a traversal yields labeled paths.
-
-image::path-data-structure.png[width=350]
-
-[gremlin-groovy,modern]
-----
-path = g.V(1).as('a').has('name').as('b').
- out('knows').out('created').as('c').
- has('name','ripple').values('name').as('d').
- identity().as('e').path().next()
-path.size()
-path.objects()
-path.labels()
-path.a
-path.b
-path.c
-path.d == path.e
-----
-
-[[profile-step]]
-Profile Step
-~~~~~~~~~~~~
-
-The `profile()`-step (*sideEffect*) exists to allow developers to profile their traversals to determine statistical
-information like step runtime, counts, etc.
-
-WARNING: Profiling a Traversal will impede the Traversal's performance. This overhead is mostly excluded from the
-profile results, but durations are not exact. Thus, durations are best considered in relation to each other.
-
-[gremlin-groovy,modern]
-----
-g.V().out('created').repeat(both()).times(3).hasLabel('person').values('age').sum().profile().cap(TraversalMetrics.METRICS_KEY)
-----
-
-The `profile()`-step generates a `TraversalMetrics` sideEffect object that contains the following information:
-
-* `Step`: A step within the traversal being profiled.
-* `Count`: The number of _represented_ traversers that passed through the step.
-* `Traversers`: The number of traversers that passed through the step.
-* `Time (ms)`: The total time the step was actively executing its behavior.
-* `% Dur`: The percentage of total time spent in the step.
-
-image:gremlin-exercise.png[width=120,float=left] It is important to understand the difference between `Count`
-and `Traversers`. Traversers can be merged and as such, when two traversers are "the same" they may be aggregated
-into a single traverser. That new traverser has a `Traverser.bulk()` that is the sum of the two merged traverser
-bulks. On the other hand, the `Count` represents the sum of all `Traverser.bulk()` results and thus, expresses the
-number of "represented" (not enumerated) traversers. `Traversers` will always be less than or equal to `Count`.
-
-For traversal compilation information, please see <<explain-step,`explain()`>>-step.
-
-[[range-step]]
-Range Step
-~~~~~~~~~~
-
-As traversers propagate through the traversal, it is possible to only allow a certain number of them to pass through
-with `range()`-step (*filter*). When the low-end of the range is not met, objects are continued to be iterated. When
-within the low and high range (both inclusive), traversers are emitted. Finally, when above the high range, the
-traversal breaks out of iteration.
-
-[gremlin-groovy,modern]
-----
-g.V().range(0,3)
-g.V().range(1,3)
-g.V().repeat(both()).times(1000000).emit().range(6,10)
-----
-
-The `range()`-step can also be applied with `Scope.local`, in which case it operates on the incoming collection.
-For example, it is possible to produce a `Map<String, String>` for each traversed path, but containing only the second
-property value (the "b" step).
-
-[gremlin-groovy,modern]
-----
-g.V().as('a').out().as('b').in().as('c').select('a','b','c').by('name').range(local,1,2)
-----
-
-The next example uses the <<the-crew-toy-graph,The Crew>> toy data set. It produces a `List<String>` containing the
-second and third location for each vertex.
-
-[gremlin-groovy,theCrew]
-----
-g.V().valueMap().select('location').range(local, 1, 3)
-----
-
-[[repeat-step]]
-Repeat Step
-~~~~~~~~~~~
-
-image::gremlin-fade.png[width=350]
-
-The `repeat()`-step (*branch*) is used for looping over a traversal given some break predicate. Below are some
-examples of `repeat()`-step in action.
-
-[gremlin-groovy,modern]
-----
-g.V(1).repeat(out()).times(2).path().by('name') <1>
-g.V().until(has('name','ripple')).
- repeat(out()).path().by('name') <2>
-----
-
-<1> do-while semantics stating to do `out()` 2 times.
-<2> while-do semantics stating to break if the traverser is at a vertex named "ripple".
-
-IMPORTANT: There are two modulators for `repeat()`: `until()` and `emit()`. If `until()` comes after `repeat()` it is
-do/while looping. If `until()` comes before `repeat()` it is while/do looping. If `emit()` is placed after `repeat()`,
-it is evaluated on the traversers leaving the repeat-traversal. If `emit()` is placed before `repeat()`, it is
-evaluated on the traversers prior to entering the repeat-traversal.
-
-The `repeat()`-step also supports an "emit predicate", where the predicate for an empty argument `emit()` is
-`true` (i.e. `emit() == emit{true}`). With `emit()`, the traverser is split in two -- the traverser exits the code
-block as well as continues back within the code block (assuming `until()` holds true).
-
-[gremlin-groovy,modern]
-----
-g.V(1).repeat(out()).times(2).emit().path().by('name') <1>
-g.V(1).emit().repeat(out()).times(2).path().by('name') <2>
-----
-
-<1> The `emit()` comes after `repeat()` and thus, emission happens after the `repeat()` traversal is executed. Thus,
-no one vertex paths exist.
-<2> The `emit()` comes before `repeat()` and thus, emission happens prior to the `repeat()` traversal being executed.
-Thus, one vertex paths exist.
-
-The `emit()`-modulator can take an arbitrary predicate.
-
-[gremlin-groovy,modern]
-----
-g.V(1).repeat(out()).times(2).emit(has('lang')).path().by('name')
-----
-
-image::repeat-step.png[width=500]
-
-[gremlin-groovy,modern]
-----
-g.V(1).repeat(out()).times(2).emit().path().by('name')
-----
-
-The first time through the `repeat()`, the vertices lop, vadas, and josh are seen. Given that `loops==1`, the
-traverser repeats. However, because the emit-predicate is declared true, those vertices are emitted. The next time through
- `repeat()`, the vertices traversed are ripple and lop (Josh's created projects, as lop and vadas have no out edges).
- Given that `loops==2`, the until-predicate fails and ripple and lop are emitted.
-Therefore, the traverser has seen the vertices: lop, vadas, josh, ripple, and lop.
-
-Finally, note that both `emit()` and `until()` can take a traversal and in such, situations, the predicate is
-determined by `traversal.hasNext()`. A few examples are provided below.
-
-[gremlin-groovy,modern]
-----
-g.V(1).repeat(out()).until(hasLabel('software')).path().by('name') <1>
-g.V(1).emit(hasLabel('person')).repeat(out()).path().by('name') <2>
-g.V(1).repeat(out()).until(outE().count().is(0)).path().by('name') <3>
-----
-
-<1> Starting from vertex 1, keep taking outgoing edges until a software vertex is reached.
-<2> Starting from vertex 1, and in an infinite loop, emit the vertex if it is a person and then traverser the outgoing edges.
-<3> Starting from vertex 1, keep taking outgoing edges until a vertex is reached that has no more outgoing edges.
-
-WARNING: The anonymous traversal of `emit()` and `until()` (not `repeat()`) process their current objects "locally."
-In OLAP, where the atomic unit of computing is the the vertex and its local "star graph," it is important that the
-anonymous traversals do not leave the confines of the vertex's star graph. In other words, they can not traverse to
-an adjacent vertex's properties or edges.
-
-[[sack-step]]
-Sack Step
-~~~~~~~~~
-
-image:gremlin-sacks-running.png[width=175,float=right] A traverser can contain a local data structure called a "sack".
-The `sack()`-step is used to read and write sacks (*sideEffect* or *map*). Each sack of each traverser is created
-when using `GraphTraversal.withSack(initialValueSupplier,splitOperator?,mergeOperator?)`.
-
-* *Initial value supplier*: A `Supplier` providing the initial value of each traverser's sack.
-* *Split operator*: a `UnaryOperator` that clones the traverser's sack when the traverser splits. If no split operator
-is provided, then `UnaryOperator.identity()` is assumed.
-* *Merge operator*: A `BinaryOperator` that unites two traverser's sack when they are merged. If no merge operator is
-provided, then traversers with sacks can not be merged.
-
-Two trivial examples are presented below to demonstrate the *initial value supplier*. In the first example below, a
-traverser is created at each vertex in the graph (`g.V()`), with a 1.0 sack (`withSack(1.0f)`), and then the sack
-value is accessed (`sack()`). In the second example, a random float supplier is used to generate sack values.
-
-[gremlin-groovy,modern]
-----
-g.withSack(1.0f).V().sack()
-rand = new Random()
-g.withSack {rand.nextFloat()}.V().sack()
-----
-
-A more complicated initial value supplier example is presented below where the sack values are used in a running
-computation and then emitted at the end of the traversal. When an edge is traversed, the edge weight is multiplied
-by the sack value (`sack(mult).by('weight')`). Note that the <<by-step,`by()`>>-modulator can be any arbitrary traversal.
-
-[gremlin-groovy,modern]
-----
-g.withSack(1.0f).V().repeat(outE().sack(mult).by('weight').inV()).times(2)
-g.withSack(1.0f).V().repeat(outE().sack(mult).by('weight').inV()).times(2).sack()
-g.withSack(1.0f).V().repeat(outE().sack(mult).by('weight').inV()).times(2).path().
- by().by('weight')
-----
-
-image:gremlin-sacks-standing.png[width=100,float=left] When complex objects are used (i.e. non-primitives), then a
-*split operator* should be defined to ensure that each traverser gets a clone of its parent's sack. The first example
-does not use a split operator and as such, the same map is propagated to all traversers (a global data structure). The
-second example, demonstrates how `Map.clone()` ensures that each traverser's sack contains a unique, local sack.
-
-[gremlin-groovy,modern]
-----
-g.withSack {[:]}.V().out().out().
- sack {m,v -> m[v.value('name')] = v.value('lang'); m}.sack() // BAD: single map
-g.withSack {[:]}{it.clone()}.V().out().out().
- sack {m,v -> m[v.value('name')] = v.value('lang'); m}.sack() // GOOD: cloned map
-----
-
-NOTE: For primitives (i.e. integers, longs, floats, etc.), a split operator is not required as a primitives are
-encoded in the memory address of the sack, not as a reference to an object.
-
-If a *merge operator* is not provided, then traversers with sacks can not be bulked. However, in many situations,
-merging the sacks of two traversers at the same location is algorithmically sound and good to provide so as to gain
-the bulking optimization. In the examples below, the binary merge operator is `Operator.sum`. Thus, when two traverser
-merge, their respective sacks are added together.
-
-[gremlin-groovy,modern]
-----
-g.withSack(1.0f,sum).V(1).local(outE('knows').barrier(normSack).inV()) <1>
-g.withSack(1.0f,sum).V(1).local(outE('knows').barrier(normSack).inV()).sack() <2>
-g.withSack(1.0f,sum).V(1).local(outE('knows').barrier(normSack).inV()).in('knows') <3>
-g.withSack(1.0f,sum).V(1).local(outE('knows').barrier(normSack).inV()).in('knows').sack() <4>
-g.withSack(1.0f,sum).V(1).local(outE('knows').barrier(normSack).inV()).in('knows').barrier().sack() <5>
-g.withBulk(false).withSack(1.0f,sum).V(1).local(outE('knows').barrier(normSack).inV()).in('knows').barrier().sack() <6>
-----
-
-<1> The knows-adjacent vertices of vertex 1 are vertices 2 and 4.
-<2> The `local(...barrier(normSack)...)` ensures that all traversers leaving vertex 1 have an evenly distributed amount of the initial 1.0 "energy" (50-50).
-<3> Going from vertices 2 and 4 yield two traversers at vertex 1.
-<4> Those two traversers each have a sack of 0.5.
-<5> The `barrier()` merges the two traversers at vertex 1 into a single traverser whose sack is 1.0.
-<6> There is now a single traverser with bulk of 2 and sack of 1.0 and thus, setting `withBulk(false)` yields the expected 1.0.
-
-
-[[sample-step]]
-Sample Step
-~~~~~~~~~~~
-
-The `sample()`-step is useful for sampling some number of traversers previous in the traversal.
-
-[gremlin-groovy,modern]
-----
-g.V().outE().sample(1).values('weight')
-g.V().outE().sample(1).by('weight').values('weight')
-g.V().outE().sample(2).by('weight').values('weight')
-----
-
-One of the more interesting use cases for `sample()` is when it is used in conjunction with <<local-step,`local()`>>.
-The combination of the two steps supports the execution of link:http://en.wikipedia.org/wiki/Random_walk[random walks].
-In the example below, the traversal starts are vertex 1 and selects one edge to traverse based on a probability
-distribution generated by the weights of the edges. The output is always a single path as by selecting a single edge,
-the traverser never splits and continues down a single path in the graph.
-
-[gremlin-groovy,modern]
-----
-g.V(1).repeat(local(
- bothE().sample(1).by('weight').otherV()
- )).times(5)
-g.V(1).repeat(local(
- bothE().sample(1).by('weight').otherV()
- )).times(5).path()
-g.V(1).repeat(local(
- bothE().sample(1).by('weight').otherV()
- )).times(10).path()
-----
-
-[[select-step]]
-Select Step
-~~~~~~~~~~~
-
-link:http://en.wikipedia.org/wiki/Functional_programming[Functional languages] make use of function composition and
-lazy evaluation to create complex computations from primitive operations. This is exactly what `Traversal` does. One
-of the differentiating aspects of Gremlin's data flow approach to graph processing is that the flow need not always go
-"forward," but in fact, can go back to a previously seen area of computation. Examples include <<path-step,`path()`>>
-as well as the `select()`-step (*map*). There are two general ways to use `select()`-step.
-
-. Select labeled steps within a path (as defined by `as()` in a traversal).
-. Select objects out of a `Map<String,Object>` flow (i.e. a sub-map).
-
-The first use case is demonstrated via example below.
-
-[gremlin-groovy,modern]
-----
-g.V().as('a').out().as('b').out().as('c') // no select
-g.V().as('a').out().as('b').out().as('c').select('a','b','c')
-g.V().as('a').out().as('b').out().as('c').select('a','b')
-g.V().as('a').out().as('b').out().as('c').select('a','b').by('name')
-g.V().as('a').out().as('b').out().as('c').select('a') <1>
-----
-
-<1> If the selection is one step, no map is returned.
-
-When there is only one label selected, then a single object is returned. This is useful for stepping back in a
-computation and easily moving forward again on the object reverted to.
-
-[gremlin-groovy,modern]
-----
-g.V().out().out()
-g.V().out().out().path()
-g.V().as('x').out().out().select('x')
-g.V().out().as('x').out().select('x')
-g.V().out().out().as('x').select('x') // pointless
-----
-
-NOTE: When executing a traversal with `select()` on a standard traversal engine (i.e. OLTP), `select()` will do its
-best to avoid calculating the path history and instead, will rely on a global data structure for storing the currently
-selected object. As such, if only a subset of the path walked is required, `select()` should be used over the more
-resource intensive <<path-step,`path()`>>-step.
-
-When the set of keys or values (i.e. columns) of a path or map are needed, use `select(keys)` and `select(values)`,
-respectively. This is especially useful when one is only interested in the top N elements in a `groupCount()`
-ranking.
-
-[gremlin-groovy]
-----
-graph.io(graphml()).readGraph('data/grateful-dead.xml')
-g = graph.traversal()
-g.V().hasLabel('song').out('followedBy').groupCount().by('name').
- order(local).by(valueDecr).limit(local, 5)
-g.V().hasLabel('song').out('followedBy').groupCount().by('name').
- order(local).by(valueDecr).limit(local, 5).select(keys)
-g.V().hasLabel('song').out('followedBy').groupCount().by('name').
- order(local).by(valueDecr).limit(local, 5).select(keys).unfold()
-----
-
-Similarly, for extracting the values from a path or map.
-
-[gremlin-groovy]
-----
-graph.io(graphml()).readGraph('data/grateful-dead.xml')
-g = graph.traversal()
-g.V().hasLabel('song').out('sungBy').groupCount().by('name') <1>
-g.V().hasLabel('song').out('sungBy').groupCount().by('name').select(values) <2>
-g.V().hasLabel('song').out('sungBy').groupCount().by('name').select(values).unfold().
- groupCount().order(local).by(valueDecr).limit(local, 5) <3>
-----
-
-<1> Which artist sung how many songs?
-<2> Get an anonymized set of song repertoire sizes.
-<3> What are the 5 most common song repertoire sizes?
-
-CAUTION: Note that `by()`-modulation is not supported with `select(keys)` and `select(values)`.
-
-[[using-where-with-select]]
-Using Where with Select
-^^^^^^^^^^^^^^^^^^^^^^^
-
-Like <<match-step,`match()`>>-step, it is possible to use `where()`, as where is a filter that processes
-`Map<String,Object>` streams.
-
-[gremlin-groovy,modern]
-----
-g.V().as('a').out('created').in('created').as('b').select('a','b').by('name') <1>
-g.V().as('a').out('created').in('created').as('b').
- select('a','b').by('name').where('a',neq('b')) <2>
-g.V().as('a').out('created').in('created').as('b').
- select('a','b'). <3>
- where('a',neq('b')).
- where(__.as('a').out('knows').as('b')).
- select('a','b').by('name')
-----
-
-<1> A standard `select()` that generates a `Map<String,Object>` of variables bindings in the path (i.e. `a` and `b`)
-for the sake of a running example.
-<2> The `select().by('name')` projects each binding vertex to their name property value and `where()` operates to
-ensure respective `a` and `b` strings are not the same.
-<3> The first `select()` projects a vertex binding set. A binding is filtered if `a` vertex equals `b` vertex. A
-binding is filtered if `a` doesn't know `b`. The second and final `select()` projects the name of the vertices.
-
-[[simplepath-step]]
-SimplePath Step
-~~~~~~~~~~~~~~~
-
-image::simplepath-step.png[width=400]
-
-When it is important that a traverser not repeat its path through the graph, `simplePath()`-step should be used
-(*filter*). The <<path-data-structure,path>> information of the traverser is analyzed and if the path has repeated
-objects in it, the traverser is filtered. If cyclic behavior is desired, see <<cyclicpath-step,`cyclicPath()`>>.
-
-[gremlin-groovy,modern]
-----
-g.V(1).both().both()
-g.V(1).both().both().simplePath()
-g.V(1).both().both().simplePath().path()
-----
-
-[[store-step]]
-Store Step
-~~~~~~~~~~
-
-When link:http://en.wikipedia.org/wiki/Lazy_evaluation[lazy] aggregation is needed, `store()`-step (*sideEffect*)
-should be used over <<aggregate-step,`aggregate()`>>. The two steps differ in that `store()` does not block and only
-stores objects in its side-effect collection as they pass through.
-
-[gremlin-groovy,modern]
-----
-g.V().aggregate('x').limit(1).cap('x')
-g.V().store('x').limit(1).cap('x')
-----
-
-It is interesting to note that there are three results in the `store()` side-effect even though the interval
-selection is for 2 objects. Realize that when the third object is on its way to the `range()` filter (i.e. `[0..1]`),
-it passes through `store()` and thus, stored before filtered.
-
-[gremlin-groovy,modern]
-----
-g.E().store('x').by('weight').cap('x')
-----
-
-[[subgraph-step]]
-Subgraph Step
-~~~~~~~~~~~~~
-
-image::subgraph-logo.png[width=380]
-
-Extracting a portion of a graph from a larger one for analysis, visualization or other purposes is a fairly common
-use case for graph analysts and developers. The `subgraph()`-step (*sideEffect*) provides a way to produce an
-link:http://mathworld.wolfram.com/Edge-InducedSubgraph.html[edge-induced subgraph] from virtually any traversal.
-The following example demonstrates how to produce the "knows" subgraph:
-
-[gremlin-groovy,modern]
-----
-subGraph = g.E().hasLabel('knows').subgraph('subGraph').cap('subGraph').next() <1>
-sg = subGraph.traversal(standard())
-sg.E() <2>
-----
-
-<1> As this function produces "edge-induced" subgraphs, `subgraph()` must be called at edge steps.
-<2> The subgraph contains only "knows" edges.
-
-A more common subgraphing use case is to get all of the graph structure surrounding a single vertex:
-
-[gremlin-groovy,modern]
-----
-subGraph = g.V(3).repeat(__.inE().subgraph('subGraph').outV()).times(3).cap('subGraph').next() <1>
-sg = subGraph.traversal(standard())
-sg.E()
-----
-
-<1> Starting at vertex `3`, traverse 3 steps away on in-edges, outputting all of that into the subgraph.
-
-There can be multiple `subgraph()` calls within the same traversal. Each operating against either the same graph
-(i.e. same side-effect key) or different graphs (i.e. different side-effect keys).
-
-[gremlin-groovy,modern]
-----
-t = g.V().outE('knows').subgraph('knowsG').inV().outE('created').subgraph('createdG').
- inV().inE('created').subgraph('createdG').iterate()
-t.sideEffects.get('knowsG').get().traversal(standard()).E()
-t.sideEffects.get('createdG').get().traversal(standard()).E()
-----
-
-IMPORTANT: The `subgraph()`-step only writes to graphs that support user supplied ids for its elements. Moreover,
-if no graph is specified via `withSideEffect()`, then <<tinkergraph-gremlin,TinkerGraph>> is assumed.
-
-[[sum-step]]
-Sum Step
-~~~~~~~~
-
-The `sum()`-step (*map*) operates on a stream of numbers and sums the numbers together to yield a double. Note that
-the current traverser number is multiplied by the traverser bulk to determine how many such numbers are being
-represented.
-
-[gremlin-groovy,modern]
-----
-g.V().values('age').sum()
-g.V().repeat(both()).times(3).values('age').sum()
-----
-
-IMPORTANT: `sum(local)` determines the sum of the current, local object (not the objects in the traversal stream).
-This works for `Collection`-type objects. For any other object, a sum of `Double.NaN` is returned.
-
-[[tail-step]]
-Tail Step
-~~~~~~~~~
-
-image::tail-step.png[width=530]
-
-The `tail()`-step is analogous to <<limit-step,`limit()`>>-step, except that it emits the last `n`-objects instead of
-the first `n`-objects.
-
-[gremlin-groovy,modern]
-----
-g.V().values('name').order()
-g.V().values('name').order().tail() <1>
-g.V().values('name').order().tail(1) <2>
-g.V().values('name').order().tail(3) <3>
-----
-
-<1> Last name (alphabetically).
-<2> Same as statement 1.
-<3> Last three names.
-
-The `tail()`-step can also be applied with `Scope.local`, in which case it operates on the incoming collection.
-
-[gremlin-groovy,modern]
-----
-g.V().as('a').out().as('a').out().as('a').select('a').by(tail(local)).values('name') <1>
-g.V().as('a').out().as('a').out().as('a').select('a').by(unfold().values('name').fold()).tail(local) <2>
-g.V().as('a').out().as('a').out().as('a').select('a').by(unfold().values('name').fold()).tail(local, 2) <3>
-g.V().valueMap().tail(local) <4>
-----
-
-<1> Only the most recent name from the "a" step (`List<Vertex>` becomes `Vertex`).
-<2> Same result as statement 1 (`List<String>` becomes `String`).
-<3> `List<String>` for each path containing the last two names from the 'a' step.
-<4> `Map<String, Object>` for each vertex, but containing only the last property value.
-
-[[timelimit-step]]
-TimeLimit Step
-~~~~~~~~~~~~~~
-
-In many situations, a graph traversal is not about getting an exact answer as its about getting a relative ranking.
-A classic example is link:http://en.wikipedia.org/wiki/Recommender_system[recommendation]. What is desired is a
-relative ranking of vertices, not their absolute rank. Next, it may be desirable to have the traversal execute for
-no more than 2 milliseconds. In such situations, `timeLimit()`-step (*filter*) can be used.
-
-image::timelimit-step.png[width=400]
-
-NOTE: The method `clock(int runs, Closure code)` is a utility preloaded in the <<gremlin-console,Gremlin Console>>
-that can be used to time execution of a body of code.
-
-[gremlin-groovy,modern]
-----
-g.V().repeat(both().groupCount('m')).times(16).cap('m').order(local).by(valueDecr).next()
-clock(1) {g.V().repeat(both().groupCount('m')).times(16).cap('m').order(local).by(valueDecr).next()}
-g.V().repeat(timeLimit(2).both().groupCount('m')).times(16).cap('m').order(local).by(valueDecr).next()
-clock(1) {g.V().repeat(timeLimit(2).both().groupCount('m')).times(16).cap('m').order(local).by(valueDecr).next()}
-----
-
-In essence, the relative order is respected, even through the number of traversers at each vertex is not. The primary
-benefit being that the calculation is guaranteed to complete at the specified time limit (in milliseconds). Finally,
-note that the internal clock of `timeLimit()`-step starts when the first traverser enters it. When the time limit is
-reached, any `next()` evaluation of the step will yield a `NoSuchElementException` and any `hasNext()` evaluation will
-yield `false`.
-
-[[tree-step]]
-Tree Step
-~~~~~~~~~
-
-From any one element (i.e. vertex or edge), the emanating paths from that element can be aggregated to form a
-link:http://en.wikipedia.org/wiki/Tree_(data_structure)[tree]. Gremlin provides `tree()`-step (*sideEffect*) for such
-this situation.
-
-image::tree-step.png[width=450]
-
-[gremlin-groovy,modern]
-----
-tree = g.V().out().out().tree().next()
-----
-
-It is important to see how the paths of all the emanating traversers are united to form the tree.
-
-image::tree-step2.png[width=500]
-
-The resultant tree data structure can then be manipulated (see `Tree` JavaDoc).
-
-[gremlin-groovy,modern]
-----
-tree = g.V().out().out().tree().by('name').next()
-tree['marko']
-tree['marko']['josh']
-tree.getObjectsAtDepth(3)
-----
-
-[[unfold-step]]
-Unfold Step
-~~~~~~~~~~~
-
-If the object reaching `unfold()` (*flatMap*) is an iterator, iterable, or map, then it is unrolled into a linear
-form. If not, then the object is simply emitted. Please see <<fold-step,`fold()`>> step for the inverse behavior.
-
-[gremlin-groovy,modern]
-----
-g.V(1).out().fold().inject('gremlin',[1.23,2.34])
-g.V(1).out().fold().inject('gremlin',[1.23,2.34]).unfold()
-----
-
-Note that `unfold()` does not recursively unroll iterators. Instead, `repeat()` can be used to for recursive unrolling.
-
-[gremlin-groovy,modern]
-----
-inject(1,[2,3,[4,5,[6]]])
-inject(1,[2,3,[4,5,[6]]]).unfold()
-inject(1,[2,3,[4,5,[6]]]).repeat(unfold()).until(count(local).is(1)).unfold()
-----
-
-[[union-step]]
-Union Step
-~~~~~~~~~~
-
-image::union-step.png[width=650]
-
-The `union()`-step (*branch*) supports the merging of the results of an arbitrary number of traversals. When a
-traverser reaches a `union()`-step, it is copied to each of its internal steps. The traversers emitted from `union()`
-are the outputs of the respective internal traversals.
-
-[gremlin-groovy,modern]
-----
-g.V(4).union(
- __.in().values('age'),
- out().values('lang'))
-g.V(4).union(
- __.in().values('age'),
- out().values('lang')).path()
-----
-
-[[valuemap-step]]
-ValueMap Step
-~~~~~~~~~~~~~
-
-The `valueMap()`-step yields a Map representation of the properties of an element.
-
-[gremlin-groovy,modern]
-----
-g.V().valueMap()
-g.V().valueMap('age')
-g.V().valueMap('age','blah')
-g.E().valueMap()
-----
-
-It is important to note that the map of a vertex maintains a list of values for each key. The map of an edge or
-vertex-property represents a single property (not a list). The reason is that vertices in TinkerPop3 leverage
-<<vertex-properties,vertex properties>> which are support multiple values per key. Using the <<the-crew-toy-graph,
-"The Crew">> toy graph, the point is made explicit.
-
-[gremlin-groovy,theCrew]
-----
-g.V().valueMap()
-g.V().has('name','marko').properties('location')
-g.V().has('name','marko').properties('location').valueMap()
-----
-
-If the `id`, `label`, `key`, and `value` of the `Element` is desired, then a boolean triggers its insertion into the
-returned map.
-
-[gremlin-groovy,theCrew]
-----
-g.V().hasLabel('person').valueMap(true)
-g.V().hasLabel('person').valueMap(true,'name')
-g.V().hasLabel('person').properties('location').valueMap(true)
-----
-
-[[vertex-steps]]
-Vertex Steps
-~~~~~~~~~~~~
-
-image::vertex-steps.png[width=350]
-
-The vertex steps (*flatMap*) are fundamental to the Gremlin language. Via these steps, its possible to "move" on the
-graph -- i.e. traverse.
-
-* `out(string...)`: Move to the outgoing adjacent vertices given the edge labels.
-* `in(string...)`: Move to the incoming adjacent vertices given the edge labels.
-* `both(string...)`: Move to both the incoming and outgoing adjacent vertices given the edge labels.
-* `outE(string...)`: Move to the outgoing incident edges given the edge labels.
-* `inE(string...)`: Move to the incoming incident edges given the edge labels.
-* `bothE(string...)`: Move to both the incoming and outgoing incident edges given the edge labels.
-* `outV()`: Move to the outgoing vertex.
-* `inV()`: Move to the incoming vertex.
-* `bothV()`: Move to both vertices.
-* `otherV()` : Move to the vertex that was not the vertex that was moved from.
-
-[gremlin-groovy,modern]
-----
-g.V(4)
-g.V(4).outE() <1>
-g.V(4).inE('knows') <2>
-g.V(4).inE('created') <3>
-g.V(4).bothE('knows','created','blah')
-g.V(4).bothE('knows','created','blah').otherV()
-g.V(4).both('knows','created','blah')
-g.V(4).outE().inV() <4>
-g.V(4).out() <5>
-g.V(4).inE().outV()
-g.V(4).inE().bothV()
-----
-
-<1> All outgoing edges.
-<2> All incoming knows-edges.
-<3> All incoming created-edges.
-<4> Moving forward touching edges and vertices.
-<5> Moving forward only touching vertices.
-
-[[where-step]]
-Where Step
-~~~~~~~~~~
-
-The `where()`-step filters the current object based on either the object itself (`Scope.local`) or the path history
-of the object (`Scope.global`) (*filter*). This step is typically used in conjuction with either
-<<match-step,`match()`>>-step or <<select-step,`select()`>>-step, but can be used in isolation.
-
-[gremlin-groovy,modern]
-----
-g.V(1).as('a').out('created').in('created').where(neq('a')) <1>
-g.withSideEffect('a',['josh','peter']).V(1).out('created').in('created').values('name').where(within('a')) <2>
-g.V(1).out('created').in('created').where(out('created').count().is(gt(1))).values('name') <3>
-----
-
-<1> Who are marko's collaborators, where marko can not be his own collaborator? (predicate)
-<2> Of the co-creators of marko, only keep those whose name is josh or peter. (using a sideEffect)
-<3> Which of marko's collaborators have worked on more than 1 project? (using a traversal)
-
-IMPORTANT: Please see <<using-where-with-match,`match().where()`>> and <<using-where-with-select,`select().where()`>>
-for how `where()` can be used in conjunction with `Map<String,Object>` projecting steps -- i.e. `Scope.local`.
-
-A few more examples of filtering an arbitrary object based on a anonymous traversal is provided below.
-
-[gremlin-groovy,modern]
-----
-g.V().where(out('created')).values('name') <1>
-g.V().out('knows').where(out('created')).values('name') <2>
-g.V().where(out('created').count().is(gte(2))).values('name') <3>
-g.V().where(out('knows').where(out('created'))).values('name') <4>
-g.V().where(__.not(out('created'))).where(__.in('knows')).values('name') <5>
-g.V().where(__.not(out('created')).and().in('knows')).values('name') <6>
-----
-
-<1> What are the names of the people who have created a project?
-<2> What are the names of the people that are known by someone one and have created a project?
-<3> What are the names of the people how have created two or more projects?
-<4> What are the names of the people who know someone that has created a project? (This only works in OLTP -- see the `WARNING` below)
-<5> What are the names of the people who have not created anything, but are known by someone?
-<6> The concatenation of `where()`-steps is the same as a single `where()`-step with an and'd clause.
-
-WARNING: The anonymous traversal of `where()` processes the current object "locally". In OLAP, where the atomic unit
-of computing is the the vertex and its local "star graph," it is important that the anonymous traversal does not leave
-the confines of the vertex's star graph. In other words, it can not traverse to an adjacent vertex's properties or
-edges. Note that is only a temporary limitation that will be addressed in a future version of TinkerPop3 (see
-link:https://issues.apache.org/jira/browse/TINKERPOP3-693[JIRA#693]).
-
-[[a-note-on-predicates]]
-A Note on Predicates
---------------------
-
-A `P` is a predicate of the form `Function<Object,Boolean>`. That is, given some object, return true or false. The
-provided predicates are outlined in the table below and are used in various steps such as <<has-step,`has()`>>-step,
-<<where-step,`where()`>>-step, <<is-step,`is()`>>-step, etc.
-
-[width="100%",cols="3,15",options="header"]
-|=========================================================
-| Predicate | Description
-| `eq(object)` | Is the incoming object equal to the provided object?
-| `neq(object)` | Is the incoming object not equal to the provided object?
-| `lt(number)` | Is the incoming number less than the provided number?
-| `lte(number)` | Is the incoming number less than or equal to the provided number?
-| `gt(number)` | Is the incoming number greater than the provided number?
-| `gte(number)` | Is the incoming number greater than or equal to the provided number?
-| `inside(number,number)` | Is the incoming number greater than the first provided number and less than the second?
-| `outside(number,number)` | Is the incoming number less than the first provided number and greater than the second?
-| `between(number,number)` | Is the incoming number greater than or equal to the first provided number and less than the second?
-| `within(objects...)` | Is the incoming object in the array of provided objects?
-| `without(objects...)` | Is the incoming object not in the array of the provided objects?
-|=========================================================
-
-[gremlin-groovy]
-----
-eq(2)
-not(neq(2)) <1>
-not(within('a','b','c'))
-not(within('a','b','c')).test('d') <2>
-not(within('a','b','c')).test('a')
-within(1,2,3).and(not(eq(2))).test(3) <3>
-inside(1,4).or(eq(5)).test(3) <4>
-inside(1,4).or(eq(5)).test(5)
-between(1,2) <5>
-not(between(1,2))
-----
-
-<1> The `not()` of a `P`-predicate is another `P`-predicate.
-<2> `P`-predicates are arguments to various steps which internally `test()` the incoming value.
-<3> `P`-predicates can be and'd together.
-<4> `P`-predicates can be or' together.
-<5> `and()` is a `P`-predicate and thus, a `P`-predicate can be composed of multiple `P`-predicates.
-
-Finally, note that <<where-step,`where()`>>-step takes a `P<String>`. The provided string value refers to a variable
-binding, not to the explicit string value.
-
-[gremlin-groovy,modern]
-----
-g.V().as('a').both().both().as('b').count()
-g.V().as('a').both().both().as('b').where('a',neq('b')).count()
-----
-
-NOTE: It is possible for graph system providers and users to extend `P` and provide new predicates. For instance, a
-`regex(pattern)` could be a graph system specific `P`.
-
-[[a-note-on-barrier-steps]]
-A Note on Barrier Steps
------------------------
-
-image:barrier.png[width=165,float=right] Gremlin is primarily a
-link:http://en.wikipedia.org/wiki/Lazy_evaluation[lazy], stream processing language. This means that Gremlin fully
-processes (to the best of its abilities) any traversers currently in the traversal pipeline before getting more data
-from the start/head of the traversal. However, there are numerous situations in which a completely lazy computation
-is not possible (or impractical). When a computation is not lazy, a "barrier step" exists. There are three types of
-barriers:
-
-. `Collecti
<TRUNCATED>
[05/22] incubator-tinkerpop git commit: Made subdirectories for
various "books" in the docs.
Posted by sp...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/the-graph.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/the-graph.asciidoc b/docs/src/the-graph.asciidoc
deleted file mode 100644
index eceec21..0000000
--- a/docs/src/the-graph.asciidoc
+++ /dev/null
@@ -1,771 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-[[graph]]
-The Graph
-=========
-
-image::gremlin-standing.png[width=125]
-
-Features
---------
-
-A `Feature` implementation describes the capabilities of a `Graph` instance. This interface is implemented by graph
-system providers for two purposes:
-
-. It tells users the capabilities of their `Graph` instance.
-. It allows the features they do comply with to be tested against the Gremlin Test Suite - tests that do not comply are "ignored").
-
-The following example in the Gremlin Console shows how to print all the features of a `Graph`:
-
-[gremlin-groovy]
-----
-graph = TinkerGraph.open()
-graph.features()
-----
-
-A common pattern for using features is to check their support prior to performing an operation:
-
-[gremlin-groovy]
-----
-graph.features().graph().supportsTransactions()
-graph.features().graph().supportsTransactions() ? g.tx().commit() : "no tx"
-----
-
-TIP: To ensure provider agnostic code, always check feature support prior to usage of a particular function. In that
-way, the application can behave gracefully in case a particular implementation is provided at runtime that does not
-support a function being accessed.
-
-WARNING: Assignments of a `GraphStrategy` can alter the base features of a `Graph` in dynamic ways, such that checks
-against a `Feature` may not always reflect the behavior exhibited when the `GraphStrategy` is in use.
-
-[[vertex-properties]]
-Vertex Properties
------------------
-
-image:vertex-properties.png[width=215,float=left] TinkerPop3 introduces the concept of a `VertexProperty<V>`. All the
-properties of a `Vertex` are a `VertexProperty`. A `VertexProperty` implements `Property` and as such, it has a
-key/value pair. However, `VertexProperty` also implements `Element` and thus, can have a collection of key/value
-pairs. Moreover, while an `Edge` can only have one property of key "name" (for example), a `Vertex` can have multiple
-"name" properties. With the inclusion of vertex properties, two features are introduced which ultimately advance the
-graph modelers toolkit:
-
-. Multiple properties (*multi-properties*): a vertex property key can have multiple values (i.e. a vertex can have
-multiple "name" properties).
-. Properties on properties (*meta-properties*): a vertex property can have properties (i.e. a vertex property can
-have key/value data associated with it).
-
-A collection of use cases are itemized below:
-
-. *Permissions*: Vertex properties can have key/value ACL-type permission information associated with them.
-. *Auditing*: When a vertex property is manipulated, it can have key/value information attached to it saying who the
-creator, deletor, etc. are.
-. *Provenance*: The "name" of a vertex can be declared by multiple users.
-
-A running example using vertex properties is provided below to demonstrate and explain the API.
-
-[gremlin-groovy]
-----
-graph = TinkerGraph.open()
-g = graph.traversal(standard())
-v = g.addV('name','marko','name','marko a. rodriguez').next()
-g.V(v).properties().count()
-g.V(v).properties('name').count() <1>
-g.V(v).properties()
-g.V(v).properties('name')
-g.V(v).properties('name').hasValue('marko')
-g.V(v).properties('name').hasValue('marko').property('acl','private') <2>
-g.V(v).properties('name').hasValue('marko a. rodriguez')
-g.V(v).properties('name').hasValue('marko a. rodriguez').property('acl','public')
-g.V(v).properties('name').has('acl','public').value()
-g.V(v).properties('name').has('acl','public').drop() <3>
-g.V(v).properties('name').has('acl','public').value()
-g.V(v).properties('name').has('acl','private').value()
-g.V(v).properties()
-g.V(v).properties().properties() <4>
-g.V(v).properties().property('date',2014) <5>
-g.V(v).properties().property('creator','stephen')
-g.V(v).properties().properties()
-g.V(v).properties('name').valueMap()
-g.V(v).property('name','okram') <6>
-g.V(v).properties('name')
-g.V(v).values('name') <7>
-----
-
-<1> A vertex can have zero or more properties with the same key associated with it.
-<2> A vertex property can have standard key/value properties attached to it.
-<3> Vertex property removal is identical to property removal.
-<4> It is property to get the properties of a vertex property.
-<5> A vertex property can have any number of key/value properties attached to it.
-<6> `property(...)` will remove all existing key'd properties before adding the new single property (see `VertexProperty.Cardinality`).
-<7> If only the value of a property is needed, then `values()` can be used.
-
-If the concept of vertex properties is difficult to grasp, then it may be best to think of vertex properties in terms
-of "literal vertices." A vertex can have an edge to a "literal vertex" that has a single value key/value -- e.g.
-"value=okram." The edge that points to that literal vertex has an edge-label of "name." The properties on the edge
-represent the literal vertex's properties. The "literal vertex" can not have any other edges to it (only one from the
-associated vertex).
-
-[[the-crew-toy-graph]]
-TIP: A toy graph demonstrating all of the new TinkerPop3 graph structure features is available at
-`TinkerFactory.createTheCrew()` and `data/tinkerpop-crew*`. This graph demonstrates multi-properties and meta-properties.
-
-.TinkerPop Crew
-image::the-crew-graph.png[width=685]
-
-[gremlin-groovy,theCrew]
-----
-g.V().as('a').
- properties('location').as('b').
- hasNot('endTime').as('c').
- select('a','b','c').by('name').by(value).by('startTime') // determine the current location of each person
-g.V().has('name','gremlin').inE('uses').
- order().by('skill',incr).as('a').
- outV().as('b').
- select('a','b').by('skill').by('name') // rank the users of gremlin by their skill level
-----
-
-Graph Variables
----------------
-
-TinkerPop3 introduces the concept of `Graph.Variables`. Variables are key/value pairs associated with the graph
-itself -- in essence, a `Map<String,Object>`. These variables are intended to store metadata about the graph. Example
-use cases include:
-
- * *Schema information*: What do the namespace prefixes resolve to and when was the schema last modified?
- * *Global permissions*: What are the access rights for particular groups?
- * *System user information*: Who are the admins of the system?
-
-An example of graph variables in use is presented below:
-
-[gremlin-groovy]
-----
-graph = TinkerGraph.open()
-graph.variables()
-graph.variables().set('systemAdmins',['stephen','peter','pavel'])
-graph.variables().set('systemUsers',['matthias','marko','josh'])
-graph.variables().keys()
-graph.variables().get('systemUsers')
-graph.variables().get('systemUsers').get()
-graph.variables().remove('systemAdmins')
-graph.variables().keys()
-----
-
-IMPORTANT: Graph variables are not intended to be subject to heavy, concurrent mutation nor to be used in complex
-computations. The intention is to have a location to store data about the graph for administrative purposes.
-
-[[transactions]]
-Graph Transactions
-------------------
-
-image:gremlin-coins.png[width=100,float=right] A link:http://en.wikipedia.org/wiki/Database_transaction[database transaction]
-represents a unit of work to execute against the database. Transactions are controlled by an implementation of the
-`Transaction` interface and that object can be obtained from the `Graph` interface using the `tx()` method. It is
-important to note that the `Transaction` object does not represent a "transaction" itself. It merely exposes the
-methods for working with transactions (e.g. committing, rolling back, etc).
-
-Most `Graph` implementations that `supportsTransactions` will implement an "automatic" `ThreadLocal` transaction,
-which means that when a read or write occurs after the `Graph` is instantiated a transaction is automatically
-started within that thread. There is no need to manually call a method to "create" or "start" a transaction. Simply
-modify the graph as required and call `graph.tx().commit()` to apply changes or `graph.tx().rollback()` to undo them.
-When the next read or write action occurs against the graph, a new transaction will be started within that current
-thread of execution.
-
-When using transactions in this fashion, especially in web application (e.g. REST server), it is important to ensure
-that transaction do not leak from one request to the next. In other words, unless a client is somehow bound via
-session to process every request on the same server thread, ever request must be committed or rolled back at the end
-of the request. By ensuring that the request encapsulates a transaction, it ensures that a future request processed
-on a server thread is starting in a fresh transactional state and will not have access to the remains of one from an
-earlier request. A good strategy is to rollback a transaction at the start of a request, so that if it so happens that
-a transactional leak does occur between requests somehow, a fresh transaction is assured by the fresh request.
-
-TIP: The `tx()` method is on the `Graph` interface, but it is also available on the `TraversalSource` spawned from a
-`Graph`. Calls to `TraversalSource.tx()` are proxied through to the underlying `Graph` as a convenience.
-
-Configuring
-~~~~~~~~~~~
-
-Determining when a transaction starts is dependent upon the behavior assigned to the `Transaction`. It is up to the
-`Graph` implementation to determine the default behavior and unless the implementation doesn't allow it, the behavior
-itself can be altered via these `Transaction` methods:
-
-[source,java]
-----
-public Transaction onReadWrite(final Consumer<Transaction> consumer);
-
-public Transaction onClose(final Consumer<Transaction> consumer);
-----
-
-Providing a `Consumer` function to `onReadWrite` allows definition of how a transaction starts when a read or a write
-occurs. `Transaction.READ_WRITE_BEHAVIOR` contains pre-defined `Consumer` functions to supply to the `onReadWrite`
-method. It has two options:
-
-* `AUTO` - automatic transactions where the transaction is started implicitly to the read or write operation
-* `MANUAL` - manual transactions where it is up to the user to explicitly open a transaction, throwing an exception
-if the transaction is not open
-
-Providing a `Consumer` function to `onClose` allows configuration of how a transaction is handled when
-`Transaction.close()` is called. `Transaction.CLOSE_BEHAVIOR` has several pre-defined options that can be supplied to
-this method:
-
-* `COMMIT` - automatically commit an open transaction
-* `ROLLBACK` - automatically rollback an open transaction
-* `MANUAL` - throw an exception if a transaction is open, forcing the user to explicitly close the transaction
-
-IMPORTANT: As transactions are `ThreadLocal` in nature, so are the transaction configurations for `onReadWrite` and
-`onClose`.
-
-Once there is an understanding for how transactions are configured, most of the rest of the `Transaction` interface
-is self-explanatory. Note that <<neo4j-gremlin,Neo4j-Gremlin>> is used for the examples to follow as TinkerGraph does
-not support transactions.
-
-[source,groovy]
-----
-gremlin> graph = Neo4jGraph.open('/tmp/neo4j')
-==>neo4jgraph[EmbeddedGraphDatabase [/tmp/neo4j]]
-gremlin> graph.features()
-==>FEATURES
-> GraphFeatures
->-- Transactions: true <1>
->-- Computer: false
->-- Persistence: true
-...
-gremlin> graph.tx().onReadWrite(Transaction.READ_WRITE_BEHAVIOR.AUTO) <2>
-==>org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jGraph$Neo4jTransaction@1c067c0d
-gremlin> graph.addVertex("name","stephen") <3>
-==>v[0]
-gremlin> graph.tx().commit() <4>
-==>null
-gremlin> graph.tx().onReadWrite(Transaction.READ_WRITE_BEHAVIOR.MANUAL) <5>
-==>org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jGraph$Neo4jTransaction@1c067c0d
-gremlin> graph.tx().isOpen()
-==>false
-gremlin> graph.addVertex("name","marko") <6>
-Open a transaction before attempting to read/write the transaction
-gremlin> graph.tx().open() <7>
-==>null
-gremlin> graph.addVertex("name","marko") <8>
-==>v[1]
-gremlin> graph.tx().commit()
-==>null
-----
-
-<1> Check `features` to ensure that the graph supports transactions.
-<2> By default, `Neo4jGraph` is configured with "automatic" transactions, so it is set here for demonstration purposes only.
-<3> When the vertex is added, the transaction is automatically started. From this point, more mutations can be staged
-or other read operations executed in the context of that open transaction.
-<4> Calling `commit` finalizes the transaction.
-<5> Change transaction behavior to require manual control.
-<6> Adding a vertex now results in failure because the transaction was not explicitly opened.
-<7> Explicitly open a transaction.
-<8> Adding a vertex now succeeds as the transaction was manually opened.
-
-NOTE: It may be important to consult the documentation of the `Graph` implementation when it comes to the specifics of
-how transactions will behave. TinkerPop allows some latitude in this area and implementations may not have the exact
-same behaviors and link:https://en.wikipedia.org/wiki/ACID[ACID] guarantees.
-
-Retries
-~~~~~~~
-
-There are times when transactions fail. Failure may be indicative of some permanent condition, but other failures
-might simply require the transaction to be retried for possible future success. The `Transaction` object also exposes
-a method for executing automatic transaction retries:
-
-[gremlin-groovy]
-----
-graph = Neo4jGraph.open('/tmp/neo4j')
-graph.tx().submit {it.addVertex("name","josh")}.retry(10)
-graph.tx().submit {it.addVertex("name","daniel")}.exponentialBackoff(10)
-graph.close()
-----
-
-As shown above, the `submit` method takes a `Function<Graph, R>` which is the unit of work to execute and possibly
-retry on failure. The method returns a `Transaction.Workload` object which has a number of default methods for common
-retry strategies. It is also possible to supply a custom retry function if a default one does not suit the required
-purpose.
-
-Threaded Transactions
-~~~~~~~~~~~~~~~~~~~~~
-
-Most `Graph` implementations that support transactions do so in a `ThreadLocal` manner, where the current transaction
-is bound to the current thread of execution. Consider the following example to demonstrate:
-
-[source,java]
-----
-graph.addVertex("name","stephen");
-
-Thread t1 = new Thread(() -> {
- graph.addVertex("name","josh");
-});
-
-Thread t2 = new Thread(() -> {
- graph.addVertex("name","marko");
-});
-
-t1.start()
-t2.start()
-
-t1.join()
-t2.join()
-
-graph.tx().commit();
-----
-
-The above code shows three vertices added to `graph` in three different threads: the current thread, `t1` and
-`t2`. One might expect that by the time this body of code finished executing, that there would be three vertices
-persisted to the `Graph`. However, given the `ThreadLocal` nature of transactions, there really were three separate
-transactions created in that body of code (i.e. one for each thread of execution) and the only one committed was the
-first call to `addVertex` in the primary thread of execution. The other two calls to that method within `t1` and `t2`
-were never committed and thus orphaned.
-
-A `Graph` that `supportsThreadedTransactions` is one that allows for a `Graph` to operate outside of that constraint,
-thus allowing multiple threads to operate within the same transaction. Therefore, if there was a need to have three
-different threads operating within the same transaction, the above code could be re-written as follows:
-
-[source,java]
-----
-Graph threaded = graph.tx().newThreadedTx();
-threaded.addVertex("name","stephen");
-
-Thread t1 = new Thread(() -> {
- threaded.addVertex("name","josh");
-});
-
-Thread t2 = new Thread(() -> {
- threaded.addVertex("name","marko");
-});
-
-t1.start()
-t2.start()
-
-t1.join()
-t2.join()
-
-threaded.tx().commit();
-----
-
-In the above case, the call to `graph.tx().newThreadedTx()` creates a new `Graph` instance that is unbound from the
-`ThreadLocal` transaction, thus allowing each thread to operate on it in the same context. In this case, there would
-be three separate vertices persisted to the `Graph`.
-
-Gremlin I/O
------------
-
-image:gremlin-io.png[width=250,float=right] The task of getting data in and out of `Graph` instances is the job of
-the Gremlin I/O packages. Gremlin I/O provides two interfaces for reading and writing `Graph` instances: `GraphReader`
-and `GraphWriter`. These interfaces expose methods that support:
-
-* Reading and writing an entire `Graph`
-* Reading and writing a `Traversal<Vertex>` as adjacency list format
-* Reading and writing a single `Vertex` (with and without associated `Edge` objects)
-* Reading and writing a single `Edge`
-* Reading and writing a single `VertexProperty`
-* Reading and writing a single `Property`
-* Reading and writing an arbitrary `Object`
-
-In all cases, these methods operate in the currency of `InputStream` and `OutputStream` objects, allowing graphs and
-their related elements to be written to and read from files, byte arrays, etc. The `Graph` interface offers the `io`
-method, which provides access to "reader/writer builder" objects that are pre-configured with serializers provided by
-the `Graph`, as well as helper methods for the various I/O capabilities. Unless there are very advanced requirements
-for the serialization process, it is always best to utilize the methods on the `Io` interface to construct
-`GraphReader` and `GraphWriter` instances, as the implementation may provide some custom settings that would otherwise
-have to be configured manually by the user to do the serialization.
-
-It is up to the implementations of the `GraphReader` and `GraphWriter` interfaces to choose the methods they
-implement and the manner in which they work together. The only semantics enforced and expected is that the write
-methods should produce output that is compatible with the corresponding read method (e.g. the output of
-`writeVertices` should be readable as input to `readVertices` and the output of `writeProperty` should be readable as
-input to `readProperty`).
-
-GraphML Reader/Writer
-~~~~~~~~~~~~~~~~~~~~~
-
-image:gremlin-graphml.png[width=350,float=left] The link:http://graphml.graphdrawing.org/[GraphML] file format is a
-common XML-based representation of a graph. It is widely supported by graph-related tools and libraries making it a
-solid interchange format for TinkerPop. In other words, if the intent is to work with graph data in conjunction with
-applications outside of TinkerPop, GraphML may be the best choice to do that. Common use cases might be:
-
-* Generate a graph using link:https://networkx.github.io/[NetworkX], export it with GraphML and import it to TinkerPop.
-* Produce a subgraph and export it to GraphML to be consumed by and visualized in link:https://gephi.org/[Gephi].
-* Migrate the data of an entire graph to a different graph database not supported by TinkerPop.
-
-As GraphML is a specification for the serialization of an entire graph and not the individual elements of a graph,
-methods that support input and output of single vertices, edges, etc. are not supported.
-
-CAUTION: GraphML is a "lossy" format in that it only supports primitive values for properties and does not have
-support for `Graph` variables. It will use `toString` to serialize property values outside of those primitives.
-
-CAUTION: GraphML, as a specification, allows for `<edge>` and `<node>` elements to appear in any order. The
-`GraphMLReader` will support that, however, that capability comes with a limitation. TinkerPop does not allow the
-vertex label to be changed after the vertex has been created. Therefore, if an `<edge>` element comes before the
-`<node>` the label on the vertex will be ignored. It is thus better to order `<node>` elements in the GraphML to
-appear before all `<edge>` elements if vertex labels are important to the graph.
-
-The following code shows how to write a `Graph` instance to file called `tinkerpop-modern.xml` and then how to read
-that file back into a different instance:
-
-[source,java]
-----
-final Graph graph = TinkerFactory.createModern();
-graph.io(IoCore.graphml()).writeGraph("tinkerpop-modern.xml");
-final Graph newGraph = TinkerGraph.open();
-newGraph.io(IoCore.graphml()).readGraph("tinkerpop-modern.xml");
-----
-
-If a custom configuration is required, then have the `Graph` generate a `GraphReader` or `GraphWriter` "builder" instance:
-
-[source,java]
-----
-final Graph graph = TinkerFactory.createModern();
-try (final OutputStream os = new FileOutputStream("tinkerpop-modern.xml")) {
- graph.io(IoCore.graphml()).writer().normalize(true).create().writeGraph(os, graph);
-}
-
-final Graph newGraph = TinkerGraph.open();
-try (final InputStream stream = new FileInputStream("tinkerpop-modern.xml")) {
- newGraph.io(IoCore.graphml()).reader().vertexIdKey("name").create().readGraph(stream, newGraph);
-}
-----
-
-[[graphson-reader-writer]]
-GraphSON Reader/Writer
-~~~~~~~~~~~~~~~~~~~~~~
-
-image:gremlin-graphson.png[width=350,float=left] GraphSON is a link:http://json.org/[JSON]-based format extended
-from earlier versions of TinkerPop. It is important to note that TinkerPop3's GraphSON is not backwards compatible
-with prior TinkerPop GraphSON versions. GraphSON has some support from graph-related application outside of TinkerPop,
-but it is generally best used in two cases:
-
-* A text format of the graph or its elements is desired (e.g. debugging, usage in source control, etc.)
-* The graph or its elements need to be consumed by code that is not JVM-based (e.g. JavaScript, Python, .NET, etc.)
-
-GraphSON supports all of the `GraphReader` and `GraphWriter` interface methods and can therefore read or write an
-entire `Graph`, vertices, arbitrary objects, etc. The following code shows how to write a `Graph` instance to file
-called `tinkerpop-modern.json` and then how to read that file back into a different instance:
-
-[source,java]
-----
-final Graph graph = TinkerFactory.createModern();
-graph.io(IoCore.graphson()).writeGraph("tinkerpop-modern.json");
-
-final Graph newGraph = TinkerGraph.open();
-newGraph.io(IoCore.graphson()).readGraph("tinkerpop-modern.json");
-----
-
-If a custom configuration is required, then have the `Graph` generate a `GraphReader` or `GraphWriter` "builder" instance:
-
-[source,java]
-----
-final Graph graph = TinkerFactory.createModern();
-try (final OutputStream os = new FileOutputStream("tinkerpop-modern.json")) {
- final GraphSONMapper mapper = graph.io(IoCore.graphson()).mapper().normalize(true).create()
- graph.io(IoCore.graphson()).writer().mapper(mapper).create().writeGraph(os, graph)
-}
-
-final Graph newGraph = TinkerGraph.open();
-try (final InputStream stream = new FileInputStream("tinkerpop-modern.json")) {
- newGraph.io(IoCore.graphson()).reader().vertexIdKey("name").create().readGraph(stream, newGraph);
-}
-----
-
-One of the important configuration options of the `GraphSONReader` and `GraphSONWriter` is the ability to embed type
-information into the output. By embedding the types, it becomes possible to serialize a graph without losing type
-information that might be important when being consumed by another source. The importance of this concept is
-demonstrated in the following example where a single `Vertex` is written to GraphSON using the Gremlin Console:
-
-[gremlin-groovy]
-----
-graph = TinkerFactory.createModern()
-g = graph.traversal()
-f = new FileOutputStream("vertex-1.json")
-graph.io(graphson()).writer().create().writeVertex(f, g.V(1).next(), BOTH)
-f.close()
-----
-
-The following GraphSON example shows the output of `GraphSonWriter.writeVertex()` with associated edges:
-
-[source,json]
-----
-{
- "id": 1,
- "label": "person",
- "outE": {
- "created": [
- {
- "id": 9,
- "inV": 3,
- "properties": {
- "weight": 0.4
- }
- }
- ],
- "knows": [
- {
- "id": 7,
- "inV": 2,
- "properties": {
- "weight": 0.5
- }
- },
- {
- "id": 8,
- "inV": 4,
- "properties": {
- "weight": 1
- }
- }
- ]
- },
- "properties": {
- "name": [
- {
- "id": 0,
- "value": "marko"
- }
- ],
- "age": [
- {
- "id": 1,
- "value": 29
- }
- ]
- }
-}
-----
-
-The vertex properly serializes to valid JSON but note that a consuming application will not automatically know how to
-interpret the numeric values. In coercing those Java values to JSON, such information is lost.
-
-With a minor change to the construction of the `GraphSONWriter` the lossy nature of GraphSON can be avoided:
-
-[gremlin-groovy]
-----
-graph = TinkerFactory.createModern()
-g = graph.traversal()
-f = new FileOutputStream("vertex-1.json")
-mapper = graph.io(graphson()).mapper().embedTypes(true).create()
-graph.io(graphson()).writer().mapper(mapper).create().writeVertex(f, g.V(1).next(), BOTH)
-f.close()
-----
-
-In the above code, the `embedTypes` option is set to `true` and the output below shows the difference in the output:
-
-[source,json]
-----
-{
- "@class": "java.util.HashMap",
- "id": 1,
- "label": "person",
- "outE": {
- "@class": "java.util.HashMap",
- "created": [
- "java.util.ArrayList",
- [
- {
- "@class": "java.util.HashMap",
- "id": 9,
- "inV": 3,
- "properties": {
- "@class": "java.util.HashMap",
- "weight": 0.4
- }
- }
- ]
- ],
- "knows": [
- "java.util.ArrayList",
- [
- {
- "@class": "java.util.HashMap",
- "id": 7,
- "inV": 2,
- "properties": {
- "@class": "java.util.HashMap",
- "weight": 0.5
- }
- },
- {
- "@class": "java.util.HashMap",
- "id": 8,
- "inV": 4,
- "properties": {
- "@class": "java.util.HashMap",
- "weight": 1
- }
- }
- ]
- ]
- },
- "properties": {
- "@class": "java.util.HashMap",
- "name": [
- "java.util.ArrayList",
- [
- {
- "@class": "java.util.HashMap",
- "id": [
- "java.lang.Long",
- 0
- ],
- "value": "marko"
- }
- ]
- ],
- "age": [
- "java.util.ArrayList",
- [
- {
- "@class": "java.util.HashMap",
- "id": [
- "java.lang.Long",
- 1
- ],
- "value": 29
- }
- ]
- ]
- }
-}
-----
-
-The ambiguity of components of the GraphSON is now removed by the `@class` property, which contains Java class
-information for the data it is associated with. The `@class` property is used for all non-final types, with the
-exception of a small number of "natural" types (String, Boolean, Integer, and Double) which can be correctly inferred
-from JSON typing. While the output is more verbose, it comes with the security of not losing type information. While
-non-JVM languages won't be able to consume this information automatically, at least there is a hint as to how the
-values should be coerced back into the correct types in the target language.
-
-[[gryo-reader-writer]]
-Gryo Reader/Writer
-~~~~~~~~~~~~~~~~~~
-
-image:gremlin-kryo.png[width=400,float=left] link:https://github.com/EsotericSoftware/kryo[Kryo] is a popular
-serialization package for the JVM. Gremlin-Kryo is a binary `Graph` serialization format for use on the JVM by JVM
-languages. It is designed to be space efficient, non-lossy and is promoted as the standard format to use when working
-with graph data inside of the TinkerPop stack. A list of common use cases is presented below:
-
-* Migration from one Gremlin Structure implementation to another (e.g. `TinkerGraph` to `Neo4jGraph`)
-* Serialization of individual graph elements to be sent over the network to another JVM.
-* Backups of in-memory graphs or subgraphs.
-
-CAUTION: When migrating between Gremlin Structure implementations, Kryo may not lose data, but it is important to
-consider the features of each `Graph` and whether or not the data types supported in one will be supported in the
-other. Failure to do so, may result in errors.
-
-Kryo supports all of the `GraphReader` and `GraphWriter` interface methods and can therefore read or write an entire
-`Graph`, vertices, edges, etc. The following code shows how to write a `Graph` instance to file called
-`tinkerpop-modern.kryo` and then how to read that file back into a different instance:
-
-[source,java]
-----
-final Graph graph = TinkerFactory.createModern();
-graph.io(IoCore.gryo()).writeGraph("tinkerpop-modern.kryo");
-
-final Graph newGraph = TinkerGraph.open();
-newGraph.io(IoCore.gryo()).readGraph("tinkerpop-modern.kryo")'
-----
-
-If a custom configuration is required, then have the `Graph` generate a `GraphReader` or `GraphWriter` "builder" instance:
-
-[source,java]
-----
-final Graph graph = TinkerFactory.createModern();
-try (final OutputStream os = new FileOutputStream("tinkerpop-modern.kryo")) {
- graph.io(IoCore.gryo()).writer().create().writeGraph(os, graph);
-}
-
-final Graph newGraph = TinkerGraph.open();
-try (final InputStream stream = new FileInputStream("tinkerpop-modern.kryo")) {
- newGraph.io(IoCore.gryo()).reader().vertexIdKey("name").create().readGraph(stream, newGraph);
-}
-----
-
-NOTE: The preferred extension for files names produced by Gryo is `.kryo`.
-
-TinkerPop2 Data Migration
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-image:data-migration.png[width=300,float=right] For those using TinkerPop2, migrating to TinkerPop3 will mean a number
-of programming changes, but may also require a migration of the data depending on the graph implementation. For
-example, trying to open `TinkerGraph` data from TinkerPop2 with TinkerPop3 code will not work, however opening a
-TinkerPop2 `Neo4jGraph` with a TinkerPop3 `Neo4jGraph` should work provided there aren't Neo4j version compatibility
-mismatches preventing the read.
-
-If such a situation arises that a particular TinkerPop2 `Graph` can not be read by TinkerPop3, a "legacy" data
-migration approach exists. The migration involves writing the TinkerPop2 `Graph` to GraphSON, then reading it to
-TinkerPop3 with the `LegacyGraphSONReader` (a limited implementation of the `GraphReader` interface).
-
-The following represents an example migration of the "classic" toy graph. In this example, the "classic" graph is
-saved to GraphSON using TinkerPop2.
-
-[source,groovy]
-----
-gremlin> Gremlin.version()
-==>2.5.z
-gremlin> graph = TinkerGraphFactory.createTinkerGraph()
-==>tinkergraph[vertices:6 edges:6]
-gremlin> GraphSONWriter.outputGraph(graph,'/tmp/tp2.json',GraphSONMode.EXTENDED)
-==>null
-----
-
-The above console session uses the `gremlin-groovy` distribution from TinkerPop2. It is important to generate the
-`tp2.json` file using the `EXTENDED` mode as it will include data types when necessary which will help limit
-"lossiness" on the TinkerPop3 side when imported. Once `tp2.json` is created, it can then be imported to a TinkerPop3
-`Graph`.
-
-[source,groovy]
-----
-gremlin> Gremlin.version()
-==>x.y.z
-gremlin> graph = TinkerGraph.open()
-==>tinkergraph[vertices:0 edges:0]
-gremlin> r = LegacyGraphSONReader.build().create()
-==>org.apache.tinkerpop.gremlin.structure.io.graphson.LegacyGraphSONReader@64337702
-gremlin> r.readGraph(new FileInputStream('/tmp/tp2.json'), graph)
-==>null
-gremlin> g = graph.traversal(standard())
-==>graphtraversalsource[tinkergraph[vertices:6 edges:6], standard]
-gremlin> g.E()
-==>e[11][4-created->3]
-==>e[12][6-created->3]
-==>e[7][1-knows->2]
-==>e[8][1-knows->4]
-==>e[9][1-created->3]
-==>e[10][4-created->5]
-----
-
-Namespace Conventions
----------------------
-
-End users, <<implementations,graph system providers>>, <<graphcomputer,`GraphComputer`>> algorithm designers,
-<<gremlin-plugins,GremlinPlugin>> creators, etc. all leverage properties on elements to store information. There are
-a few conventions that should be respected when naming property keys to ensure that conflicts between these
-stakeholders do not conflict.
-
-* End users are granted the _flat namespace_ (e.g. `name`, `age`, `location`) to key their properties and label their elements.
-* Graph system providers are granted the _hidden namespace_ (e.g. `~metadata`) to key their properties and labels.
-Data keyed as such is only accessible via the graph system implementation and no other stakeholders are granted read
-nor write access to data prefixed with "~" (see `Graph.Hidden`). Test coverage and exceptions exist to ensure that
-graph systems respect this hard boundary.
-* <<vertexprogram,`VertexProgram`>> and <<mapreduce,`MapReduce`>> developers should, like `GraphStrategy` developers,
-leverage _qualified namespaces_ particular to their domain (e.g. `mydomain.myvertexprogram.computedata`).
-* `GremlinPlugin` creators should prefix their plugin name with their domain (e.g. `mydomain.myplugin`).
-
-IMPORTANT: TinkerPop uses `tinkerpop.` and `gremlin.` as the prefixes for provided strategies, vertex programs, map
-reduce implementations, and plugins.
-
-The only truly protected namespace is the _hidden namespace_ provided to graph systems. From there, its up to
-engineers to respect the namespacing conventions presented.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/the-graphcomputer.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/the-graphcomputer.asciidoc b/docs/src/the-graphcomputer.asciidoc
deleted file mode 100644
index af65f07..0000000
--- a/docs/src/the-graphcomputer.asciidoc
+++ /dev/null
@@ -1,475 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-[[graphcomputer]]
-The GraphComputer
-=================
-
-image:graphcomputer-puffers.png[width=350,float=right] TinkerPop3 provides two primary means of interacting with a
-graph: link:http://en.wikipedia.org/wiki/Online_transaction_processing[online transaction processing] (OLTP) and
-link:http://en.wikipedia.org/wiki/Online_analytical_processing[online analytical processing] (OLAP). OTLP-based
-graph systems allow the user to query the graph in real-time. However, typically, real-time performance is only
-possible when a local traversal is enacted. A local traversal is one that starts at a particular vertex (or small set
-of vertices) and touches a small set of connected vertices (by any arbitrary path of arbitrary length). In short, OLTP
-queries interact with a limited set of data and respond on the order of milliseconds or seconds. On the other hand,
-with OLAP graph processing, the entire graph is processed and thus, every vertex and edge is analyzed (some times
-more than once for iterative, recursive algorithms). Due to the amount of data being processed, the results are
-typically not returned in real-time and for massive graphs (i.e. graphs represented across a cluster of machines),
-results can take on the order of minutes or hours.
-
- * *OLTP*: real-time, limited data accessed, random data access, sequential processing, querying
- * *OLAP*: long running, entire data set accessed, sequential data access, parallel processing, batch processing
-
-image::oltp-vs-olap.png[width=600]
-
-The image above demonstrates the difference between Gremlin OLTP and Gremlin OLAP. With Gremlin OLTP, the graph is
-walked by moving from vertex-to-vertex via incident edges. With Gremlin OLAP, all vertices are provided a
-`VertexProgram`. The programs send messages to one another with the topological structure of the graph acting as the
-communication network (though random message passing possible). In many respects, the messages passed are like
-the OLTP traversers moving from vertex-to-vertex. However, all messages are moving independent of one another, in
-parallel. Once a vertex program is finished computing, TinkerPop3's OLAP engine supports any number
-link:http://en.wikipedia.org/wiki/MapReduce[`MapReduce`] jobs over the resultant graph.
-
-IMPORTANT: `GraphComputer` was designed from the start to be used within a multi-JVM, distributed environment --
-in other words, a multi-machine compute cluster. As such, all the computing objects must be able to be migrated
-between JVMs. The pattern promoted is to store state information in a `Configuration` object to later be regenerated
-by a loading process. It is important to realize that `VertexProgram`, `MapReduce`, and numerous particular instances
-rely heavily on the state of the computing classes (not the structure, but the processes) to be stored in a
-`Configuration`.
-
-[[vertexprogram]]
-VertexProgram
--------------
-
-image:bsp-diagram.png[width=400,float=right] GraphComputer takes a `VertexProgram`. A VertexProgram can be thought of
-as a piece of code that is executed at each vertex in logically parallel manner until some termination condition is
-met (e.g. a number of iterations have occurred, no more data is changing in the graph, etc.). A submitted
-`VertexProgram` is copied to all the workers in the graph. A worker is not an explicit concept in the API, but is
-assumed of all `GraphComputer` implementations. At minimum each vertex is a worker (though this would be inefficient
-due to the fact that each vertex would maintain a VertexProgram). In practice, the workers partition the vertex set
-and and are responsible for the execution of the VertexProgram over all the vertices within their sphere of influence.
-The workers orchestrate the execution of the `VertexProgram.execute()` method on all their vertices in an
-link:http://en.wikipedia.org/wiki/Bulk_synchronous_parallel[bulk synchronous parallel] (BSP) fashion. The vertices
-are able to communicate with one another via messages. There are two kinds of messages in Gremlin OLAP:
-`MessageScope.Local` and `MessageScope.Global`. A local message is a message to an adjacent vertex. A global
-message is a message to any arbitrary vertex in the graph. Once the VertexProgram has completed its execution,
-any number of `MapReduce` jobs are evaluated. MapReduce jobs are provided by the user via `GraphComputer.mapReduce()`
- or by the `VertexProgram` via `VertexProgram.getMapReducers()`.
-
-image::graphcomputer.png[width=500]
-
-The example below demonstrates how to submit a VertexProgram to a graph's GraphComputer. `GraphComputer.submit()`
-yields a `Future<ComputerResult>`. The `ComputerResult` has the resultant computed graph which can be a full copy
-of the original graph (see <<hadoop-gremlin,Hadoop-Gremlin>>) or a view over the original graph (see
-<<tinkergraph,TinkerGraph>>). The ComputerResult also provides access to computational side-effects called `Memory`
-(which includes, for example, runtime, number of iterations, results of MapReduce jobs, and VertexProgram-specific
-memory manipulations).
-
-[gremlin-groovy,modern]
-----
-result = graph.compute().program(PageRankVertexProgram.build().create()).submit().get()
-result.memory().runtime
-g = result.graph().traversal(standard())
-g.V().valueMap('name',PageRankVertexProgram.PAGE_RANK)
-----
-
-NOTE: This model of "vertex-centric graph computing" was made popular by Google's
-link:http://googleresearch.blogspot.com/2009/06/large-scale-graph-computing-at-google.html[Pregel] graph engine.
-In the open source world, this model is found in OLAP graph computing systems such as link:https://giraph.apache.org/[Giraph],
-link:https://hama.apache.org/[Hama], and link:http://faunus.thinkaurelius.com[Faunus]. TinkerPop3 extends the
-popularized model with integrated post-processing <<mapreduce,MapReduce>> jobs over the vertex set.
-
-[[mapreduce]]
-MapReduce
----------
-
-The BSP model proposed by Pregel stores the results of the computation in a distributed manner as properties on the
-elements in the graph. In many situations, it is necessary to aggregate those resultant properties into a single
-result set (i.e. a statistic). For instance, assume a VertexProgram that computes a nominal cluster for each vertex
-(i.e. link:http://en.wikipedia.org/wiki/Community_structure[a graph clustering algorithm]). At the end of the
-computation, each vertex will have a property denoting the cluster it was assigned to. TinkerPop3 provides the
-ability to answer global questions about the clusters. For instance, in order to answer the following questions,
-`MapReduce` jobs are required:
-
- * How many vertices are in each cluster? (*presented below*)
- * How many unique clusters are there? (*presented below*)
- * What is the average age of each vertex in each cluster?
- * What is the degree distribution of the vertices in each cluster?
-
-A compressed representation of the `MapReduce` API in TinkerPop3 is provided below. The key idea is that the
-`map`-stage processes all vertices to emit key/value pairs. Those values are aggregated on their respective key
-for the `reduce`-stage to do its processing to ultimately yield more key/value pairs.
-
-[source,java]
-public interface MapReduce<MK, MV, RK, RV, R> {
- public void map(final Vertex vertex, final MapEmitter<MK, MV> emitter);
- public void reduce(final MK key, final Iterator<MV> values, final ReduceEmitter<RK, RV> emitter);
- // there are more methods
-}
-
-IMPORTANT: The vertex that is passed into the `MapReduce.map()` method does not contain edges. The vertex only
-contains original and computed vertex properties. This reduces the amount of data required to be loaded and ensures
-that MapReduce is used for post-processing computed results. All edge-based computing should be accomplished in the
-`VertexProgram`.
-
-image::mapreduce.png[width=650]
-
-The `MapReduce` extension to GraphComputer is made explicit when examining the
-<<peerpressurevertexprogram,`PeerPressureVertexProgram`>> and corresponding `ClusterPopulationMapReduce`.
-In the code below, the GraphComputer result returns the computed on `Graph` as well as the `Memory` of the
-computation (`ComputerResult`). The memory maintain the results of any MapReduce jobs. The cluster population
-MapReduce result states that there are 5 vertices in cluster 1 and 1 vertex in cluster 6. This can be verified
-(in a serial manner) by looking at the `PeerPressureVertexProgram.CLUSTER` property of the resultant graph. Notice
-that the property is "hidden" unless it is directly accessed via name.
-
-[gremlin-groovy,modern]
-----
-graph = TinkerFactory.createModern()
-result = graph.compute().program(PeerPressureVertexProgram.build().create()).mapReduce(ClusterPopulationMapReduce.build().create()).submit().get()
-result.memory().get('clusterPopulation')
-g = result.graph().traversal(standard())
-g.V().values(PeerPressureVertexProgram.CLUSTER).groupCount().next()
-g.V().valueMap()
-----
-
-If there are numerous statistics desired, then its possible to register as many MapReduce jobs as needed. For
-instance, the `ClusterCountMapReduce` determines how many unique clusters were created by the peer pressure algorithm.
-Below both `ClusterCountMapReduce` and `ClusterPopulationMapReduce` are computed over the resultant graph.
-
-[gremlin-groovy,modern]
-----
-result = graph.compute().program(PeerPressureVertexProgram.build().create()).
- mapReduce(ClusterPopulationMapReduce.build().create()).
- mapReduce(ClusterCountMapReduce.build().create()).submit().get()
-result.memory().clusterPopulation
-result.memory().clusterCount
-----
-
-IMPORTANT: The MapReduce model of TinkerPop3 does not support MapReduce chaining. Thus, the order in which the
-MapReduce jobs are executed is irrelevant. This is made apparent when realizing that the `map()`-stage takes a
-`Vertex` as its input and the `reduce()`-stage yields key/value pairs. Thus, the results of reduce can not feed back
-into map.
-
-A Collection of VertexPrograms
-------------------------------
-
-TinkerPop3 provides a collection of VertexPrograms that implement common algorithms. This section discusses the various implementations.
-
-IMPORTANT: The vertex programs presented are what are provided as of TinkerPop x.y.z. Over time, with future releases,
-more algorithms will be added.
-
-[[pagerankvertexprogram]]
-PageRankVertexProgram
-~~~~~~~~~~~~~~~~~~~~~
-
-image:gremlin-pagerank.png[width=400,float=right] link:http://en.wikipedia.org/wiki/PageRank[PageRank] is perhaps the
-most popular OLAP-oriented graph algorithm. This link:http://en.wikipedia.org/wiki/Centrality[eigenvector centrality]
-variant was developed by Brin and Page of Google. PageRank defines a centrality value for all vertices in the graph,
-where centrality is defined recursively where a vertex is central if it is connected to central vertices. PageRank is
-an iterative algorithm that converges to a link:http://en.wikipedia.org/wiki/Ergodicity[steady state distribution]. If
-the pageRank values are normalized to 1.0, then the pageRank value of a vertex is the probability that a random walker
-will be seen that that vertex in the graph at any arbitrary moment in time. In order to help developers understand the
-methods of a `VertexProgram`, the PageRankVertexProgram code is analyzed below.
-
-[source,java]
-----
-public class PageRankVertexProgram implements VertexProgram<Double> { <1>
-
- private MessageScope.Local<Double> incidentMessageScope = MessageScope.Local.of(__::outE); <2>
- private MessageScope.Local<Double> countMessageScope = MessageScope.Local.of(new MessageScope.Local.ReverseTraversalSupplier(this.incidentMessageScope));
-
- public static final String PAGE_RANK = "gremlin.pageRankVertexProgram.pageRank"; <3>
- public static final String EDGE_COUNT = "gremlin.pageRankVertexProgram.edgeCount";
-
- private static final String VERTEX_COUNT = "gremlin.pageRankVertexProgram.vertexCount";
- private static final String ALPHA = "gremlin.pageRankVertexProgram.alpha";
- private static final String TOTAL_ITERATIONS = "gremlin.pageRankVertexProgram.totalIterations";
- private static final String INCIDENT_TRAVERSAL_SUPPLIER = "gremlin.pageRankVertexProgram.incidentTraversalSupplier";
-
- private ConfigurationTraversal<Vertex, Edge> configurationTraversal;
- private double vertexCountAsDouble = 1.0d;
- private double alpha = 0.85d;
- private int totalIterations = 30;
-
- private static final Set<String> COMPUTE_KEYS = new HashSet<>(Arrays.asList(PAGE_RANK, EDGE_COUNT));
-
- private PageRankVertexProgram() {}
-
- @Override
- public void loadState(final Graph graph, final Configuration configuration) { <4>
- if (configuration.containsKey(TRAVERSAL_SUPPLIER)) {
- this.configurationTraversal = ConfigurationTraversal.loadState(graph, configuration, TRAVERSAL_SUPPLIER);
- this.incidentMessageScope = MessageScope.Local.of(this.configurationTraversal);
- this.countMessageScope = MessageScope.Local.of(new MessageScope.Local.ReverseTraversalSupplier(this.incidentMessageScope));
- }
- this.vertexCountAsDouble = configuration.getDouble(VERTEX_COUNT, 1.0d);
- this.alpha = configuration.getDouble(ALPHA, 0.85d);
- this.totalIterations = configuration.getInt(TOTAL_ITERATIONS, 30);
- }
-
- @Override
- public void storeState(final Configuration configuration) {
- configuration.setProperty(VERTEX_PROGRAM, PageRankVertexProgram.class.getName());
- configuration.setProperty(VERTEX_COUNT, this.vertexCountAsDouble);
- configuration.setProperty(ALPHA, this.alpha);
- configuration.setProperty(TOTAL_ITERATIONS, this.totalIterations);
- if (null != this.traversalSupplier) {
- this.traversalSupplier.storeState(configuration);
- }
- }
-
- @Override
- public Set<String> getElementComputeKeys() { <5>
- return COMPUTE_KEYS;
- }
-
- @Override
- public Optional<MessageCombiner<Double>> getMessageCombiner() {
- return (Optional) PageRankMessageCombiner.instance();
- }
-
- @Override
- public Set<MessageScope> getMessageScopes(final int iteration) {
- final Set<MessageScope> set = new HashSet<>();
- set.add(0 == iteration ? this.countMessageScope : this.incidentMessageScope);
- return set;
- }
-
- @Override
- public void setup(final Memory memory) {
-
- }
-
- @Override
- public void execute(final Vertex vertex, Messenger<Double> messenger, final Memory memory) { <6>
- if (memory.isInitialIteration()) { <7>
- messenger.sendMessage(this.countMessageScope, 1.0d);
- } else if (1 == memory.getIteration()) { <8>
- double initialPageRank = 1.0d / this.vertexCountAsDouble;
- double edgeCount = IteratorUtils.reduce(messenger.receiveMessages(), 0.0d, (a, b) -> a + b);
- vertex.property(PAGE_RANK, initialPageRank);
- vertex.property(EDGE_COUNT, edgeCount);
- messenger.sendMessage(this.incidentMessageScope, initialPageRank / edgeCount);
- } else { <9>
- double newPageRank = IteratorUtils.reduce(messenger.receiveMessages(), 0.0d, (a, b) -> a + b);
- newPageRank = (this.alpha * newPageRank) + ((1.0d - this.alpha) / this.vertexCountAsDouble);
- vertex.property(PAGE_RANK, newPageRank);
- messenger.sendMessage(this.incidentMessageScope, newPageRank / vertex.<Double>value(EDGE_COUNT));
- }
- }
-
- @Override
- public boolean terminate(final Memory memory) { <10>
- return memory.getIteration() >= this.totalIterations;
- }
-
- @Override
- public String toString() {
- return StringFactory.vertexProgramString(this, "alpha=" + this.alpha + ",iterations=" + this.totalIterations);
- }
-}
-----
-
-<1> `PageRankVertexProgram` implements `VertexProgram<Double>` because the messages it sends are Java doubles.
-<2> The default path of energy propagation is via outgoing edges from the current vertex.
-<3> The resulting PageRank values for the vertices are stored as a hidden property.
-<4> A vertex program is constructed using an Apache `Configuration` to ensure easy dissemination across a cluster of JVMs.
-<5> A vertex program must define the "compute keys" that are the properties being operated on during the computation.
-<6> The "while"-loop of the vertex program.
-<7> In order to determine how to distribute the energy to neighbors, a "1"-count is used to determine how many incident vertices exist for the `MessageScope`.
-<8> Initially, each vertex is provided an equal amount of energy represented as a double.
-<9> Energy is aggregated, computed on according to the PageRank algorithm, and then disseminated according to the defined `MessageScope.Local`.
-<10> The computation is terminated after a pre-defined number of iterations.
-
-The above `PageRankVertexProgram` is used as follows.
-
-[gremlin-groovy,modern]
-----
-result = graph.compute().program(PageRankVertexProgram.build().create()).submit().get()
-result.memory().runtime
-g = result.graph().traversal(standard())
-g.V().valueMap('name',PageRankVertexProgram.PAGE_RANK)
-----
-
-
-[[peerpressurevertexprogram]]
-PeerPressureVertexProgram
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-The `PeerPressureVertexProgram` is a clustering algorithm that assigns a nominal value to each vertex in the graph.
-The nominal value represents the vertex's cluster. If two vertices have the same nominal value, then they are in the
-same cluster. The algorithm proceeds in the following manner.
-
- . Every vertex assigns itself to a unique cluster ID (initially, its vertex ID).
- . Every vertex determines its per neighbor vote strength as 1.0d / incident edges count.
- . Every vertex sends its cluster ID and vote strength to its adjacent vertices as a `Pair<Serializable,Double>`
- . Every vertex generates a vote energy distribution of received cluster IDs and changes its current cluster ID to the most frequent cluster ID.
- .. If there is a tie, then the cluster with the lowest `toString()` comparison is selected.
- . Steps 3 and 4 repeat until either a max number of iterations has occurred or no vertex has adjusted its cluster anymore.
-
-[[bulkloadervertexprogram]]
-BulkLoaderVertexProgram
-~~~~~~~~~~~~~~~~~~~~~~~
-
-The `BulkLoaderVertexProgram` provides a generalized way for loading graphs of any size (preferably large sized graphs)
-into a persistent `Graph`. The input can be any existing `Graph` database supporting TinkerPop3 or any of the Hadoop
-GraphInputFormats (e.g. `GraphSONInputFormat`, `GryoInputFormat` or `ScriptInputFormat`). The following example
-demonstrates how to load data from one TinkerGraph to another:
-
-[gremlin-groovy]
-----
-writeGraphConf = new BaseConfiguration()
-writeGraphConf.setProperty("gremlin.graph", "org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph")
-writeGraphConf.setProperty("gremlin.tinkergraph.graphFormat", "gryo")
-writeGraphConf.setProperty("gremlin.tinkergraph.graphLocation", "/tmp/tinkergraph.kryo")
-modern = TinkerFactory.createModern()
-blvp = BulkLoaderVertexProgram.build().
- keepOriginalIds(false).
- writeGraph(writeGraphConf).create(modern)
-modern.compute().workers(1).program(blvp).submit().get()
-graph = GraphFactory.open(writeGraphConf)
-g = graph.traversal()
-g.V().valueMap()
-graph.close()
-----
-
-.Available configuration options
-[width="800px",options="header"]
-|========================================
-|Builder Method |Purpose | Default Value
-|`bulkLoader(Class\|String)` | Sets the class of the bulk loader implementation. | `IncrementalBulkLoader`
-|`vertexIdProperty(String)` | Sets the name of the property in the target graph that holds the vertex id from the
-source graph. | `bulkLoader.vertex.id`
-|`keepOriginalIds(boolean)` |Whether to keep the id's from the source graph in the target graph or not. It's
-recommended to keep them if it's planned to do further bulk loads using the same datasources. | `true`
-|`userSuppliedIds(boolean)` |Whether to use the id's from the source graph as id's in the target graph. If set to
-`true`, `vertexIdProperty` will be ignored. Note, that the target graph must support user supplied identifiers. | `false`
-|`intermediateBatchSize(int)` |Sets the batch size for intermediate transactions. This is per thread in a
-multi-threaded environment. +0+ means that transactions will only be committed at the end of an iteration cycle.
-It's recommended to tune this property for the target graph and not use the default value of +0+. | `0`
-|`writeGraph(String)` | Sets the path to a `GraphFactory` compatible configuration file for the target graph. | _none_
-|========================================
-
-NOTE: `BulkLoaderVertexProgram` comes with a default `BulkLoader` implementation, namely `IncrementalBulkLoader`. It
-will work for the most use-cases, but has one limitation though: It doesn't support multi-valued properties.
-`IncrementalBulkLoader` will handle every property as a single-valued property. A custom `BulkLoader` implementation
-has to be used if the default behavior is not sufficient.
-
-NOTE: A custom `BulkLoader` implementation for incremental loading should use `GraphTraversal` methods to create/update
-elements (e.g. `g.addV()` instead of `graph.addVertex()`). This way the `BulkLoaderVertexProgram` is able to efficiently
-track changes in the underlying graph and can apply several optimization techniques.
-
-[[traversalvertexprogram]]
-TraversalVertexProgram
-~~~~~~~~~~~~~~~~~~~~~~
-
-image:traversal-vertex-program.png[width=250,float=left] The `TraversalVertexProgram` is a "special" VertexProgram in
-that it can be executed via a `GraphTraversal` with a `ComputerTraversalEngine`. In Gremlin, it is possible to have
-the same traversal executed using either the standard OTLP-engine or the `GraphComputer` OLAP-engine. The difference
-being where the traversal is submitted.
-
-NOTE: This model of graph traversal in a BSP system was first implemented by the
-link:http://faunus.thinkaurelius.com[Faunus] graph analytics engine and originally described in
-link:http://markorodriguez.com/2011/04/19/local-and-distributed-traversal-engines/[Local and Distributed Traversal Engines].
-
-[gremlin-groovy,modern]
-----
-g = graph.traversal(standard())
-g.V().both().hasLabel('person').values('age').groupCount().next() // OLTP
-g = graph.traversal(computer())
-g.V().both().hasLabel('person').values('age').groupCount().next() // OLAP
-----
-
-image::olap-traversal.png[width=650]
-
-In the OLAP example above, a `TraversalVertexProgram` is (logically) sent to each vertex in the graph. Each instance
-evaluation requires (logically) 5 BSP iterations and each iteration is interpreted as such:
-
- . `g.V()`: Put a traverser on each vertex in the graph.
- . `both()`: Propagate each traverser to the vertices `both`-adjacent to its current vertex.
- . `hasLabel('person')`: If the vertex is not a person, kill the traversers at that vertex.
- . `values('age')`: Have all the traversers reference the integer age of their current vertex.
- . `groupCount()`: Count how many times a particular age has been seen.
-
-While 5 iterations were presented, in fact, `TraversalVertexProgram` will execute the traversal in only
-3 iterations. The reason being is that `hasLabel('person').values('age').groupCount()` can all be executed in a
-single iteration as any message sent would simply be to the current executing vertex. Thus, a simple optimization
-exists in Gremlin OLAP called "reflexive message passing" which simulates non-message-passing BSP iterations within a
-single BSP iteration.
-
-When the computation is complete a <<mapreduce,MapReduce>> job executes which aggregates all the `groupCount()`
-sideEffect Map (i.e. "`HashMap`") objects on each vertex into a single local representation (thus, turning the
-distributed Map representation into a local Map representation).
-
-////
-The same OLAP traversal can be executed using the standard `g.compute()` model, though at the expense of verbosity.
-`TraversalVertexProgram` provides a fluent `Builder` for constructing a `TraversalVertexProgram`. The specified
-`traversal()` can be either a `Supplier<Traversal>` object, a `Supplier<Traversal>` class, or a
-link:http://en.wikipedia.org/wiki/Scripting_for_the_Java_Platform[JSR-223] script that will generate (i.e. supply) a
-`Traversal`. If `traversal()` is supplied a single string, it is assumed that "gremlin-groovy" is the `ScriptEngine`
-to use. If two strings are supplied, then the first string denotes the `ScriptEngine` to evaluate the second string
-script with in order to generate (i.e. supply) the `Traversal`.
-
-[gremlin-groovy,modern]
-----
-//g.engine(computer())
-//result = g.compute().program(TraversalVertexProgram.build().traversal(g.V().both().hasLabel('person').values('age').groupCount('a')).create()).submit().get()
-//result.memory().a
-//result.memory().iteration
-//result.memory().runtime
-----
-////
-
-[[distributed-gremlin-gotchas]]
-Distributed Gremlin Gotchas
-^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Gremlin OLTP is not identical to Gremlin OLAP.
-
-IMPORTANT: There are two primary theoretical differences between Gremlin OLTP and Gremlin OLAP. First, Gremlin OLTP
-(via `Traversal`) leverages a link:http://en.wikipedia.org/wiki/Depth-first_search[depth-first] execution engine.
-Depth-first execution has a limited memory footprint due to link:http://en.wikipedia.org/wiki/Lazy_evaluation[lazy evaluation].
-On the other hand, Gremlin OLAP (via `TraversalVertexProgram`) leverages a
-link:http://en.wikipedia.org/wiki/Breadth-first_search[breadth-first] execution engine which maintains a larger memory
-footprint, but a better time complexity due to vertex-local traversers being able to be merged. The second difference
-is that Gremlin OLTP is executed in a serial fashion, while Gremlin OLAP is executed in a parallel fashion. These two
-fundamental differences lead to the behaviors enumerated below.
-
-image::gremlin-without-a-cause.png[width=200,float=right]
-
-. Traversal sideEffects are represented as a distributed data structure across the graph's vertex set. It is not
-possible to get a global view of a sideEffect until it is aggregated via a <<mapreduce,MapReduce>> job. In some
-situations, the local vertex representation of the sideEffect is sufficient to ensure the intended semantics of the
-traversal are respected. However, this is not generally true so be wary of traversals that require global views of a
-sideEffect.
-. When evaluating traversals that rely on path information (i.e. the history of the traversal), practical
-computational limits can easily be reached due the link:http://en.wikipedia.org/wiki/Combinatorial_explosion[combinatoric explosion]
-of data. With path computing enabled, every traverser is unique and thus, must be enumerated as opposed to being
-counted/merged. The difference being a collection of paths vs. a single 64-bit long at a single vertex. For more
-information on this concept, please see link:http://thinkaurelius.com/2012/11/11/faunus-provides-big-graph-data-analytics/[Faunus Provides Big Graph Data].
-. When traversals of the form `x.as('a').y.someSideEffectStep('a').z` are evaluated, the `a` object is stored in the
-path information of the traverser and thus, such traversals (may) turn on path calculations when executed on a
-`GraphComputer`
-. Steps that are concerned with the global ordering of traversers do not have a meaningful representation in
-OLAP. For example, what does <<order-step,`order()`>>-step mean when all traversers are being processed in parallel?
-Even if the traversers were aggregated and ordered, then at the next step they would return to being executed in
-parallel and thus, in an unpredictable order. When `order()`-like steps are executed at the end of a traversal (i.e
-the final step), the `TraverserMapReduce` job ensures the resultant serial representation is ordered accordingly.
-. Steps that are concerned with providing a global aggregate to the next step of computation do not have a correlate
-in OLAP. For example, <<fold-step,`fold()`>>-step can only fold up the objects at each executing vertex. Next, even
-if a global fold was possible, where would it go? Which vertex would be the host of the data structure? The
-`fold()`-step only makes sense as an end-step whereby a MapReduce job can generate the proper global-to-local data
-reduction.
[08/22] incubator-tinkerpop git commit: Made subdirectories for
various "books" in the docs.
Posted by sp...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/reference/index.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/reference/index.asciidoc b/docs/src/reference/index.asciidoc
new file mode 100644
index 0000000..5a01291
--- /dev/null
+++ b/docs/src/reference/index.asciidoc
@@ -0,0 +1,39 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+image::apache-tinkerpop-logo.png[width=500]
+
+*x.y.z*
+
+:toc-position: left
+
+include::preface.asciidoc[]
+
+include::intro.asciidoc[]
+
+include::the-graph.asciidoc[]
+
+include::the-traversal.asciidoc[]
+
+include::the-graphcomputer.asciidoc[]
+
+include::gremlin-applications.asciidoc[]
+
+include::implementations.asciidoc[]
+
+include::conclusion.asciidoc[]
+
+include::acknowledgements.asciidoc[]
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/reference/intro.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/reference/intro.asciidoc b/docs/src/reference/intro.asciidoc
new file mode 100644
index 0000000..2affc01
--- /dev/null
+++ b/docs/src/reference/intro.asciidoc
@@ -0,0 +1,397 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+[[intro]]
+Introduction to Graph Computing
+===============================
+
+image::graph-computing.png[width=350]
+
+[source,xml]
+<dependency>
+ <groupId>org.apache.tinkerpop</groupId>
+ <artifactId>gremlin-core</artifactId>
+ <version>x.y.z</version>
+</dependency>
+
+A link:http://en.wikipedia.org/wiki/Graph_(data_structure)[graph] is a data structure composed of vertices (nodes,
+dots) and edges (arcs, lines). When modeling a graph in a computer and applying it to modern data sets and practices,
+the generic mathematically-oriented, binary graph is extended to support both labels and key/value properties. This
+structure is known as a property graph. More formally, it is a directed, binary, attributed multi-graph. An example
+property graph is diagrammed below. This graph example will be used extensively throughout the documentation and is
+called "TinkerPop Classic" as it is the original demo graph distributed with TinkerPop0 back in 2009 (i.e. the good
+ol' days -- it was the best of times and it was the worst of times).
+
+TIP: The TinkerPop graph is available with <<tinkergraph-gremlin,TinkerGraph>> via `TinkerFactory.createModern()`.
+TinkerGraph is the reference implementation of TinkerPop3 and is used in nearly all the examples in this documentation.
+Note that there also exists the classic `TinkerFactory.createClassic()` which is the graph used in TinkerPop2 and does
+not include vertex labels.
+
+[[tinkerpop-modern]]
+.TinkerPop Modern
+image::tinkerpop-modern.png[width=500]
+
+TinkerPop3 is the third incarnation of the TinkerPop graph computing framework. Similar to computing in general, graph
+computing makes a distinction between *structure* (graph) and *process* (traversal). The structure of the graph is the
+data model defined by a vertex/edge/property link:http://en.wikipedia.org/wiki/Network_topology[topology]. The process
+of the graph is the means by which the structure is analyzed. The typical form of graph processing is called a
+link:http://en.wikipedia.org/wiki/Graph_traversal[traversal].
+
+.Primary components of the TinkerPop3 *structure* API
+ * `Graph`: maintains a set of vertices and edges, and access to database functions such as transactions.
+ * `Element`: maintains a collection of properties and a string label denoting the element type.
+ ** `Vertex`: extends Element and maintains a set of incoming and outgoing edges.
+ ** `Edge`: extends Element and maintains an incoming and outgoing vertex.
+ * `Property<V>`: a string key associated with a `V` value.
+ ** `VertexProperty<V>`: a string key associated with a `V` value as well as a collection of `Property<U>` properties (*vertices only*)
+
+.Primary components of the TinkerPop3 *process* API
+ * `TraversalSource`: a generator of traversals for a particular graph, link:http://en.wikipedia.org/wiki/Domain-specific_language[domain specific language] (DSL), and execution engine.
+ ** `Traversal<S,E>`: a functional data flow process transforming objects of type `S` into object of type `E`.
+ *** `GraphTraversal`: a traversal DSL that is oriented towards the semantics of the raw graph (i.e. vertices, edges, etc.).
+ * `GraphComputer`: a system that processes the graph in parallel and potentially, distributed over a multi-machine cluster.
+ ** `VertexProgram`: code executed at all vertices in a logically parallel manner with intercommunication via message passing.
+ ** `MapReduce`: a computations that analyzes all vertices in the graph in parallel and yields a single reduced result.
+
+IMPORTANT: TinkerPop3 is licensed under the popular link:http://www.apache.org/licenses/LICENSE-2.0.html[Apache2]
+free software license. However, note that the underlying graph engine used with TinkerPop3 may have a difference
+license. Thus, be sure to respect the license caveats of the graph system product.
+
+image:tinkerpop-enabled.png[width=135,float=left] When a graph system implements the TinkerPop3 structure and process
+link:http://en.wikipedia.org/wiki/Application_programming_interface[APIs], their technology is considered
+_TinkerPop3-enabled_ and becomes nearly indistinguishable from any other TinkerPop-enabled graph system save for
+their respective time and space complexity. The purpose of this documentation is to describe the structure/process
+dichotomy at length and in doing so, explain how to leverage TinkerPop3 for the sole purpose of graph system-agnostic
+graph computing. Before deep-diving into the various structure/process APIs, a short introductory review of both APIs
+is provided.
+
+NOTE: The TinkerPop3 API rides a fine line between providing concise "query language" method names and respecting
+Java method naming standards. The general convention used throughout TinkerPop3 is that if a method is "user exposed,"
+then a concise name is provided (e.g. `out()`, `path()`, `repeat()`). If the method is primarily for graph systems
+providers, then the standard Java naming convention is followed (e.g. `getNextStep()`, `getSteps()`,
+`getElementComputeKeys()`).
+
+The Graph Structure
+-------------------
+
+image:gremlin-standing.png[width=125,float=left] A graph's structure is the topology formed by the explicit references
+between its vertices, edges, and properties. A vertex has incident edges. A vertex is adjacent to another vertex if
+they share an incident edge. A property is attached to an element and an element has a set of properties. A property
+is a key/value pair, where the key is always a character `String`. The graph structure API of TinkerPop3 provides the
+methods necessary to create such a structure. The TinkerPop graph previously diagrammed can be created with the
+following Java8 code. Note that this graph is available as an in-memory TinkerGraph using
+`TinkerFactory.createClassic()`.
+
+[source,java]
+Graph graph = TinkerGraph.open(); <1>
+Vertex marko = graph.addVertex(T.label, "person", T.id, 1, "name", "marko", "age", 29); <2>
+Vertex vadas = graph.addVertex(T.label, "person", T.id, 2, "name", "vadas", "age", 27);
+Vertex lop = graph.addVertex(T.label, "software", T.id, 3, "name", "lop", "lang", "java");
+Vertex josh = graph.addVertex(T.label, "person", T.id, 4, "name", "josh", "age", 32);
+Vertex ripple = graph.addVertex(T.label, "software", T.id, 5, "name", "ripple", "lang", "java");
+Vertex peter = graph.addVertex(T.label, "person", T.id, 6, "name", "peter", "age", 35);
+marko.addEdge("knows", vadas, T.id, 7, "weight", 0.5f); <3>
+marko.addEdge("knows", josh, T.id, 8, "weight", 1.0f);
+marko.addEdge("created", lop, T.id, 9, "weight", 0.4f);
+josh.addEdge("created", ripple, T.id, 10, "weight", 1.0f);
+josh.addEdge("created", lop, T.id, 11, "weight", 0.4f);
+peter.addEdge("created", lop, T.id, 12, "weight", 0.2f);
+
+<1> Create a new in-memory `TinkerGraph` and assign it to the variable `graph`.
+<2> Create a vertex along with a set of key/value pairs with `T.label` being the vertex label and `T.id` being the vertex id.
+<3> Create an edge along with a set of key/value pairs with the edge label being specified as the first argument.
+
+In the above code all the vertices are created first and then their respective edges. There are two "accessor tokens":
+`T.id` and `T.label`. When any of these, along with a set of other key value pairs is provided to
+`Graph.addVertex(Object...)` or `Vertex.addEdge(String,Vertex,Object...)`, the respective element is created along
+with the provided key/value pair properties appended to it.
+
+CAUTION: Many graph systems do not allow the user to specify an element ID and in such cases, an exception is thrown.
+
+NOTE: In TinkerPop3, vertices are allowed a single immutable string label (similar to an edge label). This
+functionality did not exist in TinkerPop2. Likewise, element id's are immutable as they were in TinkerPop2.
+
+Mutating the Graph
+~~~~~~~~~~~~~~~~~~
+
+Below is a sequence of basic graph mutation operations represented in Java8. One of the major differences between
+TinkerPop2 and TinkerPop3 is that in TinkerPop3, the Java convention of using setters and getters has been abandoned
+in favor of a syntax that is more aligned with the syntax of Gremlin-Groovy in TinkerPop2. Given that Gremlin-Java8
+and Gremlin-Groovy are nearly identical due to the inclusion of Java8 lambdas, a big efforts was made to ensure that
+both languages are as similar as possible.
+
+CAUTION: In the code examples presented throughout this documentation, either Gremlin-Java8 or Gremlin-Groovy is used.
+It is possible to determine which derivative of Gremlin is being used by "mousing over" on the code block and see
+either "JAVA" or "GROOVY" pop up in the top right corner of the code block.
+
+image:basic-mutation.png[width=240,float=right]
+[source,java]
+// create a new graph
+Graph graph = TinkerGraph.open();
+// add a software vertex with a name property
+Vertex gremlin = graph.addVertex(T.label, "software",
+ "name", "gremlin"); <1>
+// only one vertex should exist
+assert(IteratorUtils.count(graph.vertices()) == 1)
+// no edges should exist as none have been created
+assert(IteratorUtils.count(graph.edges()) == 0)
+// add a new property
+gremlin.property("created",2009) <2>
+// add a new software vertex to the graph
+Vertex blueprints = graph.addVertex(T.label, "software",
+ "name", "blueprints"); <3>
+// connect gremlin to blueprints via a dependsOn-edge
+gremlin.addEdge("dependsOn",blueprints); <4>
+// now there are two vertices and one edge
+assert(IteratorUtils.count(graph.vertices()) == 2)
+assert(IteratorUtils.count(graph.edges()) == 1)
+// add a property to blueprints
+blueprints.property("created",2010) <5>
+// remove that property
+blueprints.property("created").remove() <6>
+// connect gremlin to blueprints via encapsulates
+gremlin.addEdge("encapsulates",blueprints) <7>
+assert(IteratorUtils.count(graph.vertices()) == 2)
+assert(IteratorUtils.count(graph.edges()) == 2)
+// removing a vertex removes all its incident edges as well
+blueprints.remove() <8>
+gremlin.remove() <9>
+// the graph is now empty
+assert(IteratorUtils.count(graph.vertices()) == 0)
+assert(IteratorUtils.count(graph.edges()) == 0)
+// tada!
+
+IMPORTANT: image:groovy-logo.png[width=175,float=left] Gremlin-Groovy leverages the
+link:http://groovy.codehaus.org/[Groovy 2.x language] to express Gremlin traversals. One of the major benefits of
+Groovy is the inclusion of a runtime console that makes it easy for developers to practice with the Gremlin language
+and for production users to connect to their graph and execute traversals in an interactive manner. Moreover,
+Gremlin-Groovy provides various syntax simplifications.
+
+TIP: image:gremlin-sugar.png[width=100,float=left] For those wishing to use the Gremlin2 syntax, please see
+<<sugar-plugin,SugarPlugin>>. This plugin provides syntactic sugar at, typically, a runtime cost. It can be loaded
+programmatically via `SugarLoader.load()`. Once loaded, it is possible to do `g.V.out.name` instead of
+`g.V().out().values('name')` as well as a host of other conveniences.
+
+Here is the same code, but using Gremlin-Groovy in the <<gremlin-console,Gremlin Console>>.
+
+[source,groovy]
+----
+$ bin/gremlin.sh
+
+ \,,,/
+ (o o)
+-----oOOo-(3)-oOOo-----
+gremlin> graph = TinkerGraph.open()
+==>tinkergraph[vertices:0 edges:0]
+gremlin> gremlin = graph.addVertex(label,'software','name','gremlin')
+==>v[0]
+gremlin> gremlin.property('created',2009)
+==>vp[created->2009]
+gremlin> blueprints = graph.addVertex(label,'software','name','blueprints')
+==>v[3]
+gremlin> gremlin.addEdge('dependsOn',blueprints)
+==>e[5][0-dependsOn->3]
+gremlin> blueprints.property('created',2010)
+==>vp[created->2010]
+gremlin> blueprints.property('created').remove()
+==>null
+gremlin> gremlin.addEdge('encapsulates',blueprints)
+==>e[7][0-encapsulates->3]
+gremlin> blueprints.remove()
+==>null
+gremlin> gremlin.remove()
+==>null
+----
+
+IMPORTANT: TinkerGraph is not a transactional graph. For more information on transaction handling (for those graph
+systems that support them) see the section dedicated to <<transactions,transactions>>.
+
+The Graph Process
+-----------------
+
+image:gremlin-running.png[width=125,float=left] The primary way in which graphs are processed are via graph
+traversals. The TinkerPop3 process API is focused on allowing users to create graph traversals in a
+syntactically-friendly way over the structures defined in the previous section. A traversal is an algorithmic walk
+across the elements of a graph according to the referential structure explicit within the graph data structure.
+For example: _"What software does vertex 1's friends work on?"_ This English-statement can be represented in the
+following algorithmic/traversal fashion:
+
+ . Start at vertex 1.
+ . Walk the incident knows-edges to the respective adjacent friend vertices of 1.
+ . Move from those friend-vertices to software-vertices via created-edges.
+ . Finally, select the name-property value of the current software-vertices.
+
+Traversals in Gremlin are spawned from a `TraversalSource`. The `GraphTraversalSource` is the typical "graph-oriented"
+DSL used throughout the documentation and will most likely be the most used DSL in a TinkerPop application.
+`GraphTraversalSource` provides two traversal methods.
+
+ . `GraphTraversalSource.V(Object... ids)`: generates a traversal starting at vertices in the graph (if no ids are provided, all vertices).
+ . `GraphTraversalSource.E(Object... ids)`: generates a traversal starting at edges in the graph (if no ids are provided, all edges).
+
+The return type of `V()` and `E()` is a `GraphTraversal`. A GraphTraversal maintains numerous methods that return
+`GraphTraversal`. In this way, a `GraphTraversal` supports function composition. Each method of `GraphTraversal` is
+called a step and each step modulates the results of the previous step in one of five general ways.
+
+ . `map`: transform the incoming traverser's object to another object (S → E).
+ . `flatMap`: transform the incoming traverser's object to an iterator of other objects (S → E*).
+ . `filter`: allow or disallow the traverser from proceeding to the next step (S → S ∪ ∅).
+ . `sideEffect`: allow the traverser to proceed unchanged, but yield some computational sideEffect in the process (S ↬ S).
+ . `branch`: split the traverser and send each to an arbitrary location in the traversal (S → { S~1~ → E*, ..., S~n~ → E* } → E*).
+
+Nearly every step in GraphTraversal either extends `MapStep`, `FlatMapStep`, `FilterStep`, `SideEffectStep`, or `BranchStep`.
+
+TIP: `GraphTraversal` is a link:http://en.wikipedia.org/wiki/Monoid[monoid] in that it is an algebraic structure
+that has a single binary operation that is associative. The binary operation is function composition (i.e. method
+chaining) and its identity is the step `identity()`. This is related to a
+link:http://en.wikipedia.org/wiki/Monad_(functional_programming)[monad] as popularized by the functional programming
+community.
+
+Given the TinkerPop graph, the following query will return the names of all the people that the marko-vertex knows.
+The following query is demonstrated using Gremlin-Groovy.
+
+[source,groovy]
+----
+$ bin/gremlin.sh
+
+ \,,,/
+ (o o)
+-----oOOo-(3)-oOOo-----
+gremlin> graph = TinkerFactory.createModern() // <1>
+==>tinkergraph[vertices:6 edges:6]
+gremlin> g = graph.traversal(standard()) // <2>
+==>graphtraversalsource[tinkergraph[vertices:6 edges:6], standard]
+gremlin> g.V().has('name','marko').out('knows').values('name') // <3>
+==>vadas
+==>josh
+----
+
+<1> Open the toy graph and reference it by the variable `graph`.
+<2> Create a graph traversal source from the graph using the standard, OLTP traversal engine.
+<3> Spawn a traversal off the traversal source that determines the names of the people that the marko-vertex knows.
+
+.The Name of The People That Marko Knows
+image::tinkerpop-classic-ex1.png[width=500]
+
+Or, if the marko-vertex is already realized with a direct reference pointer (i.e. a variable), then the traversal can
+be spawned off that vertex.
+
+[gremlin-groovy,modern]
+----
+marko = g.V().has('name','marko').next() <1>
+g.V(marko).out('knows') <2>
+g.V(marko).out('knows').values('name') <3>
+----
+
+<1> Set the variable `marko` to the the vertex in the graph `g` named "marko".
+<2> Get the vertices that are outgoing adjacent to the marko-vertex via knows-edges.
+<3> Get the names of the marko-vertex's friends.
+
+The Traverser
+~~~~~~~~~~~~~
+
+When a traversal is executed, the source of the traversal is on the left of the expression (e.g. vertex 1), the steps
+are the middle of the traversal (e.g. `out('knows')` and `values('name')`), and the results are "traversal.next()'d"
+out of the right of the traversal (e.g. "vadas" and "josh").
+
+image::traversal-mechanics.png[width=500]
+
+In TinkerPop3, the objects propagating through the traversal are wrapped in a `Traverser<T>`. The traverser concept
+is new to TinkerPop3 and provides the means by which steps remain stateless. A traverser maintains all the metadata
+about the traversal -- e.g., how many times the traverser has gone through a loop, the path history of the traverser,
+the current object being traversed, etc. Traverser metadata may be accessed by a step. A classic example is the
+<<path-step,`path()`>>-step.
+
+[gremlin-groovy,modern]
+----
+g.V(marko).out('knows').values('name').path()
+----
+
+CAUTION: Path calculation is costly in terms of space as an array of previously seen objects is stored in each path
+of the respective traverser. Thus, a traversal strategy analyzes the traversal to determine if path metadata is
+required. If not, then path calculations are turned off.
+
+Another example is the <<repeat-step,`repeat()`>>-step which takes into account the number of times the traverser
+has gone through a particular section of the traversal expression (i.e. a loop).
+
+[gremlin-groovy,modern]
+----
+g.V(marko).repeat(out()).times(2).values('name')
+----
+
+CAUTION: A Traversal's result are never ordered unless explicitly by means of <<order-step,`order()`>>-step. Thus,
+never rely on the iteration order between TinkerPop3 releases and even within a release (as traversal optimizations
+may alter the flow).
+
+On Gremlin Language Variants
+----------------------------
+
+Gremlin is written in Java8. There are various language variants of Gremlin such as Gremlin-Groovy (packaged with
+TinkerPop3), link:https://github.com/mpollmeier/gremlin-scala[Gremlin-Scala], Gremlin-JavaScript, Gremlin-Clojure
+(known as link:https://github.com/clojurewerkz/ogre[Ogre]), etc. It is best to think of Gremlin as a style of graph
+traversing that is not bound to a particular programming language per se. Within a programming language familiar to
+the developer, there is a Gremlin variant that they can use that leverages the idioms of that language. At minimum,
+a programming language providing a Gremlin implementation must support
+link:http://en.wikipedia.org/wiki/Method_chaining[function chaining] (with
+link:http://en.wikipedia.org/wiki/Anonymous_function[lambdas/anonymous functions] being a "nice to have" if the
+variants wishes to offer arbitrary computations beyond the provided Gremlin steps).
+
+Throughout the documentation, the examples provided are primarily written in Gremlin-Groovy. The reason for this is
+the <<gremlin-console,Gremlin Console>> whereby an interactive programming environment exists that does not require
+code compilation. For learning TinkerPop3 and interacting with a live graph system in an ad hoc manner, the Gremlin
+Console is invaluable. However, for developers interested in working with Gremlin-Java, a few Groovy-to-Java patterns
+are presented below.
+
+[source,groovy]
+// Gremlin-Groovy
+g.V().out('knows').values('name') <1>
+g.V().out('knows').map{it.get().value('name') + ' is the friend name'} <2>
+g.V().out('knows').sideEffect(System.out.&println) <3>
+g.V().as('person').out('knows').as('friend').select().by{it.value('name').length()} <4>
+
+[source,java]
+// Gremlin-Java
+g.V().out("knows").values("name") <1>
+g.V().out("knows").map(t -> t.get().value("name") + " is the friend name") <2>
+g.V().out("knows").sideEffect(System.out::println) <3>
+g.V().as("person").out("knows").as("friend").select().by((Function<Vertex, Integer>) v -> v.<String>value("name").length()) <4>
+
+<1> All the non-lambda step chaining is identical in Gremlin-Groovy and Gremlin-Java. However, note that Groovy
+supports `'` strings as well as `"` strings.
+<2> In Groovy, lambdas are called closures and have a different syntax, where Groovy supports the `it` keyword and
+Java doesn't with all parameters requiring naming.
+<3> The syntax for method references differs slightly between link:https://docs.oracle.com/javase/tutorial/java/javaOO/methodreferences.html[Java]
+and link:http://mrhaki.blogspot.de/2009/08/groovy-goodness-turn-methods-into.html[Gremlin-Groovy].
+<4> Groovy is lenient on object typing and Java is not. When the parameter type of the lambda is not known,
+typecasting is required.
+
+Graph System Integration
+------------------------
+
+image:provider-integration.png[width=395,float=right] TinkerPop is a framework composed of various interoperable
+components. At the foundation there is the <<graph,core TinkerPop3 API>> which defines what a `Graph`, `Vertex`,
+`Edge`, etc. are. At minimum a graph system provider must implement the core API. Once implemented, the Gremlin
+<<traversal,traversal language>> is available to the graph system's users. However, the provider can go further and
+develop specific <<traversalstrategy,`TraversalStrategy`>> optimizations that allow the graph system to inspect a
+Gremlin query at runtime and optimize it for its particular implementation (e.g. index lookups, step reordering). If
+the graph system is a graph processor (i.e. provides OLAP capabilities), the system should implement the
+<<graphcomputer,`GraphComputer`>> API. This API defines how messages/traversers are passed between communicating
+workers (i.e. threads and/or machines). Once implemented, the same Gremlin traversals execute against both the graph
+database (OLTP) and the graph processor (OLAP). Note that the Gremlin language interprets the graph in terms of
+vertices and edges -- i.e. Gremlin is a graph-based domain specific language. Users can create their own domain
+specific languages to process the graph in terms of higher-order constructs such as people, companies, and their
+various relationships. Finally, <<gremlin-server,Gremlin Server>> can be leveraged to allow over the wire
+communication with the TinkerPop-enabled graph system. Gremlin Server provides a configurable communication interface
+along with metrics and monitoring capabilities. In total, this is The TinkerPop.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/reference/preface.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/reference/preface.asciidoc b/docs/src/reference/preface.asciidoc
new file mode 100644
index 0000000..142f935
--- /dev/null
+++ b/docs/src/reference/preface.asciidoc
@@ -0,0 +1,97 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+[[preface]]
+TinkerPop3 Documentation
+========================
+
+In the beginning...
+
+TinkerPop0
+----------
+
+Gremlin realized. The more he did so, the more ideas he created. The more ideas he created, the more they related.
+Into a concatenation of that which he accepted wholeheartedly and that which perhaps may ultimately come to be through
+concerted will, a world took form which was seemingly separate from his own realization of it. However, the world
+birthed could not bear its own weight without the logic Gremlin had come to accept -- the logic of left is not right,
+up not down, and west far from east unless one goes the other way. Gremlin's realization required Gremlin's
+realization. Perhaps, the world is simply an idea that he once had -- The TinkerPop.
+
+image::gremlin-logo.png[width=300]
+
+TinkerPop1
+----------
+
+What is The TinkerPop? Where is The TinkerPop? Who is The TinkerPop? When is The TinkerPop?. The more he wonder, the
+more these thoughts blurred into a seeming identity -- distinctions unclear. Unwilling to accept the morass of the
+maze he wandered, Gremlin crafted a collection of machines to help hold the fabric together: Blueprints, Pipes,
+Frames, Furnace, and Rexster. With their help, could he stave off the thought he was not ready to have? Could he hold
+back The TinkerPop by searching for The TinkerPop?
+
+ "If I haven't found it, it is not here and now."
+
+image::gremlin-and-friends.png[width=500]
+
+Upon their realization of existence, the machines turned to their link:http://non-aliencreatures.wikia.com/wiki/Machine_Elf[machine elf] creator and asked:
+
+ "Why am I what I am?"
+
+Gremlin responded:
+
+ "You will help me realize the ultimate realization -- The TinkerPop. The world you find yourself in and the logic
+ that allows you to move about it is because of the TinkerPop."
+
+The machines wondered:
+
+ "If what is is the TinkerPop, then perhaps we are The TinkerPop and our realization is simply the realization of
+ the TinkerPop?"
+
+Would the machines, by their very nature of realizing The TinkerPop, be The TinkerPop? Or, on the same side of the
+coin, do the machines simply provide the scaffolding by which Gremlin's world sustains itself and yielding its
+justification by means of the word "The TinkerPop?" Regardless, it all turns out the same -- The TinkerPop.
+
+TinkerPop2
+----------
+
+Gremlin spoke:
+
+ "Please listen to what I have to say. I am no closer to The TinkerPop. However, all along The TinkerPop has
+ espoused the form I willed upon it... this is the same form I have willed upon you, my machine friends. Let me
+ train you in the ways of my thought such that it can continue indefinitely."
+
+image::tinkerpop-reading.png[width=450]
+
+The machines, simply moving algorithmically through Gremlin's world, endorsed his logic. Gremlin labored to make them
+more efficient, more expressive, better capable of reasoning upon his thoughts. Faster, quickly, now towards the
+world's end, where there would be forever currently, emanatingly engulfing that which is -- The TinkerPop.
+
+TinkerPop3
+----------
+
+image::tinkerpop3-splash.png[width=450]
+
+Gremlin approached The TinkerPop. The closer he got, the more his world dissolved -- west is right, around is
+straight, and form nothing more than nothing. With each step towards The TinkerPop, more and more of all the other
+worlds made possible were laid upon his paradoxed mind. Everything is everything in The TinkerPop, and when the dust
+settled, Gremlin emerged Gremlitron. He realized that all that he realized was just a realization and that all
+realized realizations are just as real. For that is -- The TinkerPop.
+
+image::gremlintron.png[width=400]
+
+NOTE: TinkerPop2 and below made a sharp distinction between the various TinkerPop projects: Blueprints, Pipes,
+Gremlin, Frames, Furnace, and Rexster. With TinkerPop3, all of these projects have been merged and are generally
+known as Gremlin. *Blueprints* -> Gremlin Structure API : *Pipes* -> `GraphTraversal` : *Frames* -> `Traversal` :
+*Furnace* -> `GraphComputer` and `VertexProgram` : *Rexster* -> GremlinServer.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/reference/the-graph.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/reference/the-graph.asciidoc b/docs/src/reference/the-graph.asciidoc
new file mode 100644
index 0000000..eceec21
--- /dev/null
+++ b/docs/src/reference/the-graph.asciidoc
@@ -0,0 +1,771 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+[[graph]]
+The Graph
+=========
+
+image::gremlin-standing.png[width=125]
+
+Features
+--------
+
+A `Feature` implementation describes the capabilities of a `Graph` instance. This interface is implemented by graph
+system providers for two purposes:
+
+. It tells users the capabilities of their `Graph` instance.
+. It allows the features they do comply with to be tested against the Gremlin Test Suite - tests that do not comply are "ignored").
+
+The following example in the Gremlin Console shows how to print all the features of a `Graph`:
+
+[gremlin-groovy]
+----
+graph = TinkerGraph.open()
+graph.features()
+----
+
+A common pattern for using features is to check their support prior to performing an operation:
+
+[gremlin-groovy]
+----
+graph.features().graph().supportsTransactions()
+graph.features().graph().supportsTransactions() ? g.tx().commit() : "no tx"
+----
+
+TIP: To ensure provider agnostic code, always check feature support prior to usage of a particular function. In that
+way, the application can behave gracefully in case a particular implementation is provided at runtime that does not
+support a function being accessed.
+
+WARNING: Assignments of a `GraphStrategy` can alter the base features of a `Graph` in dynamic ways, such that checks
+against a `Feature` may not always reflect the behavior exhibited when the `GraphStrategy` is in use.
+
+[[vertex-properties]]
+Vertex Properties
+-----------------
+
+image:vertex-properties.png[width=215,float=left] TinkerPop3 introduces the concept of a `VertexProperty<V>`. All the
+properties of a `Vertex` are a `VertexProperty`. A `VertexProperty` implements `Property` and as such, it has a
+key/value pair. However, `VertexProperty` also implements `Element` and thus, can have a collection of key/value
+pairs. Moreover, while an `Edge` can only have one property of key "name" (for example), a `Vertex` can have multiple
+"name" properties. With the inclusion of vertex properties, two features are introduced which ultimately advance the
+graph modelers toolkit:
+
+. Multiple properties (*multi-properties*): a vertex property key can have multiple values (i.e. a vertex can have
+multiple "name" properties).
+. Properties on properties (*meta-properties*): a vertex property can have properties (i.e. a vertex property can
+have key/value data associated with it).
+
+A collection of use cases are itemized below:
+
+. *Permissions*: Vertex properties can have key/value ACL-type permission information associated with them.
+. *Auditing*: When a vertex property is manipulated, it can have key/value information attached to it saying who the
+creator, deletor, etc. are.
+. *Provenance*: The "name" of a vertex can be declared by multiple users.
+
+A running example using vertex properties is provided below to demonstrate and explain the API.
+
+[gremlin-groovy]
+----
+graph = TinkerGraph.open()
+g = graph.traversal(standard())
+v = g.addV('name','marko','name','marko a. rodriguez').next()
+g.V(v).properties().count()
+g.V(v).properties('name').count() <1>
+g.V(v).properties()
+g.V(v).properties('name')
+g.V(v).properties('name').hasValue('marko')
+g.V(v).properties('name').hasValue('marko').property('acl','private') <2>
+g.V(v).properties('name').hasValue('marko a. rodriguez')
+g.V(v).properties('name').hasValue('marko a. rodriguez').property('acl','public')
+g.V(v).properties('name').has('acl','public').value()
+g.V(v).properties('name').has('acl','public').drop() <3>
+g.V(v).properties('name').has('acl','public').value()
+g.V(v).properties('name').has('acl','private').value()
+g.V(v).properties()
+g.V(v).properties().properties() <4>
+g.V(v).properties().property('date',2014) <5>
+g.V(v).properties().property('creator','stephen')
+g.V(v).properties().properties()
+g.V(v).properties('name').valueMap()
+g.V(v).property('name','okram') <6>
+g.V(v).properties('name')
+g.V(v).values('name') <7>
+----
+
+<1> A vertex can have zero or more properties with the same key associated with it.
+<2> A vertex property can have standard key/value properties attached to it.
+<3> Vertex property removal is identical to property removal.
+<4> It is property to get the properties of a vertex property.
+<5> A vertex property can have any number of key/value properties attached to it.
+<6> `property(...)` will remove all existing key'd properties before adding the new single property (see `VertexProperty.Cardinality`).
+<7> If only the value of a property is needed, then `values()` can be used.
+
+If the concept of vertex properties is difficult to grasp, then it may be best to think of vertex properties in terms
+of "literal vertices." A vertex can have an edge to a "literal vertex" that has a single value key/value -- e.g.
+"value=okram." The edge that points to that literal vertex has an edge-label of "name." The properties on the edge
+represent the literal vertex's properties. The "literal vertex" can not have any other edges to it (only one from the
+associated vertex).
+
+[[the-crew-toy-graph]]
+TIP: A toy graph demonstrating all of the new TinkerPop3 graph structure features is available at
+`TinkerFactory.createTheCrew()` and `data/tinkerpop-crew*`. This graph demonstrates multi-properties and meta-properties.
+
+.TinkerPop Crew
+image::the-crew-graph.png[width=685]
+
+[gremlin-groovy,theCrew]
+----
+g.V().as('a').
+ properties('location').as('b').
+ hasNot('endTime').as('c').
+ select('a','b','c').by('name').by(value).by('startTime') // determine the current location of each person
+g.V().has('name','gremlin').inE('uses').
+ order().by('skill',incr).as('a').
+ outV().as('b').
+ select('a','b').by('skill').by('name') // rank the users of gremlin by their skill level
+----
+
+Graph Variables
+---------------
+
+TinkerPop3 introduces the concept of `Graph.Variables`. Variables are key/value pairs associated with the graph
+itself -- in essence, a `Map<String,Object>`. These variables are intended to store metadata about the graph. Example
+use cases include:
+
+ * *Schema information*: What do the namespace prefixes resolve to and when was the schema last modified?
+ * *Global permissions*: What are the access rights for particular groups?
+ * *System user information*: Who are the admins of the system?
+
+An example of graph variables in use is presented below:
+
+[gremlin-groovy]
+----
+graph = TinkerGraph.open()
+graph.variables()
+graph.variables().set('systemAdmins',['stephen','peter','pavel'])
+graph.variables().set('systemUsers',['matthias','marko','josh'])
+graph.variables().keys()
+graph.variables().get('systemUsers')
+graph.variables().get('systemUsers').get()
+graph.variables().remove('systemAdmins')
+graph.variables().keys()
+----
+
+IMPORTANT: Graph variables are not intended to be subject to heavy, concurrent mutation nor to be used in complex
+computations. The intention is to have a location to store data about the graph for administrative purposes.
+
+[[transactions]]
+Graph Transactions
+------------------
+
+image:gremlin-coins.png[width=100,float=right] A link:http://en.wikipedia.org/wiki/Database_transaction[database transaction]
+represents a unit of work to execute against the database. Transactions are controlled by an implementation of the
+`Transaction` interface and that object can be obtained from the `Graph` interface using the `tx()` method. It is
+important to note that the `Transaction` object does not represent a "transaction" itself. It merely exposes the
+methods for working with transactions (e.g. committing, rolling back, etc).
+
+Most `Graph` implementations that `supportsTransactions` will implement an "automatic" `ThreadLocal` transaction,
+which means that when a read or write occurs after the `Graph` is instantiated a transaction is automatically
+started within that thread. There is no need to manually call a method to "create" or "start" a transaction. Simply
+modify the graph as required and call `graph.tx().commit()` to apply changes or `graph.tx().rollback()` to undo them.
+When the next read or write action occurs against the graph, a new transaction will be started within that current
+thread of execution.
+
+When using transactions in this fashion, especially in web application (e.g. REST server), it is important to ensure
+that transaction do not leak from one request to the next. In other words, unless a client is somehow bound via
+session to process every request on the same server thread, ever request must be committed or rolled back at the end
+of the request. By ensuring that the request encapsulates a transaction, it ensures that a future request processed
+on a server thread is starting in a fresh transactional state and will not have access to the remains of one from an
+earlier request. A good strategy is to rollback a transaction at the start of a request, so that if it so happens that
+a transactional leak does occur between requests somehow, a fresh transaction is assured by the fresh request.
+
+TIP: The `tx()` method is on the `Graph` interface, but it is also available on the `TraversalSource` spawned from a
+`Graph`. Calls to `TraversalSource.tx()` are proxied through to the underlying `Graph` as a convenience.
+
+Configuring
+~~~~~~~~~~~
+
+Determining when a transaction starts is dependent upon the behavior assigned to the `Transaction`. It is up to the
+`Graph` implementation to determine the default behavior and unless the implementation doesn't allow it, the behavior
+itself can be altered via these `Transaction` methods:
+
+[source,java]
+----
+public Transaction onReadWrite(final Consumer<Transaction> consumer);
+
+public Transaction onClose(final Consumer<Transaction> consumer);
+----
+
+Providing a `Consumer` function to `onReadWrite` allows definition of how a transaction starts when a read or a write
+occurs. `Transaction.READ_WRITE_BEHAVIOR` contains pre-defined `Consumer` functions to supply to the `onReadWrite`
+method. It has two options:
+
+* `AUTO` - automatic transactions where the transaction is started implicitly to the read or write operation
+* `MANUAL` - manual transactions where it is up to the user to explicitly open a transaction, throwing an exception
+if the transaction is not open
+
+Providing a `Consumer` function to `onClose` allows configuration of how a transaction is handled when
+`Transaction.close()` is called. `Transaction.CLOSE_BEHAVIOR` has several pre-defined options that can be supplied to
+this method:
+
+* `COMMIT` - automatically commit an open transaction
+* `ROLLBACK` - automatically rollback an open transaction
+* `MANUAL` - throw an exception if a transaction is open, forcing the user to explicitly close the transaction
+
+IMPORTANT: As transactions are `ThreadLocal` in nature, so are the transaction configurations for `onReadWrite` and
+`onClose`.
+
+Once there is an understanding for how transactions are configured, most of the rest of the `Transaction` interface
+is self-explanatory. Note that <<neo4j-gremlin,Neo4j-Gremlin>> is used for the examples to follow as TinkerGraph does
+not support transactions.
+
+[source,groovy]
+----
+gremlin> graph = Neo4jGraph.open('/tmp/neo4j')
+==>neo4jgraph[EmbeddedGraphDatabase [/tmp/neo4j]]
+gremlin> graph.features()
+==>FEATURES
+> GraphFeatures
+>-- Transactions: true <1>
+>-- Computer: false
+>-- Persistence: true
+...
+gremlin> graph.tx().onReadWrite(Transaction.READ_WRITE_BEHAVIOR.AUTO) <2>
+==>org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jGraph$Neo4jTransaction@1c067c0d
+gremlin> graph.addVertex("name","stephen") <3>
+==>v[0]
+gremlin> graph.tx().commit() <4>
+==>null
+gremlin> graph.tx().onReadWrite(Transaction.READ_WRITE_BEHAVIOR.MANUAL) <5>
+==>org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jGraph$Neo4jTransaction@1c067c0d
+gremlin> graph.tx().isOpen()
+==>false
+gremlin> graph.addVertex("name","marko") <6>
+Open a transaction before attempting to read/write the transaction
+gremlin> graph.tx().open() <7>
+==>null
+gremlin> graph.addVertex("name","marko") <8>
+==>v[1]
+gremlin> graph.tx().commit()
+==>null
+----
+
+<1> Check `features` to ensure that the graph supports transactions.
+<2> By default, `Neo4jGraph` is configured with "automatic" transactions, so it is set here for demonstration purposes only.
+<3> When the vertex is added, the transaction is automatically started. From this point, more mutations can be staged
+or other read operations executed in the context of that open transaction.
+<4> Calling `commit` finalizes the transaction.
+<5> Change transaction behavior to require manual control.
+<6> Adding a vertex now results in failure because the transaction was not explicitly opened.
+<7> Explicitly open a transaction.
+<8> Adding a vertex now succeeds as the transaction was manually opened.
+
+NOTE: It may be important to consult the documentation of the `Graph` implementation when it comes to the specifics of
+how transactions will behave. TinkerPop allows some latitude in this area and implementations may not have the exact
+same behaviors and link:https://en.wikipedia.org/wiki/ACID[ACID] guarantees.
+
+Retries
+~~~~~~~
+
+There are times when transactions fail. Failure may be indicative of some permanent condition, but other failures
+might simply require the transaction to be retried for possible future success. The `Transaction` object also exposes
+a method for executing automatic transaction retries:
+
+[gremlin-groovy]
+----
+graph = Neo4jGraph.open('/tmp/neo4j')
+graph.tx().submit {it.addVertex("name","josh")}.retry(10)
+graph.tx().submit {it.addVertex("name","daniel")}.exponentialBackoff(10)
+graph.close()
+----
+
+As shown above, the `submit` method takes a `Function<Graph, R>` which is the unit of work to execute and possibly
+retry on failure. The method returns a `Transaction.Workload` object which has a number of default methods for common
+retry strategies. It is also possible to supply a custom retry function if a default one does not suit the required
+purpose.
+
+Threaded Transactions
+~~~~~~~~~~~~~~~~~~~~~
+
+Most `Graph` implementations that support transactions do so in a `ThreadLocal` manner, where the current transaction
+is bound to the current thread of execution. Consider the following example to demonstrate:
+
+[source,java]
+----
+graph.addVertex("name","stephen");
+
+Thread t1 = new Thread(() -> {
+ graph.addVertex("name","josh");
+});
+
+Thread t2 = new Thread(() -> {
+ graph.addVertex("name","marko");
+});
+
+t1.start()
+t2.start()
+
+t1.join()
+t2.join()
+
+graph.tx().commit();
+----
+
+The above code shows three vertices added to `graph` in three different threads: the current thread, `t1` and
+`t2`. One might expect that by the time this body of code finished executing, that there would be three vertices
+persisted to the `Graph`. However, given the `ThreadLocal` nature of transactions, there really were three separate
+transactions created in that body of code (i.e. one for each thread of execution) and the only one committed was the
+first call to `addVertex` in the primary thread of execution. The other two calls to that method within `t1` and `t2`
+were never committed and thus orphaned.
+
+A `Graph` that `supportsThreadedTransactions` is one that allows for a `Graph` to operate outside of that constraint,
+thus allowing multiple threads to operate within the same transaction. Therefore, if there was a need to have three
+different threads operating within the same transaction, the above code could be re-written as follows:
+
+[source,java]
+----
+Graph threaded = graph.tx().newThreadedTx();
+threaded.addVertex("name","stephen");
+
+Thread t1 = new Thread(() -> {
+ threaded.addVertex("name","josh");
+});
+
+Thread t2 = new Thread(() -> {
+ threaded.addVertex("name","marko");
+});
+
+t1.start()
+t2.start()
+
+t1.join()
+t2.join()
+
+threaded.tx().commit();
+----
+
+In the above case, the call to `graph.tx().newThreadedTx()` creates a new `Graph` instance that is unbound from the
+`ThreadLocal` transaction, thus allowing each thread to operate on it in the same context. In this case, there would
+be three separate vertices persisted to the `Graph`.
+
+Gremlin I/O
+-----------
+
+image:gremlin-io.png[width=250,float=right] The task of getting data in and out of `Graph` instances is the job of
+the Gremlin I/O packages. Gremlin I/O provides two interfaces for reading and writing `Graph` instances: `GraphReader`
+and `GraphWriter`. These interfaces expose methods that support:
+
+* Reading and writing an entire `Graph`
+* Reading and writing a `Traversal<Vertex>` as adjacency list format
+* Reading and writing a single `Vertex` (with and without associated `Edge` objects)
+* Reading and writing a single `Edge`
+* Reading and writing a single `VertexProperty`
+* Reading and writing a single `Property`
+* Reading and writing an arbitrary `Object`
+
+In all cases, these methods operate in the currency of `InputStream` and `OutputStream` objects, allowing graphs and
+their related elements to be written to and read from files, byte arrays, etc. The `Graph` interface offers the `io`
+method, which provides access to "reader/writer builder" objects that are pre-configured with serializers provided by
+the `Graph`, as well as helper methods for the various I/O capabilities. Unless there are very advanced requirements
+for the serialization process, it is always best to utilize the methods on the `Io` interface to construct
+`GraphReader` and `GraphWriter` instances, as the implementation may provide some custom settings that would otherwise
+have to be configured manually by the user to do the serialization.
+
+It is up to the implementations of the `GraphReader` and `GraphWriter` interfaces to choose the methods they
+implement and the manner in which they work together. The only semantics enforced and expected is that the write
+methods should produce output that is compatible with the corresponding read method (e.g. the output of
+`writeVertices` should be readable as input to `readVertices` and the output of `writeProperty` should be readable as
+input to `readProperty`).
+
+GraphML Reader/Writer
+~~~~~~~~~~~~~~~~~~~~~
+
+image:gremlin-graphml.png[width=350,float=left] The link:http://graphml.graphdrawing.org/[GraphML] file format is a
+common XML-based representation of a graph. It is widely supported by graph-related tools and libraries making it a
+solid interchange format for TinkerPop. In other words, if the intent is to work with graph data in conjunction with
+applications outside of TinkerPop, GraphML may be the best choice to do that. Common use cases might be:
+
+* Generate a graph using link:https://networkx.github.io/[NetworkX], export it with GraphML and import it to TinkerPop.
+* Produce a subgraph and export it to GraphML to be consumed by and visualized in link:https://gephi.org/[Gephi].
+* Migrate the data of an entire graph to a different graph database not supported by TinkerPop.
+
+As GraphML is a specification for the serialization of an entire graph and not the individual elements of a graph,
+methods that support input and output of single vertices, edges, etc. are not supported.
+
+CAUTION: GraphML is a "lossy" format in that it only supports primitive values for properties and does not have
+support for `Graph` variables. It will use `toString` to serialize property values outside of those primitives.
+
+CAUTION: GraphML, as a specification, allows for `<edge>` and `<node>` elements to appear in any order. The
+`GraphMLReader` will support that, however, that capability comes with a limitation. TinkerPop does not allow the
+vertex label to be changed after the vertex has been created. Therefore, if an `<edge>` element comes before the
+`<node>` the label on the vertex will be ignored. It is thus better to order `<node>` elements in the GraphML to
+appear before all `<edge>` elements if vertex labels are important to the graph.
+
+The following code shows how to write a `Graph` instance to file called `tinkerpop-modern.xml` and then how to read
+that file back into a different instance:
+
+[source,java]
+----
+final Graph graph = TinkerFactory.createModern();
+graph.io(IoCore.graphml()).writeGraph("tinkerpop-modern.xml");
+final Graph newGraph = TinkerGraph.open();
+newGraph.io(IoCore.graphml()).readGraph("tinkerpop-modern.xml");
+----
+
+If a custom configuration is required, then have the `Graph` generate a `GraphReader` or `GraphWriter` "builder" instance:
+
+[source,java]
+----
+final Graph graph = TinkerFactory.createModern();
+try (final OutputStream os = new FileOutputStream("tinkerpop-modern.xml")) {
+ graph.io(IoCore.graphml()).writer().normalize(true).create().writeGraph(os, graph);
+}
+
+final Graph newGraph = TinkerGraph.open();
+try (final InputStream stream = new FileInputStream("tinkerpop-modern.xml")) {
+ newGraph.io(IoCore.graphml()).reader().vertexIdKey("name").create().readGraph(stream, newGraph);
+}
+----
+
+[[graphson-reader-writer]]
+GraphSON Reader/Writer
+~~~~~~~~~~~~~~~~~~~~~~
+
+image:gremlin-graphson.png[width=350,float=left] GraphSON is a link:http://json.org/[JSON]-based format extended
+from earlier versions of TinkerPop. It is important to note that TinkerPop3's GraphSON is not backwards compatible
+with prior TinkerPop GraphSON versions. GraphSON has some support from graph-related application outside of TinkerPop,
+but it is generally best used in two cases:
+
+* A text format of the graph or its elements is desired (e.g. debugging, usage in source control, etc.)
+* The graph or its elements need to be consumed by code that is not JVM-based (e.g. JavaScript, Python, .NET, etc.)
+
+GraphSON supports all of the `GraphReader` and `GraphWriter` interface methods and can therefore read or write an
+entire `Graph`, vertices, arbitrary objects, etc. The following code shows how to write a `Graph` instance to file
+called `tinkerpop-modern.json` and then how to read that file back into a different instance:
+
+[source,java]
+----
+final Graph graph = TinkerFactory.createModern();
+graph.io(IoCore.graphson()).writeGraph("tinkerpop-modern.json");
+
+final Graph newGraph = TinkerGraph.open();
+newGraph.io(IoCore.graphson()).readGraph("tinkerpop-modern.json");
+----
+
+If a custom configuration is required, then have the `Graph` generate a `GraphReader` or `GraphWriter` "builder" instance:
+
+[source,java]
+----
+final Graph graph = TinkerFactory.createModern();
+try (final OutputStream os = new FileOutputStream("tinkerpop-modern.json")) {
+ final GraphSONMapper mapper = graph.io(IoCore.graphson()).mapper().normalize(true).create()
+ graph.io(IoCore.graphson()).writer().mapper(mapper).create().writeGraph(os, graph)
+}
+
+final Graph newGraph = TinkerGraph.open();
+try (final InputStream stream = new FileInputStream("tinkerpop-modern.json")) {
+ newGraph.io(IoCore.graphson()).reader().vertexIdKey("name").create().readGraph(stream, newGraph);
+}
+----
+
+One of the important configuration options of the `GraphSONReader` and `GraphSONWriter` is the ability to embed type
+information into the output. By embedding the types, it becomes possible to serialize a graph without losing type
+information that might be important when being consumed by another source. The importance of this concept is
+demonstrated in the following example where a single `Vertex` is written to GraphSON using the Gremlin Console:
+
+[gremlin-groovy]
+----
+graph = TinkerFactory.createModern()
+g = graph.traversal()
+f = new FileOutputStream("vertex-1.json")
+graph.io(graphson()).writer().create().writeVertex(f, g.V(1).next(), BOTH)
+f.close()
+----
+
+The following GraphSON example shows the output of `GraphSonWriter.writeVertex()` with associated edges:
+
+[source,json]
+----
+{
+ "id": 1,
+ "label": "person",
+ "outE": {
+ "created": [
+ {
+ "id": 9,
+ "inV": 3,
+ "properties": {
+ "weight": 0.4
+ }
+ }
+ ],
+ "knows": [
+ {
+ "id": 7,
+ "inV": 2,
+ "properties": {
+ "weight": 0.5
+ }
+ },
+ {
+ "id": 8,
+ "inV": 4,
+ "properties": {
+ "weight": 1
+ }
+ }
+ ]
+ },
+ "properties": {
+ "name": [
+ {
+ "id": 0,
+ "value": "marko"
+ }
+ ],
+ "age": [
+ {
+ "id": 1,
+ "value": 29
+ }
+ ]
+ }
+}
+----
+
+The vertex properly serializes to valid JSON but note that a consuming application will not automatically know how to
+interpret the numeric values. In coercing those Java values to JSON, such information is lost.
+
+With a minor change to the construction of the `GraphSONWriter` the lossy nature of GraphSON can be avoided:
+
+[gremlin-groovy]
+----
+graph = TinkerFactory.createModern()
+g = graph.traversal()
+f = new FileOutputStream("vertex-1.json")
+mapper = graph.io(graphson()).mapper().embedTypes(true).create()
+graph.io(graphson()).writer().mapper(mapper).create().writeVertex(f, g.V(1).next(), BOTH)
+f.close()
+----
+
+In the above code, the `embedTypes` option is set to `true` and the output below shows the difference in the output:
+
+[source,json]
+----
+{
+ "@class": "java.util.HashMap",
+ "id": 1,
+ "label": "person",
+ "outE": {
+ "@class": "java.util.HashMap",
+ "created": [
+ "java.util.ArrayList",
+ [
+ {
+ "@class": "java.util.HashMap",
+ "id": 9,
+ "inV": 3,
+ "properties": {
+ "@class": "java.util.HashMap",
+ "weight": 0.4
+ }
+ }
+ ]
+ ],
+ "knows": [
+ "java.util.ArrayList",
+ [
+ {
+ "@class": "java.util.HashMap",
+ "id": 7,
+ "inV": 2,
+ "properties": {
+ "@class": "java.util.HashMap",
+ "weight": 0.5
+ }
+ },
+ {
+ "@class": "java.util.HashMap",
+ "id": 8,
+ "inV": 4,
+ "properties": {
+ "@class": "java.util.HashMap",
+ "weight": 1
+ }
+ }
+ ]
+ ]
+ },
+ "properties": {
+ "@class": "java.util.HashMap",
+ "name": [
+ "java.util.ArrayList",
+ [
+ {
+ "@class": "java.util.HashMap",
+ "id": [
+ "java.lang.Long",
+ 0
+ ],
+ "value": "marko"
+ }
+ ]
+ ],
+ "age": [
+ "java.util.ArrayList",
+ [
+ {
+ "@class": "java.util.HashMap",
+ "id": [
+ "java.lang.Long",
+ 1
+ ],
+ "value": 29
+ }
+ ]
+ ]
+ }
+}
+----
+
+The ambiguity of components of the GraphSON is now removed by the `@class` property, which contains Java class
+information for the data it is associated with. The `@class` property is used for all non-final types, with the
+exception of a small number of "natural" types (String, Boolean, Integer, and Double) which can be correctly inferred
+from JSON typing. While the output is more verbose, it comes with the security of not losing type information. While
+non-JVM languages won't be able to consume this information automatically, at least there is a hint as to how the
+values should be coerced back into the correct types in the target language.
+
+[[gryo-reader-writer]]
+Gryo Reader/Writer
+~~~~~~~~~~~~~~~~~~
+
+image:gremlin-kryo.png[width=400,float=left] link:https://github.com/EsotericSoftware/kryo[Kryo] is a popular
+serialization package for the JVM. Gremlin-Kryo is a binary `Graph` serialization format for use on the JVM by JVM
+languages. It is designed to be space efficient, non-lossy and is promoted as the standard format to use when working
+with graph data inside of the TinkerPop stack. A list of common use cases is presented below:
+
+* Migration from one Gremlin Structure implementation to another (e.g. `TinkerGraph` to `Neo4jGraph`)
+* Serialization of individual graph elements to be sent over the network to another JVM.
+* Backups of in-memory graphs or subgraphs.
+
+CAUTION: When migrating between Gremlin Structure implementations, Kryo may not lose data, but it is important to
+consider the features of each `Graph` and whether or not the data types supported in one will be supported in the
+other. Failure to do so, may result in errors.
+
+Kryo supports all of the `GraphReader` and `GraphWriter` interface methods and can therefore read or write an entire
+`Graph`, vertices, edges, etc. The following code shows how to write a `Graph` instance to file called
+`tinkerpop-modern.kryo` and then how to read that file back into a different instance:
+
+[source,java]
+----
+final Graph graph = TinkerFactory.createModern();
+graph.io(IoCore.gryo()).writeGraph("tinkerpop-modern.kryo");
+
+final Graph newGraph = TinkerGraph.open();
+newGraph.io(IoCore.gryo()).readGraph("tinkerpop-modern.kryo")'
+----
+
+If a custom configuration is required, then have the `Graph` generate a `GraphReader` or `GraphWriter` "builder" instance:
+
+[source,java]
+----
+final Graph graph = TinkerFactory.createModern();
+try (final OutputStream os = new FileOutputStream("tinkerpop-modern.kryo")) {
+ graph.io(IoCore.gryo()).writer().create().writeGraph(os, graph);
+}
+
+final Graph newGraph = TinkerGraph.open();
+try (final InputStream stream = new FileInputStream("tinkerpop-modern.kryo")) {
+ newGraph.io(IoCore.gryo()).reader().vertexIdKey("name").create().readGraph(stream, newGraph);
+}
+----
+
+NOTE: The preferred extension for files names produced by Gryo is `.kryo`.
+
+TinkerPop2 Data Migration
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+image:data-migration.png[width=300,float=right] For those using TinkerPop2, migrating to TinkerPop3 will mean a number
+of programming changes, but may also require a migration of the data depending on the graph implementation. For
+example, trying to open `TinkerGraph` data from TinkerPop2 with TinkerPop3 code will not work, however opening a
+TinkerPop2 `Neo4jGraph` with a TinkerPop3 `Neo4jGraph` should work provided there aren't Neo4j version compatibility
+mismatches preventing the read.
+
+If such a situation arises that a particular TinkerPop2 `Graph` can not be read by TinkerPop3, a "legacy" data
+migration approach exists. The migration involves writing the TinkerPop2 `Graph` to GraphSON, then reading it to
+TinkerPop3 with the `LegacyGraphSONReader` (a limited implementation of the `GraphReader` interface).
+
+The following represents an example migration of the "classic" toy graph. In this example, the "classic" graph is
+saved to GraphSON using TinkerPop2.
+
+[source,groovy]
+----
+gremlin> Gremlin.version()
+==>2.5.z
+gremlin> graph = TinkerGraphFactory.createTinkerGraph()
+==>tinkergraph[vertices:6 edges:6]
+gremlin> GraphSONWriter.outputGraph(graph,'/tmp/tp2.json',GraphSONMode.EXTENDED)
+==>null
+----
+
+The above console session uses the `gremlin-groovy` distribution from TinkerPop2. It is important to generate the
+`tp2.json` file using the `EXTENDED` mode as it will include data types when necessary which will help limit
+"lossiness" on the TinkerPop3 side when imported. Once `tp2.json` is created, it can then be imported to a TinkerPop3
+`Graph`.
+
+[source,groovy]
+----
+gremlin> Gremlin.version()
+==>x.y.z
+gremlin> graph = TinkerGraph.open()
+==>tinkergraph[vertices:0 edges:0]
+gremlin> r = LegacyGraphSONReader.build().create()
+==>org.apache.tinkerpop.gremlin.structure.io.graphson.LegacyGraphSONReader@64337702
+gremlin> r.readGraph(new FileInputStream('/tmp/tp2.json'), graph)
+==>null
+gremlin> g = graph.traversal(standard())
+==>graphtraversalsource[tinkergraph[vertices:6 edges:6], standard]
+gremlin> g.E()
+==>e[11][4-created->3]
+==>e[12][6-created->3]
+==>e[7][1-knows->2]
+==>e[8][1-knows->4]
+==>e[9][1-created->3]
+==>e[10][4-created->5]
+----
+
+Namespace Conventions
+---------------------
+
+End users, <<implementations,graph system providers>>, <<graphcomputer,`GraphComputer`>> algorithm designers,
+<<gremlin-plugins,GremlinPlugin>> creators, etc. all leverage properties on elements to store information. There are
+a few conventions that should be respected when naming property keys to ensure that conflicts between these
+stakeholders do not conflict.
+
+* End users are granted the _flat namespace_ (e.g. `name`, `age`, `location`) to key their properties and label their elements.
+* Graph system providers are granted the _hidden namespace_ (e.g. `~metadata`) to key their properties and labels.
+Data keyed as such is only accessible via the graph system implementation and no other stakeholders are granted read
+nor write access to data prefixed with "~" (see `Graph.Hidden`). Test coverage and exceptions exist to ensure that
+graph systems respect this hard boundary.
+* <<vertexprogram,`VertexProgram`>> and <<mapreduce,`MapReduce`>> developers should, like `GraphStrategy` developers,
+leverage _qualified namespaces_ particular to their domain (e.g. `mydomain.myvertexprogram.computedata`).
+* `GremlinPlugin` creators should prefix their plugin name with their domain (e.g. `mydomain.myplugin`).
+
+IMPORTANT: TinkerPop uses `tinkerpop.` and `gremlin.` as the prefixes for provided strategies, vertex programs, map
+reduce implementations, and plugins.
+
+The only truly protected namespace is the _hidden namespace_ provided to graph systems. From there, its up to
+engineers to respect the namespacing conventions presented.
[09/22] incubator-tinkerpop git commit: Made subdirectories for
various "books" in the docs.
Posted by sp...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/reference/implementations.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/reference/implementations.asciidoc b/docs/src/reference/implementations.asciidoc
new file mode 100644
index 0000000..b5c2c4d
--- /dev/null
+++ b/docs/src/reference/implementations.asciidoc
@@ -0,0 +1,1764 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+[[implementations]]
+Implementations
+===============
+
+image::gremlin-racecar.png[width=325]
+
+[[graph-system-provider-requirements]]
+Graph System Provider Requirements
+----------------------------------
+
+image:tinkerpop-enabled.png[width=140,float=left] At the core of TinkerPop3 is a Java8 API. The implementation of this
+core API and its validation via the `gremlin-test` suite is all that is required of a graph system provider wishing to
+provide a TinkerPop3-enabled graph engine. Once a graph system has a valid implementation, then all the applications
+provided by TinkerPop (e.g. Gremlin Console, Gremlin Server, etc.) and 3rd-party developers (e.g. Gremlin-Scala,
+Gremlin-JS, etc.) will integrate properly. Finally, please feel free to use the logo on the left to promote your
+TinkerPop3 implementation.
+
+Implementing Gremlin-Core
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The classes that a graph system provider should focus on implementing are itemized below. It is a good idea to study
+the <<tinkergraph-gremlin,TinkerGraph>> (in-memory OLTP and OLAP in `tinkergraph-gremlin`), <<neo4j-gremlin,Neo4jGraph>>
+(OTLP w/ transactions in `neo4j-gremlin`) and/or <<hadoop-gremlin,HadoopGraph>> (OLAP in `hadoop-gremlin`)
+implementations for ideas and patterns.
+
+. Online Transactional Processing Graph Systems (*OLTP*)
+ .. Structure API: `Graph`, `Element`, `Vertex`, `Edge`, `Property` and `Transaction` (if transactions are supported).
+ .. Process API: `TraversalStrategy` instances for optimizing Gremlin traversals to the provider's graph system (i.e. `TinkerGraphStepStrategy`).
+. Online Analytics Processing Graph Systems (*OLAP*)
+ .. Everything required of OTLP is required of OLAP (but not vice versa).
+ .. GraphComputer API: `GraphComputer`, `Messenger`, `Memory`.
+
+Please consider the following implementation notes:
+
+* Be sure your `Graph` implementation is named as `XXXGraph` (e.g. TinkerGraph, Neo4jGraph, HadoopGraph, etc.).
+* Use `StringHelper` to ensuring that the `toString()` representation of classes are consistent with other implementations.
+* Ensure that your implementation's `Features` (Graph, Vertex, etc.) are correct so that test cases handle particulars accordingly.
+* Use the numerous static method helper classes such as `ElementHelper`, `GraphComputerHelper`, `VertexProgramHelper`, etc.
+* There are a number of default methods on the provided interfaces that are semantically correct. However, if they are
+not efficient for the implementation, override them.
+* Implement the `structure/` package interfaces first and then, if desired, interfaces in the `process/` package interfaces.
+* `ComputerGraph` is a `Wrapper` system that ensure proper semantics during a GraphComputer computation.
+
+[[oltp-implementations]]
+OLTP Implementations
+^^^^^^^^^^^^^^^^^^^^
+
+image:pipes-character-1.png[width=110,float=right] The most important interfaces to implement are in the `structure/`
+package. These include interfaces like Graph, Vertex, Edge, Property, Transaction, etc. The `StructureStandardSuite`
+will ensure that the semantics of the methods implemented are correct. Moreover, there are numerous `Exceptions`
+classes with static exceptions that should be thrown by the graph system so that all the exceptions and their
+messages are consistent amongst all TinkerPop3 implementations.
+
+[[olap-implementations]]
+OLAP Implementations
+^^^^^^^^^^^^^^^^^^^^
+
+image:furnace-character-1.png[width=110,float=right] Implementing the OLAP interfaces may be a bit more complicated.
+Note that before OLAP interfaces are implemented, it is necessary for the OLTP interfaces to be, at minimal,
+implemented as specified in <<oltp-implementations,OLTP Implementations>>. A summary of each required interface
+implementation is presented below:
+
+. `GraphComputer`: A fluent builder for specifying an isolation level, a VertexProgram, and any number of MapReduce jobs to be submitted.
+. `Memory`: A global blackboard for ANDing, ORing, INCRing, and SETing values for specified keys.
+. `Messenger`: The system that collects and distributes messages being propagated by vertices executing the VertexProgram application.
+. `MapReduce.MapEmitter`: The system that collects key/value pairs being emitted by the MapReduce applications map-phase.
+. `MapReduce.ReduceEmitter`: The system that collects key/value pairs being emitted by the MapReduce applications combine- and reduce-phases.
+
+NOTE: The VertexProgram and MapReduce interfaces in the `process/computer/` package are not required by the graph
+system. Instead, these are interfaces to be implemented by application developers writing VertexPrograms and MapReduce jobs.
+
+IMPORTANT: TinkerPop3 provides three OLAP implementations: <<tinkergraph-gremlin,TinkerGraphComputer>> (TinkerGraph),
+<<giraphgraphcomputer,GiraphGraphComputer>> (HadoopGraph), and <<sparkgraphcomputer,`SparkGraphComputer`>> (Hadoop).
+Given the complexity of the OLAP system, it is good to study and copy many of the patterns used in these reference
+implementations.
+
+Implementing GraphComputer
+++++++++++++++++++++++++++
+
+image:furnace-character-3.png[width=150,float=right] The most complex method in GraphComputer is the `submit()`-method. The method must do the following:
+
+. Ensure the the GraphComputer has not already been executed.
+. Ensure that at least there is a VertexProgram or 1 MapReduce job.
+. If there is a VertexProgram, validate that it can execute on the GraphComputer given the respectively defined features.
+. Create the Memory to be used for the computation.
+. Execute the VertexProgram.setup() method once and only once.
+. Execute the VertexProgram.execute() method for each vertex.
+. Execute the VertexProgram.terminate() method once and if true, repeat VertexProgram.execute().
+. When VertexProgram.terminate() returns true, move to MapReduce job execution.
+. MapReduce jobs are not required to be executed in any specified order.
+. For each Vertex, execute MapReduce.map(). Then (if defined) execute MapReduce.combine() and MapReduce.reduce().
+. Update Memory with runtime information.
+. Construct a new `ComputerResult` containing the compute Graph and Memory.
+
+Implementing Memory
++++++++++++++++++++
+
+image:gremlin-brain.png[width=175,float=left] The Memory object is initially defined by `VertexProgram.setup()`.
+The memory data is available in the first round of the `VertexProgram.execute()` method. Each Vertex, when executing
+the VertexProgram, can update the Memory in its round. However, the update is not seen by the other vertices until
+the next round. At the end of the first round, all the updates are aggregated and the new memory data is available
+on the second round. This process repeats until the VertexProgram terminates.
+
+Implementing Messenger
+++++++++++++++++++++++
+
+The Messenger object is similar to the Memory object in that a vertex can read and write to the Messenger. However,
+the data it reads are the messages sent to the vertex in the previous step and the data it writes are the messages
+that will be readable by the receiving vertices in the subsequent round.
+
+Implementing MapReduce Emitters
++++++++++++++++++++++++++++++++
+
+image:hadoop-logo-notext.png[width=150,float=left] The MapReduce framework in TinkerPop3 is similar to the model
+popularized by link:http://apache.hadoop.org[Hadoop]. The primary difference is that all Mappers process the vertices
+of the graph, not an arbitrary key/value pair. However, the vertices' edges can not be accessed -- only their
+properties. This greatly reduces the amount of data needed to be pushed through the MapReduce engine as any edge
+information required, can be computed in the VertexProgram.execute() method. Moreover, at this stage, vertices can
+not be mutated, only their token and property data read. A Gremlin OLAP system needs to provide implementations for
+to particular classes: `MapReduce.MapEmitter` and `MapReduce.ReduceEmitter`. TinkerGraph's implementation is provided
+below which demonstrates the simplicity of the algorithm (especially when the data is all within the same JVM).
+
+[source,java]
+----
+public class TinkerMapEmitter<K, V> implements MapReduce.MapEmitter<K, V> {
+
+ public Map<K, Queue<V>> reduceMap;
+ public Queue<KeyValue<K, V>> mapQueue;
+ private final boolean doReduce;
+
+ public TinkerMapEmitter(final boolean doReduce) { <1>
+ this.doReduce = doReduce;
+ if (this.doReduce)
+ this.reduceMap = new ConcurrentHashMap<>();
+ else
+ this.mapQueue = new ConcurrentLinkedQueue<>();
+ }
+
+ @Override
+ public void emit(K key, V value) {
+ if (this.doReduce)
+ this.reduceMap.computeIfAbsent(key, k -> new ConcurrentLinkedQueue<>()).add(value); <2>
+ else
+ this.mapQueue.add(new KeyValue<>(key, value)); <3>
+ }
+
+ protected void complete(final MapReduce<K, V, ?, ?, ?> mapReduce) {
+ if (!this.doReduce && mapReduce.getMapKeySort().isPresent()) { <4>
+ final Comparator<K> comparator = mapReduce.getMapKeySort().get();
+ final List<KeyValue<K, V>> list = new ArrayList<>(this.mapQueue);
+ Collections.sort(list, Comparator.comparing(KeyValue::getKey, comparator));
+ this.mapQueue.clear();
+ this.mapQueue.addAll(list);
+ } else if (mapReduce.getMapKeySort().isPresent()) {
+ final Comparator<K> comparator = mapReduce.getMapKeySort().get();
+ final List<Map.Entry<K, Queue<V>>> list = new ArrayList<>();
+ list.addAll(this.reduceMap.entrySet());
+ Collections.sort(list, Comparator.comparing(Map.Entry::getKey, comparator));
+ this.reduceMap = new LinkedHashMap<>();
+ list.forEach(entry -> this.reduceMap.put(entry.getKey(), entry.getValue()));
+ }
+ }
+}
+----
+
+<1> If the MapReduce job has a reduce, then use one data structure (`reduceMap`), else use another (`mapList`). The
+difference being that a reduction requires a grouping by key and therefore, the `Map<K,Queue<V>>` definition. If no
+reduction/grouping is required, then a simple `Queue<KeyValue<K,V>>` can be leveraged.
+<2> If reduce is to follow, then increment the Map with a new value for the key. `MapHelper` is a TinkerPop3 class
+with static methods for adding data to a Map.
+<3> If no reduce is to follow, then simply append a KeyValue to the queue.
+<4> When the map phase is complete, any map-result sorting required can be executed at this point.
+
+[source,java]
+----
+public class TinkerReduceEmitter<OK, OV> implements MapReduce.ReduceEmitter<OK, OV> {
+
+ protected Queue<KeyValue<OK, OV>> reduceQueue = new ConcurrentLinkedQueue<>();
+
+ @Override
+ public void emit(final OK key, final OV value) {
+ this.reduceQueue.add(new KeyValue<>(key, value));
+ }
+
+ protected void complete(final MapReduce<?, ?, OK, OV, ?> mapReduce) {
+ if (mapReduce.getReduceKeySort().isPresent()) {
+ final Comparator<OK> comparator = mapReduce.getReduceKeySort().get();
+ final List<KeyValue<OK, OV>> list = new ArrayList<>(this.reduceQueue);
+ Collections.sort(list, Comparator.comparing(KeyValue::getKey, comparator));
+ this.reduceQueue.clear();
+ this.reduceQueue.addAll(list);
+ }
+ }
+}
+----
+
+The method `MapReduce.reduce()` is defined as:
+
+[source,java]
+public void reduce(final OK key, final Iterator<OV> values, final ReduceEmitter<OK, OV> emitter) { ... }
+
+In other words, for the TinkerGraph implementation, iterate through the entrySet of the `reduceMap` and call the
+`reduce()` method on each entry. The `reduce()` method can emit key/value pairs which are simply aggregated into a
+`Queue<KeyValue<OK,OV>>` in an analogous fashion to `TinkerMapEmitter` when no reduce is to follow. These two emitters
+are tied together in `TinkerGraphComputer.submit()`.
+
+[source,java]
+----
+...
+for (final MapReduce mapReduce : mapReducers) {
+ if (mapReduce.doStage(MapReduce.Stage.MAP)) {
+ final TinkerMapEmitter<?, ?> mapEmitter = new TinkerMapEmitter<>(mapReduce.doStage(MapReduce.Stage.REDUCE));
+ final SynchronizedIterator<Vertex> vertices = new SynchronizedIterator<>(this.graph.vertices());
+ workers.setMapReduce(mapReduce);
+ workers.mapReduceWorkerStart(MapReduce.Stage.MAP);
+ workers.executeMapReduce(workerMapReduce -> {
+ while (true) {
+ final Vertex vertex = vertices.next();
+ if (null == vertex) return;
+ workerMapReduce.map(ComputerGraph.mapReduce(vertex), mapEmitter);
+ }
+ });
+ workers.mapReduceWorkerEnd(MapReduce.Stage.MAP);
+
+ // sort results if a map output sort is defined
+ mapEmitter.complete(mapReduce);
+
+ // no need to run combiners as this is single machine
+ if (mapReduce.doStage(MapReduce.Stage.REDUCE)) {
+ final TinkerReduceEmitter<?, ?> reduceEmitter = new TinkerReduceEmitter<>();
+ final SynchronizedIterator<Map.Entry<?, Queue<?>>> keyValues = new SynchronizedIterator((Iterator) mapEmitter.reduceMap.entrySet().iterator());
+ workers.mapReduceWorkerStart(MapReduce.Stage.REDUCE);
+ workers.executeMapReduce(workerMapReduce -> {
+ while (true) {
+ final Map.Entry<?, Queue<?>> entry = keyValues.next();
+ if (null == entry) return;
+ workerMapReduce.reduce(entry.getKey(), entry.getValue().iterator(), reduceEmitter);
+ }
+ });
+ workers.mapReduceWorkerEnd(MapReduce.Stage.REDUCE);
+ reduceEmitter.complete(mapReduce); // sort results if a reduce output sort is defined
+ mapReduce.addResultToMemory(this.memory, reduceEmitter.reduceQueue.iterator()); <1>
+ } else {
+ mapReduce.addResultToMemory(this.memory, mapEmitter.mapQueue.iterator()); <2>
+ }
+ }
+}
+...
+----
+
+<1> Note that the final results of the reducer are provided to the Memory as specified by the application developer's
+`MapReduce.addResultToMemory()` implementation.
+<2> If there is no reduce stage, the the map-stage results are inserted into Memory as specified by the application
+developer's `MapReduce.addResultToMemory()` implementation.
+
+[[io-implementations]]
+IO Implementations
+^^^^^^^^^^^^^^^^^^
+
+If a `Graph` requires custom serializers for IO to work properly, implement the `Graph.io` method. A typical example
+of where a `Graph` would require such a custom serializers is if their identifier system uses non-primitive values,
+such as OrientDB's `Rid` class. From basic serialization of a single `Vertex` all the way up the stack to Gremlin
+Server, the need to know how to handle these complex identifiers is an important requirement.
+
+The first step to implementing custom serializers is to first implement the `IoRegistry` interface and register the
+custom classes and serializers to it. Each `Io` implementation has different requirements for what it expects from the
+`IoRegistry`:
+
+* *GraphML* - No custom serializers expected/allowed.
+* *GraphSON* - Register a Jackson `SimpleModule`. The `SimpleModule` encapsulates specific classes to be serialized,
+so it does not need to be registered to a specific class in the `IoRegistry` (use `null`).
+* *Gryo* - Expects registration of one of three objects:
+** Register just the custom class with a `null` Kryo `Serializer` implementation - this class will use default "field-level" Kryo serialization.
+** Register the custom class with a specific Kryo `Serializer' implementation.
+** Register the custom class with a `Function<Kryo, Serializer>` for those cases where the Kryo `Serializer` requires the `Kryo` instance to get constructed.
+
+This implementation should provide a zero-arg constructor as the stack may require instantiation via reflection.
+Consider extending `AbstractIoRegistry` for convenience as follows:
+
+[source,java]
+----
+public class MyGraphIoRegistry extends AbstractIoRegistry {
+ public MyGraphIoRegistry() {
+ register(GraphSONIo.class, null, new MyGraphSimpleModule());
+ register(GryoIo.class, MyGraphIdClass.class, new MyGraphIdSerializer());
+ }
+}
+----
+
+In the `Graph.io` method, provide the `IoRegistry` object to the supplied `Builder` and call the `create` method to
+return that `Io` instance as follows:
+
+[source,java]
+----
+public <I extends Io> I io(final Io.Builder<I> builder) {
+ return (I) builder.graph(this).registry(myGraphIoRegistry).create();
+}}
+----
+
+In this way, `Graph` implementations can pre-configure custom serializers for IO interactions and users will not need
+to know about those details. Following this pattern will ensure proper execution of the test suite as well as
+simplified usage for end-users.
+
+IMPORTANT: Proper implementation of IO is critical to successful `Graph` operations in Gremlin Server. The Test Suite
+does have "serialization" tests that provide some assurance that an implementation is working properly, but those
+tests cannot make assertions against any specifics of a custom serializer. It is the responsibility of the
+implementer to test the specifics of their custom serializers.
+
+TIP: Consider separating serializer code into its own module, if possible, so that clients that use the `Graph`
+implementation remotely don't need a full dependency on the entire `Graph` - just the IO components and related
+classes being serialized.
+
+[[validating-with-gremlin-test]]
+Validating with Gremlin-Test
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+image:gremlin-edumacated.png[width=225]
+
+[source,xml]
+<dependency>
+ <groupId>org.apache.tinkerpop</groupId>
+ <artifactId>gremlin-test</artifactId>
+ <version>x.y.z</version>
+</dependency>
+<dependency>
+ <groupId>org.apache.tinkerpop</groupId>
+ <artifactId>gremlin-groovy-test</artifactId>
+ <version>x.y.z</version>
+</dependency>
+
+The operational semantics of any OLTP or OLAP implementation are validated by `gremlin-test` and functional
+interoperability with the Groovy environment is ensured by `gremlin-groovy-test`. To implement these tests, provide
+test case implementations as shown below, where `XXX` below denotes the name of the graph implementation (e.g.
+TinkerGraph, Neo4jGraph, HadoopGraph, etc.).
+
+[source,java]
+----
+// Structure API tests
+@RunWith(StructureStandardSuite.class)
+@GraphProviderClass(provider = XXXGraphProvider.class, graph = XXXGraph.class)
+public class XXXStructureStandardTest {}
+
+// Process API tests
+@RunWith(ProcessComputerSuite.class)
+@GraphProviderClass(provider = XXXGraphProvider.class, graph = XXXGraph.class)
+public class XXXProcessComputerTest {}
+
+@RunWith(ProcessStandardSuite.class)
+@GraphProviderClass(provider = XXXGraphProvider.class, graph = XXXGraph.class)
+public class XXXProcessStandardTest {}
+
+@RunWith(GroovyEnvironmentSuite.class)
+@GraphProviderClass(provider = XXXProvider.class, graph = TinkerGraph.class)
+public class XXXGroovyEnvironmentTest {}
+
+@RunWith(GroovyProcessStandardSuite.class)
+@GraphProviderClass(provider = XXXGraphProvider.class, graph = TinkerGraph.class)
+public class XXXGroovyProcessStandardTest {}
+
+@RunWith(GroovyProcessComputerSuite.class)
+@GraphProviderClass(provider = XXXGraphComputerProvider.class, graph = TinkerGraph.class)
+public class XXXGroovyProcessComputerTest {}
+----
+
+The above set of tests represent the minimum test suite set to implement. There are other "integration" and
+"performance" tests that should be considered optional. Implementing those tests requires the same pattern as shown above.
+
+IMPORTANT: It is as important to look at "ignored" tests as it is to look at ones that fail. The `gremlin-test`
+suite utilizes the `Feature` implementation exposed by the `Graph` to determine which tests to execute. If a test
+utilizes features that are not supported by the graph, it will ignore them. While that may be fine, implementers
+should validate that the ignored tests are appropriately bypassed and that there are no mistakes in their feature
+definitions. Moreover, implementers should consider filling gaps in their own test suites, especially when
+IO-related tests are being ignored.
+
+The only test-class that requires any code investment is the `GraphProvider` implementation class. This class is a
+used by the test suite to construct `Graph` configurations and instances and provides information about the
+implementation itself. In most cases, it is best to simply extend `AbstractGraphProvider` as it provides many
+default implementations of the `GraphProvider` interface.
+
+Finally, specify the test suites that will be supported by the `Graph` implementation using the `@Graph.OptIn`
+annotation. See the `TinkerGraph` implementation below as an example:
+
+[source,java]
+----
+@Graph.OptIn(Graph.OptIn.SUITE_STRUCTURE_STANDARD)
+@Graph.OptIn(Graph.OptIn.SUITE_PROCESS_STANDARD)
+@Graph.OptIn(Graph.OptIn.SUITE_PROCESS_COMPUTER)
+@Graph.OptIn(Graph.OptIn.SUITE_GROOVY_PROCESS_STANDARD)
+@Graph.OptIn(Graph.OptIn.SUITE_GROOVY_PROCESS_COMPUTER)
+@Graph.OptIn(Graph.OptIn.SUITE_GROOVY_ENVIRONMENT)
+public class TinkerGraph implements Graph {
+----
+
+Only include annotations for the suites the implementation will support. Note that implementing the suite, but
+not specifying the appropriate annotation will prevent the suite from running (an obvious error message will appear
+in this case when running the mis-configured suite).
+
+There are times when there may be a specific test in the suite that the implementation cannot support (despite the
+features it implements) or should not otherwise be executed. It is possible for implementers to "opt-out" of a test
+by using the `@Graph.OptOut` annotation. The following is an example of this annotation usage as taken from
+`HadoopGraph`:
+
+[source, java]
+----
+@Graph.OptIn(Graph.OptIn.SUITE_PROCESS_STANDARD)
+@Graph.OptIn(Graph.OptIn.SUITE_PROCESS_COMPUTER)
+@Graph.OptOut(
+ test = "org.apache.tinkerpop.gremlin.process.graph.step.map.MatchTest$Traversals",
+ method = "g_V_matchXa_hasXname_GarciaX__a_inXwrittenByX_b__a_inXsungByX_bX",
+ reason = "Hadoop-Gremlin is OLAP-oriented and for OLTP operations, linear-scan joins are required. This particular tests takes many minutes to execute.")
+@Graph.OptOut(
+ test = "org.apache.tinkerpop.gremlin.process.graph.step.map.MatchTest$Traversals",
+ method = "g_V_matchXa_inXsungByX_b__a_inXsungByX_c__b_outXwrittenByX_d__c_outXwrittenByX_e__d_hasXname_George_HarisonX__e_hasXname_Bob_MarleyXX",
+ reason = "Hadoop-Gremlin is OLAP-oriented and for OLTP operations, linear-scan joins are required. This particular tests takes many minutes to execute.")
+@Graph.OptOut(
+ test = "org.apache.tinkerpop.gremlin.process.computer.GraphComputerTest",
+ method = "shouldNotAllowBadMemoryKeys",
+ reason = "Hadoop does a hard kill on failure and stops threads which stops test cases. Exception handling semantics are correct though.")
+@Graph.OptOut(
+ test = "org.apache.tinkerpop.gremlin.process.computer.GraphComputerTest",
+ method = "shouldRequireRegisteringMemoryKeys",
+ reason = "Hadoop does a hard kill on failure and stops threads which stops test cases. Exception handling semantics are correct though.")
+public class HadoopGraph implements Graph {
+----
+
+The above examples show how to ignore individual tests. It is also possible to:
+
+* Ignore an entire test case (i.e. all the methods within the test) by setting the `method` to "*".
+* Ignore a "base" test class such that test that extend from those classes will all be ignored. This style of
+ignoring is useful for Gremlin "process" tests that have bases classes that are extended by various Gremlin flavors (e.g. groovy).
+* Ignore a `GraphComputer` test based on the type of `GraphComputer` being used. Specify the "computer" attribute on
+the `OptOut` (which is an array specification) which should have a value of the `GraphComputer` implementation class
+that should ignore that test. This attribute should be left empty for "standard" execution and by default all
+`GraphComputer` implementations will be included in the `OptOut` so if there are multiple implementations, explicitly
+specify the ones that should be excluded.
+
+Also note that some of the tests in the Gremlin Test Suite are parameterized tests and require an additional level of
+specificity to be properly ignored. To ignore these types of tests, examine the name template of the parameterized
+tests. It is defined by a Java annotation that looks like this:
+
+[source, java]
+@Parameterized.Parameters(name = "expect({0})")
+
+The annotation above shows that the name of each parameterized test will be prefixed with "expect" and have
+parentheses wrapped around the first parameter (at index 0) value supplied to each test. This information can
+only be garnered by studying the test set up itself. Once the pattern is determined and the specific unique name of
+the parameterized test is identified, add it to the `specific` property on the `OptOut` annotation in addition to
+the other arguments.
+
+These annotations help provide users a level of transparency into test suite compliance (via the
+xref:describe-graph[describeGraph()] utility function). It also allows implementers to have a lot of flexibility in
+terms of how they wish to support TinkerPop. For example, maybe there is a single test case that prevents an
+implementer from claiming support of a `Feature`. The implementer could choose to either not support the `Feature`
+or to support it but "opt-out" of the test with a "reason" as to why so that users understand the limitation.
+
+IMPORTANT: Before using `OptOut` be sure that the reason for using it is sound and it is more of a last resort.
+It is possible that a test from the suite doesn't properly represent the expectations of a feature, is too broad or
+narrow for the semantics it is trying to enforce or simply contains a bug. Please consider raising issues in the
+developer mailing list with such concerns before assuming `OptOut` is the only answer.
+
+IMPORTANT: There are no tests that specifically validate complete compliance with Gremlin Server. Generally speaking,
+a `Graph` that passes the full Test Suite, should be compliant with Gremlin Server. The one area where problems can
+occur is in serialization. Always ensure that IO is properly implemented, that custom serializers are tested fully
+and ultimately integration test the `Graph` with an actual Gremlin Server instance.
+
+CAUTION: Configuring tests to run in parallel might result in errors that are difficult to debug as there is some
+shared state in test execution around graph configuration. It is therefore recommended that parallelism be turned
+off for the test suite (the Maven SureFire Plugin is configured this way by default). It may also be important to
+include this setting, `<reuseForks>false</reuseForks>`, in the SureFire configuration if tests are failing in an
+unexplainable way.
+
+Accessibility via GremlinPlugin
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+image:gremlin-plugin.png[width=100,float=left] The applications distributed with TinkerPop3 do not distribute with
+any graph system implementations besides TinkerGraph. If your implementation is stored in a Maven repository (e.g.
+Maven Central Repository), then it is best to provide a `GremlinPlugin` implementation so the respective jars can be
+downloaded according and when required by the user. Neo4j's GremlinPlugin is provided below for reference.
+
+[source,java]
+----
+public class Neo4jGremlinPlugin implements GremlinPlugin {
+
+ private static final String IMPORT = "import ";
+ private static final String DOT_STAR = ".*";
+
+ private static final Set<String> IMPORTS = new HashSet<String>() {{
+ add(IMPORT + Neo4jGraph.class.getPackage().getName() + DOT_STAR);
+ }};
+
+ @Override
+ public String getName() {
+ return "neo4j";
+ }
+
+ @Override
+ public void pluginTo(final PluginAcceptor pluginAcceptor) {
+ pluginAcceptor.addImports(IMPORTS);
+ }
+}
+----
+
+With the above plugin implementations, users can now download respective binaries for Gremlin Console, Gremlin Server, etc.
+
+[source,groovy]
+gremlin> g = Neo4jGraph.open('/tmp/neo4j')
+No such property: Neo4jGraph for class: groovysh_evaluate
+Display stack trace? [yN]
+gremlin> :install org.apache.tinkerpop neo4j-gremlin x.y.z
+==>loaded: [org.apache.tinkerpop, neo4j-gremlin, …]
+gremlin> :plugin use tinkerpop.neo4j
+==>tinkerpop.neo4j activated
+gremlin> g = Neo4jGraph.open('/tmp/neo4j')
+==>neo4jgraph[EmbeddedGraphDatabase [/tmp/neo4j]]
+
+In-Depth Implementations
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+image:gremlin-painting.png[width=200,float=right] The graph system implementation details presented thus far are
+minimum requirements necessary to yield a valid TinkerPop3 implementation. However, there are other areas that a
+graph system provider can tweak to provide an implementation more optimized for their underlying graph engine. Typical
+areas of focus include:
+
+* Traversal Strategies: A <<traversalstrategy,TraversalStrategy>> can be used to alter a traversal prior to its
+execution. A typical example is converting a pattern of `g.V().has('name','marko')` into a global index lookup for
+all vertices with name "marko". In this way, a `O(|V|)` lookup becomes an `O(log(|V|))`. Please review
+`TinkerGraphStepStrategy` for ideas.
+* Step Implementations: Every <<graph-traversal-steps,step>> is ultimately referenced by the `GraphTraversal`
+interface. It is possible to extend `GraphTraversal` to use a graph system specific step implementation.
+
+
+[[tinkergraph-gremlin]]
+TinkerGraph-Gremlin
+-------------------
+
+[source,xml]
+----
+<dependency>
+ <groupId>org.apache.tinkerpop</groupId>
+ <artifactId>tinkergraph-gremlin</artifactId>
+ <version>x.y.z</version>
+</dependency>
+----
+
+image:tinkerpop-character.png[width=100,float=left] TinkerGraph is a single machine, in-memory (with optional
+persistence), non-transactional graph engine that provides both OLTP and OLAP functionality. It is deployed with
+TinkerPop3 and serves as the reference implementation for other providers to study in order to understand the
+semantics of the various methods of the TinkerPop3 API. Constructing a simple graph in Java8 is presented below.
+
+[source,java]
+Graph g = TinkerGraph.open();
+Vertex marko = g.addVertex("name","marko","age",29);
+Vertex lop = g.addVertex("name","lop","lang","java");
+marko.addEdge("created",lop,"weight",0.6d);
+
+The above graph creates two vertices named "marko" and "lop" and connects them via a created-edge with a weight=0.6
+property. Next, the graph can be queried as such.
+
+[source,java]
+g.V().has("name","marko").out("created").values("name")
+
+The `g.V().has("name","marko")` part of the query can be executed in two ways.
+
+ * A linear scan of all vertices filtering out those vertices that don't have the name "marko"
+ * A `O(log(|V|))` index lookup for all vertices with the name "marko"
+
+Given the initial graph construction in the first code block, no index was defined and thus, a linear scan is executed.
+However, if the graph was constructed as such, then an index lookup would be used.
+
+[source,java]
+Graph g = TinkerGraph.open();
+g.createIndex("name",Vertex.class)
+
+The execution times for a vertex lookup by property is provided below for both no-index and indexed version of
+TinkerGraph over the Grateful Dead graph.
+
+[gremlin-groovy]
+----
+graph = TinkerGraph.open()
+g = graph.traversal()
+graph.io(graphml()).readGraph('data/grateful-dead.xml')
+clock(1000) {g.V().has('name','Garcia').iterate()} <1>
+graph = TinkerGraph.open()
+g = graph.traversal()
+graph.createIndex('name',Vertex.class)
+graph.io(graphml()).readGraph('data/grateful-dead.xml')
+clock(1000){g.V().has('name','Garcia').iterate()} <2>
+----
+
+<1> Determine the average runtime of 1000 vertex lookups when no `name`-index is defined.
+<2> Determine the average runtime of 1000 vertex lookups when a `name`-index is defined.
+
+IMPORTANT: Each graph system will have different mechanism by which indices and schemas are defined. TinkerPop3
+does not require any conformance in this area. In TinkerGraph, the only definitions are around indices. With other
+graph systems, property value types, indices, edge labels, etc. may be required to be defined _a priori_ to adding
+data to the graph.
+
+NOTE: TinkerGraph is distributed with Gremlin Server and is therefore automatically available to it for configuration.
+
+Configuration
+~~~~~~~~~~~~~
+
+TinkerGraph has several settings that can be provided on creation via `Configuration` object:
+
+[width="100%",cols="2,10",options="header"]
+|=========================================================
+|Property |Description
+|gremlin.graph |`org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph`
+|gremlin.tinkergraph.vertexIdManager |The `IdManager` implementation to use for vertices.
+|gremlin.tinkergraph.edgeIdManager |The `IdManager` implementation to use for edges.
+|gremlin.tinkergraph.vertexPropertyIdManager |The `IdManager` implementation to use for vertex properties.
+|gremlin.tinkergraph.defaultVertexPropertyCardinality |The default `VertexProperty.Cardinality` to use when `Vertex.property(k,v)` is called.
+|gremlin.tinkergraph.graphLocation |The path and file name for where TinkerGraph should persist the graph data. If a
+value is specified here, the the `gremlin.tinkergraph.graphFormat` should also be specified. If this value is not
+included (default), then the graph will stay in-memory and not be loaded/persisted to disk.
+|gremlin.tinkergraph.graphFormat |The format to use to serialize the graph which may be one of the following:
+`graphml`, `graphson`, or `gryo`. If a value is specified here, the the `gremlin.tinkergraph.graphLocation` should
+also be specified. If this value is not included (default), then the graph will stay in-memory and not be
+loaded/persisted to disk.
+|=========================================================
+
+The `IdManager` settings above refer to how TinkerGraph will control identifiers for vertices, edges and vertex
+properties. There are several options for each of these settings: `ANY`, `LONG`, `INTEGER`, `UUID`, or the fully
+qualified class name of an `IdManager` implementation on the classpath. When not specified, the default values
+for all settings is `ANY`, meaning that the graph will work with any object on the JVM as the identifier and will
+generate new identifiers from `Long` when the identifier is not user supplied. TinkerGraph will also expect the
+user to understand the types used for identifiers when querying, meaning that `g.V(1)` and `g.V(1L)` could return
+two different vertices. `LONG`, `INTEGER` and `UUID` settings will try to coerce identifier values to the expected
+type as well as generate new identifiers with that specified type.
+
+If the TinkerGraph is configured for persistence with `gremlin.tinkergraph.graphLocation` and
+`gremlin.tinkergraph.graphFormat`, then the graph will be written to the specified location with the specified
+format when `Graph.close()` is called. In addition, if these settings are present, TinkerGraph will attempt to
+load the graph from the specified location.
+
+It is important to consider the data being imported to TinkerGraph with respect to `defaultVertexPropertyCardinality`
+setting. For example, if a `.gryo` file is known to contain multi-property data, be sure to set the default
+cardinality to `list` or else the data will import as `single`. Consider the following:
+
+[gremlin-groovy]
+----
+graph = TinkerGraph.open()
+graph.io(gryo()).readGraph("data/tinkerpop-crew.kryo")
+g = graph.traversal()
+g.V().properties()
+conf = new BaseConfiguration()
+conf.setProperty("gremlin.tinkergraph.defaultVertexPropertyCardinality","list")
+graph = TinkerGraph.open(conf)
+graph.io(gryo()).readGraph("data/tinkerpop-crew.kryo")
+g = graph.traversal()
+g.V().properties()
+----
+
+[[neo4j-gremlin]]
+Neo4j-Gremlin
+-------------
+
+[source,xml]
+----
+<dependency>
+ <groupId>org.apache.tinkerpop</groupId>
+ <artifactId>neo4j-gremlin</artifactId>
+ <version>x.y.z</version>
+</dependency>
+<!-- neo4j-tinkerpop-api-impl is NOT Apache 2 licensed - more information below -->
+<dependency>
+ <groupId>org.neo4j</groupId>
+ <artifactId>neo4j-tinkerpop-api-impl</artifactId>
+ <version>0.1-2.2</version>
+</dependency>
+----
+
+link:http://neotechnology.com[Neo Technology] are the developers of the OLTP-based link:http://neo4j.org[Neo4j graph database].
+
+CAUTION: Unless under a commercial agreement with Neo Technology, Neo4j is licensed
+link:http://en.wikipedia.org/wiki/Affero_General_Public_License[AGPL]. The `neo4j-gremlin` module is licensed Apache2
+because it only references the Apache2-licensed Neo4j API (not its implementation). Note that neither the
+<<gremlin-console,Gremlin Console>> nor <<gremlin-server,Gremlin Server>> distribute with the Neo4j implementation
+binaries. To access the binaries, use the `:install` command to download binaries from
+link:http://search.maven.org/[Maven Central Repository].
+
+[source,groovy]
+----
+gremlin> :install org.apache.tinkerpop neo4j-gremlin x.y.z
+==>Loaded: [org.apache.tinkerpop, neo4j-gremlin, x.y.z] - restart the console to use [tinkerpop.neo4j]
+gremlin> :q
+...
+gremlin> :plugin use tinkerpop.neo4j
+==>tinkerpop.neo4j activated
+gremlin> graph = Neo4jGraph.open('/tmp/neo4j')
+==>neo4jgraph[EmbeddedGraphDatabase [/tmp/neo4j]]
+----
+
+NOTE: Neo4j link:http://docs.neo4j.org/chunked/stable/ha.html[High Availability] is currently not supported by
+Neo4j-Gremlin.
+
+TIP: To host Neo4j in <<gremlin-server,Gremlin Server>>, the dependencies must first be "installed" or otherwise
+copied to the Gremlin Server path. The automated method for doing this would be to execute
+`bin/gremlin-server.sh -i org.apache.tinkerpop neo4j-gremlin x.y.z`.
+
+Indices
+~~~~~~~
+
+Neo4j 2.x indices leverage vertex labels to partition the index space. TinkerPop3 does not provide method interfaces
+for defining schemas/indices for the underlying graph system. Thus, in order to create indices, it is important to
+call the Neo4j API directly.
+
+NOTE: `Neo4jGraphStep` will attempt to discern which indices to use when executing a traversal of the form `g.V().has()`.
+
+The Gremlin-Console session below demonstrates Neo4j indices. For more information, please refer to the Neo4j documentation:
+
+* Manipulating indices with link:http://docs.neo4j.org/chunked/stable/query-schema-index.html[Cypher].
+* Manipulating indices with the Neo4j link:http://docs.neo4j.org/chunked/stable/tutorials-java-embedded-new-index.html[Java API].
+
+[gremlin-groovy]
+----
+graph = Neo4jGraph.open('/tmp/neo4j')
+graph.cypher("CREATE INDEX ON :person(name)")
+graph.tx().commit() <1>
+graph.addVertex(label,'person','name','marko')
+graph.addVertex(label,'dog','name','puppy')
+g = graph.traversal()
+g.V().hasLabel('person').has('name','marko').values('name')
+graph.close()
+----
+
+<1> Schema mutations must happen in a different transaction than graph mutations
+
+Below demonstrates the runtime benefits of indices and demonstrates how if there is no defined index (only vertex
+labels), a linear scan of the vertex-label partition is still faster than a linear scan of all vertices.
+
+[gremlin-groovy]
+----
+graph = Neo4jGraph.open('/tmp/neo4j')
+graph.io(graphml()).readGraph('data/grateful-dead.xml')
+g = graph.traversal()
+g.tx().commit()
+clock(1000) {g.V().hasLabel('artist').has('name','Garcia').iterate()} <1>
+graph.cypher("CREATE INDEX ON :artist(name)") <2>
+g.tx().commit()
+Thread.sleep(5000) <3>
+clock(1000) {g.V().hasLabel('artist').has('name','Garcia').iterate()} <4>
+clock(1000) {g.V().has('name','Garcia').iterate()} <5>
+graph.cypher("DROP INDEX ON :artist(name)") <6>
+g.tx().commit()
+graph.close()
+----
+
+<1> Find all artists whose name is Garcia which does a linear scan of the artist vertex-label partition.
+<2> Create an index for all artist vertices on their name property.
+<3> Neo4j indices are eventually consistent so this stalls to give the index time to populate itself.
+<4> Find all artists whose name is Garcia which uses the pre-defined schema index.
+<5> Find all vertices whose name is Garcia which requires a linear scan of all the data in the graph.
+<6> Drop the created index.
+
+Multi/Meta-Properties
+~~~~~~~~~~~~~~~~~~~~~
+
+`Neo4jGraph` supports both multi- and meta-properties (see <<_vertex_properties,vertex properties>>). These features
+are not native to Neo4j and are implemented using "hidden" Neo4j nodes. For example, when a vertex has multiple
+"name" properties, each property is a new node (multi-properties) which can have properties attached to it
+(meta-properties). As such, the native, underlying representation may become difficult to query directly using
+another graph language such as <<_cypher,Cypher>>. The default setting is to disable multi- and meta-properties.
+However, if this feature is desired, then it can be activated via `gremlin.neo4j.metaProperties` and
+`gremlin.neo4j.multiProperties` configurations being set to `true`. Once the configuration is set, it can not be
+changed for the lifetime of the graph.
+
+[gremlin-groovy]
+----
+conf = new BaseConfiguration()
+conf.setProperty('gremlin.neo4j.directory','/tmp/neo4j')
+conf.setProperty('gremlin.neo4j.multiProperties',true)
+conf.setProperty('gremlin.neo4j.metaProperties',true)
+graph = Neo4jGraph.open(conf)
+g = graph.traversal()
+g.addV('name','michael','name','michael hunger','name','mhunger')
+g.V().properties('name').property('acl', 'public')
+g.V(0).valueMap()
+g.V(0).properties()
+g.V(0).properties().valueMap()
+graph.close()
+----
+
+WARNING: `Neo4jGraph` without multi- and meta-properties is in 1-to-1 correspondence with the native, underlying Neo4j
+representation. It is recommended that if the user does not require multi/meta-properties, then they should not
+enable them. Without multi- and meta-properties enabled, Neo4j can be interacted with with other tools and technologies
+that do not leverage TinkerPop.
+
+IMPORTANT: When using a multi-property enabled `Neo4jGraph`, vertices may represent their properties on "hidden
+nodes" adjacent to the vertex. If a vertex property key/value is required for indexing, then two indices are
+required -- e.g. `CREATE INDEX ON :person(name)` and `CREATE INDEX ON :vertexProperty(name)`
+(see <<_indices,Neo4j indices>>).
+
+Cypher
+~~~~~~
+
+image::gremlin-loves-cypher.png[width=400]
+
+NeoTechnology are the creators of the graph pattern-match query language link:http://www.neo4j.org/learn/cypher[Cypher].
+It is possible to leverage Cypher from within Gremlin by using the `Neo4jGraph.cypher()` graph traversal method.
+
+[gremlin-groovy]
+----
+graph = Neo4jGraph.open('/tmp/neo4j')
+graph.io(gryo()).readGraph('data/tinkerpop-modern.kryo')
+graph.cypher('MATCH (a {name:"marko"}) RETURN a')
+graph.cypher('MATCH (a {name:"marko"}) RETURN a').select('a').out('knows').values('name')
+graph.close()
+----
+
+Thus, like <<match-step,`match()`>>-step in Gremlin, it is possible to do a declarative pattern match and then move
+back into imperative Gremlin.
+
+TIP: For those developers using <<gremlin-server,Gremlin Server>> against Neo4j, it is possible to do Cypher queries
+by simply placing the Cypher string in `graph.cypher(...)` before submission to the server.
+
+Multi-Label
+~~~~~~~~~~~
+
+TinkerPop3 requires every `Element` to have a single, immutable string label (i.e. a `Vertex`, `Edge`, and
+`VertexProperty`). In Neo4j, a `Node` (vertex) can have an
+link:http://neo4j.com/docs/stable/graphdb-neo4j-labels.html[arbitrary number of labels] while a `Relationship`
+(edge) can have one and only one. Furthermore, in Neo4j, `Node` labels are mutable while `Relationship` labels are
+not. In order to handle this mismatch, three `Neo4jVertex` specific methods exist in Neo4j-Gremlin.
+
+[source,java]
+public Set<String> labels() // get all the labels of the vertex
+public void addLabel(String label) // add a label to the vertex
+public void removeLabel(String label) // remove a label from the vertex
+
+An example use case is presented below.
+
+[gremlin-groovy]
+----
+graph = Neo4jGraph.open('/tmp/neo4j')
+vertex = (Neo4jVertex) graph.addVertex('human::animal') <1>
+vertex.label() <2>
+vertex.labels() <3>
+vertex.addLabel('organism') <4>
+vertex.label()
+vertex.removeLabel('human') <5>
+vertex.labels()
+vertex.addLabel('organism') <6>
+vertex.labels()
+vertex.removeLabel('human') <7>
+vertex.label()
+g = graph.traversal()
+g.V().has(label,'organism') <8>
+g.V().has(label,of('organism')) <9>
+g.V().has(label,of('organism')).has(label,of('animal'))
+g.V().has(label,of('organism').and(of('animal')))
+graph.close()
+----
+
+<1> Typecasting to a `Neo4jVertex` is only required in Java.
+<2> The standard `Vertex.label()` method returns all the labels in alphabetical order concatenated using `::`.
+<3> `Neo4jVertex.labels()` method returns the individual labels as a set.
+<4> `Neo4jVertex.addLabel()` method adds a single label.
+<5> `Neo4jVertex.removeLabel()` method removes a single label.
+<6> Labels are unique and thus duplicate labels don't exist.
+<7> If a label that does not exist is removed, nothing happens.
+<8> `P.eq()` does a full string match and should only be used if multi-labels are not leveraged.
+<9> `LabelP.of()` is specific to `Neo4jGraph` and used for multi-label matching.
+
+IMPORTANT: `LabelP.of()` is only required if multi-labels are leveraged. `LabelP.of()` is used when
+filtering/looking-up vertices by their label(s) as the standard `P.eq()` does a direct match on the `::`-representation
+of `vertex.label()`
+
+Loading with BulkLoaderVertexProgram
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The <<bulkloadervertexprogram, BulkLoaderVertexProgram>> is a generalized bulk loader that can be used to load
+large amounts of data to and from Neo4j. The following code demonstrates how to load the modern graph from TinkerGraph
+into Neo4j:
+
+[gremlin-groovy]
+----
+wgConf = 'conf/neo4j-standalone.properties'
+modern = TinkerFactory.createModern()
+blvp = BulkLoaderVertexProgram.build().
+ keepOriginalIds(false).
+ writeGraph(wgConf).create(modern)
+modern.compute().workers(1).program(blvp).submit().get()
+graph = GraphFactory.open(wgConf)
+g = graph.traversal()
+g.V().valueMap()
+graph.close()
+----
+
+[source,properties]
+----
+# neo4j-standalone.properties
+
+gremlin.graph=org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jGraph
+gremlin.neo4j.directory=/tmp/neo4j
+gremlin.neo4j.conf.node_auto_indexing=true
+gremlin.neo4j.conf.relationship_auto_indexing=true
+----
+
+[[hadoop-gremlin]]
+Hadoop-Gremlin
+--------------
+
+[source,xml]
+----
+<dependency>
+ <groupId>org.apache.tinkerpop</groupId>
+ <artifactId>hadoop-gremlin</artifactId>
+ <version>x.y.z</version>
+</dependency>
+----
+
+image:hadoop-logo-notext.png[width=100,float=left] link:http://hadoop.apache.org/[Hadoop] is a distributed
+computing framework that is used to process data represented across a multi-machine compute cluster. When the
+data in the Hadoop cluster represents a TinkerPop3 graph, then Hadoop-Gremlin can be used to process the graph
+using both TinkerPop3's OLTP and OLAP graph computing models.
+
+IMPORTANT: This section assumes that the user has a Hadoop 2.x cluster functioning. For more information on getting
+started with Hadoop, please see the
+link:http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/SingleCluster.html[Single Node Setup]
+tutorial. Moreover, if using `GiraphGraphComputer` or `SparkGraphComputer` it is advisable that the reader also
+familiarize their self with Giraph (link:http://giraph.apache.org/quick_start.html[Getting Started]) and Spark
+(link:http://spark.apache.org/docs/latest/quick-start.html[Quick Start]).
+
+Installing Hadoop-Gremlin
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The `HADOOP_GREMLIN_LIBS` references locations that contains jars that should be uploaded to a respective
+distributed cache (link:http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html[YARN] or SparkServer).
+Note that the locations in `HADOOP_GREMLIN_LIBS` can be a colon-separated (`:`) and all jars from all locations will
+be loaded into the cluster. Typically, only the jars of the respective GraphComputer are required to be loaded (e.g.
+`GiraphGraphComputer` plugin lib directory).
+
+[source,shell]
+export HADOOP_GREMLIN_LIBS=/usr/local/gremlin-console/ext/giraph-gremlin/lib
+
+If using <<gremlin-console,Gremlin Console>>, it is important to install the Hadoop-Gremlin plugin. Note that
+Hadoop-Gremlin requires a Gremlin Console restart after installing.
+
+[source,text]
+----
+$ bin/gremlin.sh
+
+ \,,,/
+ (o o)
+-----oOOo-(3)-oOOo-----
+plugin activated: tinkerpop.server
+plugin activated: tinkerpop.utilities
+plugin activated: tinkerpop.tinkergraph
+gremlin> :install org.apache.tinkerpop hadoop-gremlin x.y.z
+==>loaded: [org.apache.tinkerpop, hadoop-gremlin, x.y.z] - restart the console to use [tinkerpop.hadoop]
+gremlin> :q
+$ bin/gremlin.sh
+
+ \,,,/
+ (o o)
+-----oOOo-(3)-oOOo-----
+plugin activated: tinkerpop.server
+plugin activated: tinkerpop.utilities
+plugin activated: tinkerpop.tinkergraph
+gremlin> :plugin use tinkerpop.hadoop
+==>tinkerpop.hadoop activated
+gremlin>
+----
+
+Properties Files
+~~~~~~~~~~~~~~~~
+
+`HadoopGraph` makes use of properties files which ultimately get turned into Apache configurations and/or
+Hadoop configurations. The example properties file presented below is located at `conf/hadoop/hadoop-gryo.properties`.
+
+[source,text]
+gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
+gremlin.hadoop.inputLocation=tinkerpop-modern.kryo
+gremlin.hadoop.graphInputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat
+gremlin.hadoop.outputLocation=output
+gremlin.hadoop.graphOutputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat
+gremlin.hadoop.jarsInDistributedCache=true
+####################################
+# Spark Configuration #
+####################################
+spark.master=local[4]
+spark.executor.memory=1g
+spark.serializer=org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer
+####################################
+# SparkGraphComputer Configuration #
+####################################
+gremlin.spark.graphInputRDD=org.apache.tinkerpop.gremlin.spark.structure.io.InputRDDFormat
+gremlin.spark.graphOutputRDD=org.apache.tinkerpop.gremlin.spark.structure.io.OutputRDDFormat
+gremlin.spark.persistContext=true
+#####################################
+# GiraphGraphComputer Configuration #
+#####################################
+giraph.minWorkers=2
+giraph.maxWorkers=2
+giraph.useOutOfCoreGraph=true
+giraph.useOutOfCoreMessages=true
+mapreduce.map.java.opts=-Xmx1024m
+mapreduce.reduce.java.opts=-Xmx1024m
+giraph.numInputThreads=2
+giraph.numComputeThreads=2
+
+A review of the Hadoop-Gremlin specific properties are provided in the table below. For the respective OLAP
+engines (<<sparkgraphcomputer,`SparkGraphComputer`>> or <<giraphgraphcomputer,`GiraphGraphComputer`>>) refer
+to their respective documentation for configuration options.
+
+[width="100%",cols="2,10",options="header"]
+|=========================================================
+|Property |Description
+|gremlin.graph |The class of the graph to construct using GraphFactory.
+|gremlin.hadoop.inputLocation |The location of the input file(s) for Hadoop-Gremlin to read the graph from.
+|gremlin.hadoop.graphInputFormat |The format that the graph input file(s) are represented in.
+|gremlin.hadoop.outputLocation |The location to write the computed HadoopGraph to.
+|gremlin.hadoop.graphOutputFormat |The format that the output file(s) should be represented in.
+|gremlin.hadoop.jarsInDistributedCache |Whether to upload the Hadoop-Gremlin jars to a distributed cache (necessary if jars are not on the machines' classpaths).
+|=========================================================
+
+
+
+Along with the properties above, the numerous link:http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/core-default.xml[Hadoop specific properties]
+can be added as needed to tune and parameterize the executed Hadoop-Gremlin job on the respective Hadoop cluster.
+
+IMPORTANT: As the size of the graphs being processed becomes large, it is important to fully understand how the
+underlying OLAP engine (e.g. Spark, Giraph, etc.) works and understand the numerous parameterizations offered by
+these systems. Such knowledge can help alleviate out of memory exceptions, slow load times, slow processing times,
+garbage collection issues, etc.
+
+OLTP Hadoop-Gremlin
+~~~~~~~~~~~~~~~~~~~
+
+image:hadoop-pipes.png[width=180,float=left] It is possible to execute OLTP operations over a `HadoopGraph`.
+However, realize that the underlying HDFS files are not random access and thus, to retrieve a vertex, a linear scan
+is required. OLTP operations are useful for peeking into the graph prior to executing a long running OLAP job -- e.g.
+`g.V().valueMap().limit(10)`.
+
+CAUTION: OLTP operations on `HadoopGraph` are not efficient. They require linear scans to execute and are unreasonable
+for large graphs. In such large graph situations, make use of <<traversalvertexprogram,TraversalVertexProgram>>
+which is the OLAP Gremlin machine.
+
+[gremlin-groovy]
+----
+hdfs.copyFromLocal('data/tinkerpop-modern.kryo', 'tinkerpop-modern.kryo')
+hdfs.ls()
+graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
+g = graph.traversal()
+g.V().count()
+g.V().out().out().values('name')
+g.V().group().by{it.value('name')[1]}.by('name').next()
+----
+
+OLAP Hadoop-Gremlin
+~~~~~~~~~~~~~~~~~~~
+
+image:hadoop-furnace.png[width=180,float=left] Hadoop-Gremlin was designed to execute OLAP operations via
+`GraphComputer`. The OLTP examples presented previously are reproduced below, but using `TraversalVertexProgram`
+for the execution of the Gremlin traversal.
+
+A `Graph` in TinkerPop3 can support any number of `GraphComputer` implementations. Out of the box, Hadoop-Gremlin
+supports the following three implementations.
+
+* <<mapreducegraphcomputer,`MapReduceGraphComputer`>>: Leverages Hadoop's MapReduce engine to execute TinkerPop3 OLAP
+computations. (*coming soon*)
+** The graph must fit within the total disk space of the Hadoop cluster (supports massive graphs). Message passing is
+coordinated via MapReduce jobs over the on-disk graph (slow traversals).
+* <<sparkgraphcomputer,`SparkGraphComputer`>>: Leverages Apache Spark to execute TinkerPop3 OLAP computations.
+** The graph may fit within the total RAM of the cluster (supports larger graphs). Message passing is coordinated via
+Spark map/reduce/join operations on in-memory and disk-cached data (average speed traversals).
+* <<giraphgraphcomputer,`GiraphGraphComputer`>>: Leverages Apache Giraph to execute TinkerPop3 OLAP computations.
+** The graph should fit within the total RAM of the Hadoop cluster (graph size restriction), though "out-of-core"
+processing is possible. Message passing is coordinated via ZooKeeper for the in-memory graph (speedy traversals).
+
+TIP: image:gremlin-sugar.png[width=50,float=left] For those wanting to use the <<sugar-plugin,SugarPlugin>> with
+their submitted traversal, do `:remote config useSugar true` as well as `:plugin use tinkerpop.sugar` at the start of
+the Gremlin Console session if it is not already activated.
+
+Note that `SparkGraphComputer` and `GiraphGraphComputer` are loaded via their respective plugins. Typically only
+one plugin or the other is loaded depending on the desired `GraphComputer` to use.
+
+[source,text]
+----
+$ bin/gremlin.sh
+
+ \,,,/
+ (o o)
+-----oOOo-(3)-oOOo-----
+plugin activated: tinkerpop.server
+plugin activated: tinkerpop.utilities
+plugin activated: tinkerpop.tinkergraph
+plugin activated: tinkerpop.hadoop
+gremlin> :install org.apache.tinkerpop giraph-gremlin x.y.z
+==>loaded: [org.apache.tinkerpop, giraph-gremlin, x.y.z] - restart the console to use [tinkerpop.giraph]
+gremlin> :install org.apache.tinkerpop spark-gremlin x.y.z
+==>loaded: [org.apache.tinkerpop, spark-gremlin, x.y.z] - restart the console to use [tinkerpop.spark]
+gremlin> :q
+$ bin/gremlin.sh
+
+ \,,,/
+ (o o)
+-----oOOo-(3)-oOOo-----
+plugin activated: tinkerpop.server
+plugin activated: tinkerpop.utilities
+plugin activated: tinkerpop.tinkergraph
+plugin activated: tinkerpop.hadoop
+gremlin> :plugin use tinkerpop.giraph
+==>tinkerpop.giraph activated
+gremlin> :plugin use tinkerpop.spark
+==>tinkerpop.spark activated
+----
+
+WARNING: Hadoop, Spark, and Giraph all depend on many of the same libraries (e.g. ZooKeeper, Snappy, Netty, Guava,
+etc.). Unfortunately, typically these dependencies are not to the same versions of the respective libraries. As such,
+it is best to *not* have both Spark and Giraph plugins loaded in the same console session nor in the same Java
+project (though intelligent `<exclusion>`-usage can help alleviate conflicts in a Java project).
+
+[[mapreducegraphcomputer]]
+MapReduceGraphComputer
+^^^^^^^^^^^^^^^^^^^^^^
+
+*COMING SOON*
+
+[[sparkgraphcomputer]]
+SparkGraphComputer
+^^^^^^^^^^^^^^^^^^
+
+[source,xml]
+----
+<dependency>
+ <groupId>org.apache.tinkerpop</groupId>
+ <artifactId>spark-gremlin</artifactId>
+ <version>x.y.z</version>
+</dependency>
+----
+
+image:spark-logo.png[width=175,float=left] link:http://spark.apache.org[Spark] is an Apache Software Foundation
+project focused on general-purpose OLAP data processing. Spark provides a hybrid in-memory/disk-based distributed
+computing model that is similar to Hadoop's MapReduce model. Spark maintains a fluent function chaining DSL that is
+arguably easier for developers to work with than native Hadoop MapReduce. Spark-Gremlin provides an implementation of
+the bulk-synchronous parallel, distributed message passing algorithm within Spark and thus, any `VertexProgram` can be
+executed over `SparkGraphComputer`.
+
+[gremlin-groovy]
+----
+graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
+g = graph.traversal(computer(SparkGraphComputer))
+g.V().count()
+g.V().out().out().values('name')
+----
+
+For using lambdas in Gremlin-Groovy, simply provide `:remote connect` a `TraversalSource` which leverages SparkGraphComputer.
+
+[gremlin-groovy]
+----
+graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
+g = graph.traversal(computer(SparkGraphComputer))
+:remote connect tinkerpop.hadoop graph g
+:> g.V().group().by{it.value('name')[1]}.by('name')
+----
+
+The `SparkGraphComputer` algorithm leverages Spark's caching abilities to reduce the amount of data shuffled across
+the wire on each iteration of the <<vertexprogram,`VertexProgram`>>. When the graph is loaded as a Spark RDD
+(Resilient Distributed Dataset) it is immediately cached as `graphRDD`. The `graphRDD` is a distributed adjacency
+list which encodes the vertex, its properties, and all its incident edges. On the first iteration, each vertex
+(in parallel) is passed through `VertexProgram.execute()`. This yields an output of the vertex's mutated state
+(i.e. updated compute keys -- `propertyX`) and its outgoing messages. This `viewOutgoingRDD` is then reduced to
+`viewIncomingRDD` where the outgoing messages are sent to their respective vertices. If a `MessageCombiner` exists
+for the vertex program, then messages are aggregated locally and globally to ultimately yield one incoming message
+for the vertex. This reduce sequence is the "message pass." If the vertex program does not terminate on this
+iteration, then the `viewIncomingRDD` is joined with the cached `graphRDD` and the process continues. When there
+are no more iterations, there is a final join and the resultant RDD is stripped of its edges and messages. This
+`mapReduceRDD` is cached and is processed by each <<mapreduce,`MapReduce`>> job in the
+<<graphcomputer,`GraphComputer`>> computation.
+
+image::spark-algorithm.png[width=775]
+
+[width="100%",cols="2,10",options="header"]
+|========================================================
+|Property |Description
+|gremlin.spark.graphInputRDD |A class for creating RDD's from underlying graph data, defaults to Hadoop `InputFormat`.
+|gremlin.spark.graphOutputRDD |A class for output RDD's, defaults to Hadoop `OutputFormat`.
+|gremlin.spark.persistContext |Whether to create a new `SparkContext` for every `SparkGraphComputer` or to reuse an existing one.
+|========================================================
+
+If the provider/user wishes to not use Hadoop `InputFormats`, it is possible to leverage Spark's RDD
+constructs directly. There is a `gremlin.spark.graphInputRDD` configuration that references a `Class<? extends
+InputRDD>`. An `InputRDD` provides a read method that takes a `SparkContext` and returns a graphRDD. Likewise, use
+`gremlin.spark.graphOutputRDD` and the respective `OutputRDD`.
+
+It is possible to persist the graph RDD between jobs within the `SparkContext` (e.g. SparkServer) by leveraging `PersistedOutputRDD`.
+Note that `gremlin.spark.persistContext` should be set to `true` or else the persisted RDD will be destroyed when the `SparkContext` closes.
+The persisted RDD is named by the `gremlin.hadoop.outputLocation` configuration (i.e. named in `SparkContext.getPersistedRDDs()`).
+Finally, `PersistedInputRDD` is used with respective `gremlin.hadoop.inputLocation` to retrieve the persisted RDD from the `SparkContext`.
+
+When using a persistent `Spark Context` the configuration used by the original Spark Configuration will be inherited by all threaded
+references to that Spark Context. The exception to this rule are those properties which have a specific thread local effect.
+
+.Thread Local Properties
+. spark.jobGroup.id
+. spark.job.description
+. spark.job.interruptOnCancel
+. spark.scheduler.pool
+
+Loading with BulkLoaderVertexProgram
+++++++++++++++++++++++++++++++++++++
+
+The <<bulkloadervertexprogram, BulkLoaderVertexProgram>> is a generalized bulk loader that can be used to load large
+amounts of data to and from different `Graph` implementations. The following code demonstrates how to load the
+Grateful Dead graph from HadoopGraph into TinkerGraph over Spark:
+
+[gremlin-groovy]
+----
+hdfs.copyFromLocal('data/grateful-dead.kryo', 'data/grateful-dead.kryo')
+readGraph = GraphFactory.open('conf/hadoop/hadoop-grateful-gryo.properties')
+writeGraph = 'conf/tinkergraph-gryo.properties'
+blvp = BulkLoaderVertexProgram.build().
+ keepOriginalIds(false).
+ writeGraph(writeGraph).create(readGraph)
+readGraph.compute(SparkGraphComputer).workers(1).program(blvp).submit().get()
+:set max-iteration 10
+graph = GraphFactory.open(writeGraph)
+g = graph.traversal()
+g.V().valueMap()
+graph.close()
+----
+
+[source,properties]
+----
+# hadoop-grateful-gryo.properties
+
+#
+# Hadoop Graph Configuration
+#
+gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
+gremlin.hadoop.graphInputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat
+gremlin.hadoop.memoryOutputFormat=org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat
+gremlin.hadoop.inputLocation=data/grateful-dead.kryo
+gremlin.hadoop.outputLocation=output
+gremlin.hadoop.deriveMemory=false
+gremlin.hadoop.jarsInDistributedCache=true
+
+#
+# SparkGraphComputer Configuration
+#
+spark.master=local[1]
+spark.executor.memory=1g
+spark.serializer=org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer
+----
+
+[source,properties]
+----
+# tinkergraph-gryo.properties
+
+gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph
+gremlin.tinkergraph.graphFormat=gryo
+gremlin.tinkergraph.graphLocation=/tmp/tinkergraph.kryo
+----
+
+IMPORTANT: The path to TinkerGraph jars needs to be included in the `HADOOP_GREMLIN_LIBS` for the above example to work.
+
+[[giraphgraphcomputer]]
+GiraphGraphComputer
+^^^^^^^^^^^^^^^^^^^
+
+[source,xml]
+----
+<dependency>
+ <groupId>org.apache.tinkerpop</groupId>
+ <artifactId>giraph-gremlin</artifactId>
+ <version>x.y.z</version>
+</dependency>
+----
+
+image:giraph-logo.png[width=100,float=left] link:http://giraph.apache.org[Giraph] is an Apache Software Foundation
+project focused on OLAP-based graph processing. Giraph makes use of the distributed graph computing paradigm made
+popular by Google's Pregel. In Giraph, developers write "vertex programs" that get executed at each vertex in
+parallel. These programs communicate with one another in a bulk synchronous parallel (BSP) manner. This model aligns
+with TinkerPop3's `GraphComputer` API. TinkerPop3 provides an implementation of `GraphComputer` that works for Giraph
+called `GiraphGraphComputer`. Moreover, with TinkerPop3's <<mapreduce,MapReduce>>-framework, the standard
+Giraph/Pregel model is extended to support an arbitrary number of MapReduce phases to aggregate and yield results
+from the graph. Below are examples using `GiraphGraphComputer` from the <<gremlin-console,Gremlin-Console>>.
+
+WARNING: Giraph uses a large number of Hadoop counters. The default for Hadoop is 120. In `mapred-site.xml` it is
+possible to increase the limit it via the `mapreduce.job.counters.max` property. A good value to use is 1000. This
+is a cluster-wide property so be sure to restart the cluster after updating.
+
+WARNING: The maximum number of workers can be no larger than the number of map-slots in the Hadoop cluster minus 1.
+For example, if the Hadoop cluster has 4 map slots, then `giraph.maxWorkers` can not be larger than 3. One map-slot
+is reserved for the master compute node and all other slots can be allocated as workers to execute the VertexPrograms
+on the vertices of the graph.
+
+If `GiraphGraphComputer` will be used as the `GraphComputer` for `HadoopGraph` then its `lib` directory should be
+specified in `HADOOP_GREMLIN_LIBS`.
+
+[source,shell]
+export HADOOP_GREMLIN_LIBS=$HADOOP_GREMLIN_LIBS:/usr/local/gremlin-console/ext/giraph-gremlin/lib
+
+Or, the user can specify the directory in the Gremlin Console.
+
+[source,groovy]
+System.setProperty('HADOOP_GREMLIN_LIBS',System.getProperty('HADOOP_GREMLIN_LIBS') + ':' + '/usr/local/gremlin-console/ext/giraph-gremlin/lib')
+
+[gremlin-groovy]
+----
+graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
+g = graph.traversal(computer(GiraphGraphComputer))
+g.V().count()
+g.V().out().out().values('name')
+----
+
+IMPORTANT: The examples above do not use lambdas (i.e. closures in Gremlin-Groovy). This makes the traversal
+serializable and thus, able to be distributed to all machines in the Hadoop cluster. If a lambda is required in a
+traversal, then the traversal must be sent as a `String` and compiled locally at each machine in the cluster. The
+following example demonstrates the `:remote` command which allows for submitting Gremlin traversals as a `String`.
+
+[gremlin-groovy]
+----
+graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
+g = graph.traversal(computer(GiraphGraphComputer))
+:remote connect tinkerpop.hadoop graph g
+:> g.V().group().by{it.value('name')[1]}.by('name')
+result
+result.memory.runtime
+result.memory.keys()
+result.memory.get('~reducing')
+----
+
+NOTE: If the user explicitly specifies `giraph.maxWorkers` and/or `giraph.numComputeThreads` in the configuration,
+then these values will be used by Giraph. However, if these are not specified and the user never calls
+`GraphComputer.workers()` then `GiraphGraphComputer` will try to compute the number of workers/threads to use based
+on the cluster's profile.
+
+Loading with BulkLoaderVertexProgram
+++++++++++++++++++++++++++++++++++++
+
+The <<bulkloadervertexprogram, BulkLoaderVertexProgram>> is a generalized bulk loader that can be used to load
+large amounts of data to and from different `Graph` implementations. The following code demonstrates how to load
+the Grateful Dead graph from HadoopGraph into TinkerGraph over Giraph:
+
+[gremlin-groovy]
+----
+hdfs.copyFromLocal('data/grateful-dead.kryo', 'data/grateful-dead.kryo')
+readGraph = GraphFactory.open('conf/hadoop/hadoop-grateful-gryo.properties')
+writeGraph = 'conf/tinkergraph-gryo.properties'
+blvp = BulkLoaderVertexProgram.build().
+ keepOriginalIds(false).
+ writeGraph(writeGraph).create(readGraph)
+readGraph.compute(GiraphGraphComputer).workers(1).program(blvp).submit().get()
+:set max-iteration 10
+graph = GraphFactory.open(writeGraph)
+g = graph.traversal()
+g.V().valueMap()
+graph.close()
+----
+
+[source,properties]
+----
+# hadoop-grateful-gryo.properties
+
+#
+# Hadoop Graph Configuration
+#
+gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
+gremlin.hadoop.graphInputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat
+gremlin.hadoop.graphOutputFormat=org.apache.hadoop.mapreduce.lib.output.NullOutputFormat
+gremlin.hadoop.memoryOutputFormat=org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat
+gremlin.hadoop.inputLocation=data/grateful-dead.kryo
+gremlin.hadoop.outputLocation=output
+gremlin.hadoop.deriveMemory=false
+gremlin.hadoop.jarsInDistributedCache=true
+
+#
+# GiraphGraphComputer Configuration
+#
+giraph.minWorkers=1
+giraph.maxWorkers=1
+giraph.useOutOfCoreGraph=true
+giraph.useOutOfCoreMessages=true
+mapred.map.child.java.opts=-Xmx1024m
+mapred.reduce.child.java.opts=-Xmx1024m
+giraph.numInputThreads=4
+giraph.numComputeThreads=4
+giraph.maxMessagesInMemory=100000
+----
+
+[source,properties]
+----
+# tinkergraph-gryo.properties
+
+gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph
+gremlin.tinkergraph.graphFormat=gryo
+gremlin.tinkergraph.graphLocation=/tmp/tinkergraph.kryo
+----
+
+NOTE: The path to TinkerGraph needs to be included in the `HADOOP_GREMLIN_LIBS` for the above example to work.
+
+Input/Output Formats
+~~~~~~~~~~~~~~~~~~~~
+
+image:adjacency-list.png[width=300,float=right] Hadoop-Gremlin provides various I/O formats -- i.e. Hadoop
+`InputFormat` and `OutputFormat`. All of the formats make use of an link:http://en.wikipedia.org/wiki/Adjacency_list[adjacency list]
+representation of the graph where each "row" represents a single vertex, its properties, and its incoming and
+outgoing edges.
+
+{empty} +
+
+[[gryo-io-format]]
+Gryo I/O Format
+^^^^^^^^^^^^^^^
+
+* **InputFormat**: `org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat`
+* **OutputFormat**: `org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat`
+
+<<gryo-reader-writer,Gryo>> is a binary graph format that leverages link:https://github.com/EsotericSoftware/kryo[Kryo]
+to make a compact, binary representation of a vertex. It is recommended that users leverage Gryo given its space/time
+savings over text-based representations.
+
+NOTE: The `GryoInputFormat` is splittable.
+
+[[graphson-io-format]]
+GraphSON I/O Format
+^^^^^^^^^^^^^^^^^^^
+
+* **InputFormat**: `org.apache.tinkerpop.gremlin.hadoop.structure.io.graphson.GraphSONInputFormat`
+* **OutputFormat**: `org.apache.tinkerpop.gremlin.hadoop.structure.io.graphson.GraphSONOutputFormat`
+
+<<graphson-reader-writer,GraphSON>> is a JSON based graph format. GraphSON is a space-expensive graph format in that
+it is a text-based markup language. However, it is convenient for many developers to work with as its structure is
+simple (easy to create and parse).
+
+The data below represents an adjacency list representation of the classic TinkerGraph toy graph in GraphSON format.
+
+[source,json]
+{"id":1,"label":"person","outE":{"created":[{"id":9,"inV":3,"properties":{"weight":0.4}}],"knows":[{"id":7,"inV":2,"properties":{"weight":0.5}},{"id":8,"inV":4,"properties":{"weight":1.0}}]},"properties":{"name":[{"id":0,"value":"marko"}],"age":[{"id":1,"value":29}]}}
+{"id":2,"label":"person","inE":{"knows":[{"id":7,"outV":1,"properties":{"weight":0.5}}]},"properties":{"name":[{"id":2,"value":"vadas"}],"age":[{"id":3,"value":27}]}}
+{"id":3,"label":"software","inE":{"created":[{"id":9,"outV":1,"properties":{"weight":0.4}},{"id":11,"outV":4,"properties":{"weight":0.4}},{"id":12,"outV":6,"properties":{"weight":0.2}}]},"properties":{"name":[{"id":4,"value":"lop"}],"lang":[{"id":5,"value":"java"}]}}
+{"id":4,"label":"person","inE":{"knows":[{"id":8,"outV":1,"properties":{"weight":1.0}}]},"outE":{"created":[{"id":10,"inV":5,"properties":{"weight":1.0}},{"id":11,"inV":3,"properties":{"weight":0.4}}]},"properties":{"name":[{"id":6,"value":"josh"}],"age":[{"id":7,"value":32}]}}
+{"id":5,"label":"software","inE":{"created":[{"id":10,"outV":4,"properties":{"weight":1.0}}]},"properties":{"name":[{"id":8,"value":"ripple"}],"lang":[{"id":9,"value":"java"}]}}
+{"id":6,"label":"person","outE":{"created":[{"id":12,"inV":3,"properties":{"weight":0.2}}]},"properties":{"name":[{"id":10,"value":"peter"}],"age":[{"id":11,"value":35}]}}
+
+[[script-io-format]]
+Script I/O Format
+^^^^^^^^^^^^^^^^^
+
+* **InputFormat**: `org.apache.tinkerpop.gremlin.hadoop.structure.io.script.ScriptInputFormat`
+* **OutputFormat**: `org.apache.tinkerpop.gremlin.hadoop.structure.io.script.ScriptOutputFormat`
+
+`ScriptInputFormat` and `ScriptOutputFormat` take an arbitrary script and use that script to either read or write
+`Vertex` objects, respectively. This can be considered the most general `InputFormat`/`OutputFormat` possible in that
+Hadoop-Gremlin uses the user provided script for all reading/writing.
+
+ScriptInputFormat
++++++++++++++++++
+
+The data below represents an adjacency list representation of the classic TinkerGraph toy graph. First line reads,
+"vertex `1`, labeled `person` having 2 property values (`marko` and `29`) has 3 outgoing edges; the first edge is
+labeled `knows`, connects the current vertex `1` with vertex `2` and has a property value `0.4`, and so on."
+
+[source]
+1:person:marko:29 knows:2:0.5,knows:4:1.0,created:3:0.4
+2:person:vadas:27
+3:project:lop:java
+4:person:josh:32 created:3:0.4,created:5:1.0
+5:project:ripple:java
+6:person:peter:35 created:3:0.2
+
+There is no corresponding `InputFormat` that can parse this particular file (or some adjacency list variant of it).
+As such, `ScriptInputFormat` can be used. With `ScriptInputFormat` a script is stored in HDFS and leveraged by each
+mapper in the Hadoop job. The script must have the following method defined:
+
+[source,groovy]
+def parse(String line, ScriptElementFactory factory) { ... }
+
+`ScriptElementFactory` provides the following 4 methods:
+
+[source,java]
+Vertex vertex(Object id); // get or create the vertex with the given id
+Vertex vertex(Object id, String label); // get or create the vertex with the given id and label
+Edge edge(Vertex out, Vertex in); // create an edge between the two given vertices
+Edge edge(Vertex out, Vertex in, String label); // create an edge between the two given vertices using the given label
+
+An appropriate `parse()` for the above adjacency list file is:
+
+[source,groovy]
+def parse(line, factory) {
+ def parts = line.split(/ /)
+ def (id, label, name, x) = parts[0].split(/:/).toList()
+ def v1 = factory.vertex(id, label)
+ if (name != null) v1.property('name', name) // first value is always the name
+ if (x != null) {
+ // second value depends on the vertex label; it's either
+ // the age of a person or the language of a project
+ if (label.equals('project')) v1.property('lang', x)
+ else v1.property('age', Integer.valueOf(x))
+ }
+ if (parts.length == 2) {
+ parts[1].split(/,/).grep { !it.isEmpty() }.each {
+ def (eLabel, refId, weight) = it.split(/:/).toList()
+ def v2 = factory.vertex(refId)
+ def edge = factory.edge(v1, v2, eLabel)
+ edge.property('weight', Double.valueOf(weight))
+ }
+ }
+ return v1
+}
+
+The resultant `Vertex` denotes whether the line parsed yielded a valid Vertex. As such, if the line is not valid
+(e.g. a comment line, a skip line, etc.), then simply return `null`.
+
+ScriptOutputFormat Support
+++++++++++++++++++++++++++
+
+The principle above can also be used to convert a vertex to an arbitrary `String` representation that is ultimately
+streamed back to a file in HDFS. This is the role of `ScriptOutputFormat`. `ScriptOutputFormat` requires that the
+provided script maintains a method with the following signature:
+
+[source,groovy]
+def stringify(Vertex vertex) { ... }
+
+An appropriate `stringify()` to produce output in the same format that was shown in the `ScriptInputFormat` sample is:
+
+[source,groovy]
+def stringify(vertex) {
+ def v = vertex.values('name', 'age', 'lang').inject(vertex.id(), vertex.label()).join(':')
+ def outE = vertex.outE().map {
+ def e = it.get()
+ e.values('weight').inject(e.label(), e.inV().next().id()).join(':')
+ }.join(',')
+ return [v, outE].join('\t')
+}
+
+Interacting with HDFS
+~~~~~~~~~~~~~~~~~~~~~
+
+The distributed file system of Hadoop is called link:http://en.wikipedia.org/wiki/Apache_Hadoop#Hadoop_distributed_file_system[HDFS].
+The results of any OLAP operation are stored in HDFS accessible via `hdfs`.
+
+[gremlin-groovy]
+----
+graph = GraphFactory.open('conf/hadoop/hadoop-gryo.properties')
+g = graph.traversal(computer(SparkGraphComputer))
+:remote connect tinkerpop.hadoop graph g
+:> g.V().group().by{it.value('name')[1]}.by('name')
+hdfs.ls()
+hdfs.ls('output')
+hdfs.ls('output/~reducing')
+hdfs.head('output/~reducing', ObjectWritable)
+----
+
+A list of the HDFS methods available are itemized below. Note that these methods are also available for the 'local' variable:
+
+[width="100%",cols="13,10",options="header"]
+|=========================================================
+| Method| Description
+|hdfs.ls(String path)| List the contents of the supplied directory.
+|hdfs.cp(String from, String to)| Copy the specified path to the specified path.
+|hdfs.exists(String path)| Whether the specified path exists.
+|hdfs.rm(String path)| Remove the specified path.
+|hdfs.rmr(String path)| Remove the specified path and its contents recurssively.
+|hdfs.copyToLocal(String from, String to)| Copy the specified HDFS path to the specified local path.
+|hdfs.copyFromLocal(String from, String to)| Copy the specified local path to the specified HDFS path.
+|hdfs.mergeToLocal(String from, String to)| Merge the files in path to the specified local path.
+|hdfs.head(String path)| Display the data in the path as text.
+|hdfs.head(String path, int lineCount)| Text display only the first `lineCount`-number of lines in the path.
+|hdfs.head(String path, int totalKeyValues, Class<Writable> writableClass)| Display the path interpreting the key values as respective writable.
+|=========================================================
+
+A Command Line Example
+~~~~~~~~~~~~~~~~~~~~~~
+
+image::pagerank-logo.png[width=300]
+
+The classic link:http://en.wikipedia.org/wiki/PageRank[PageRank] centrality algorithm can be executed over the
+TinkerPop graph from the command line using `GiraphGraphComputer`.
+
+WARNING: Be sure that the `HADOOP_GREMLIN_LIBS` references the location `lib` directory of the respective
+`GraphComputer` engine being used or else the requisite dependencies will not be uploaded to the Hadoop cluster.
+
+[source,text]
+----
+$ hdfs dfs -copyFromLocal data/tinkerpop-modern.json tinkerpop-modern.json
+$ hdfs dfs -ls
+Found 2 items
+-rw-r--r-- 1 marko supergroup 2356 2014-07-28 13:00 /user/marko/tinkerpop-modern.json
+$ hadoop jar target/giraph-gremlin-x.y.z-job.jar org.apache.tinkerpop.gremlin.giraph.process.computer.GiraphGraphComputer ../hadoop-gremlin/conf/hadoop-graphson.properties
+15/09/11 08:02:08 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
+15/09/11 08:02:11 INFO computer.GiraphGraphComputer: HadoopGremlin(Giraph): PageRankVertexProgram[alpha=0.85,iterations=30]
+15/09/11 08:02:12 INFO mapreduce.JobSubmitter: number of splits:3
+15/09/11 08:02:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1441915907347_0028
+15/09/11 08:02:12 INFO impl.YarnClientImpl: Submitted application application_1441915907347_0028
+15/09/11 08:02:12 INFO job.GiraphJob: Tracking URL: http://markos-macbook:8088/proxy/application_1441915907347_0028/
+15/09/11 08:02:12 INFO job.GiraphJob: Waiting for resources... Job will start only when it gets all 3 mappers
+15/09/11 08:03:54 INFO mapreduce.Job: Running job: job_1441915907347_0028
+15/09/11 08:03:55 INFO mapreduce.Job: Job job_1441915907347_0028 running in uber mode : false
+15/09/11 08:03:55 INFO mapreduce.Job: map 33% reduce 0%
+15/09/11 08:03:57 INFO mapreduce.Job: map 67% reduce 0%
+15/09/11 08:04:01 INFO mapreduce.Job: map 100% reduce 0%
+15/09/11 08:06:17 INFO mapreduce.Job: Job job_1441915907347_0028 completed successfully
+15/09/11 08:06:17 INFO mapreduce.Job: Counters: 80
+ File System Counters
+ FILE: Number of bytes read=0
+ FILE: Number of bytes written=483918
+ FILE: Number of read operations=0
+ FILE: Number of large read operations=0
+ FILE: Number of write operations=0
+ HDFS: Number of bytes read=1465
+ HDFS: Number of bytes written=1760
+ HDFS: Number of read operations=39
+ HDFS: Number of large read operations=0
+ HDFS: Number of write operations=20
+ Job Counters
+ Launched map tasks=3
+ Other local map tasks=3
+ Total time spent by all maps in occupied slots (ms)=458105
+ Total time spent by all reduces in occupied slots (ms)=0
+ Total time spent by all map tasks (ms)=458105
+ Total vcore-seconds taken by all map tasks=458105
+ Total megabyte-seconds taken by all map tasks=469099520
+ Map-Reduce Framework
+ Map input records=3
+ Map output records=0
+ Input split bytes=132
+ Spilled Records=0
+ Failed Shuffles=0
+ Merged Map outputs=0
+ GC time elapsed (ms)=1594
+ CPU time spent (ms)=0
+ Physical memory (bytes) snapshot=0
+ Virtual memory (bytes) snapshot=0
+ Total committed heap usage (bytes)=527958016
+ Giraph Stats
+ Aggregate edges=0
+ Aggregate finished vertices=0
+ Aggregate sent message message bytes=13535
+ Aggregate sent messages=186
+ Aggregate vertices=6
+ Current master task partition=0
+ Current workers=2
+ Last checkpointed superstep=0
+ Sent message bytes=438
+ Sent messages=6
+ Superstep=31
+ Giraph Timers
+ Initialize (ms)=2996
+ Input superstep (ms)=5209
+ Setup (ms)=59
+ Shutdown (ms)=9324
+ Superstep 0 GiraphComputation (ms)=3861
+ Superstep 1 GiraphComputation (ms)=4027
+ Superstep 10 GiraphComputation (ms)=4000
+ Superstep 11 GiraphComputation (ms)=4004
+ Superstep 12 GiraphComputation (ms)=3999
+ Superstep 13 GiraphComputation (ms)=4000
+ Superstep 14 GiraphComputation (ms)=4005
+ Superstep 15 GiraphComputation (ms)=4003
+ Superstep 16 GiraphComputation (ms)=4001
+ Superstep 17 GiraphComputation (ms)=4007
+ Superstep 18 GiraphComputation (ms)=3998
+ Superstep 19 GiraphComputation (ms)=4006
+ Superstep 2 GiraphComputation (ms)=4007
+ Superstep 20 GiraphComputation (ms)=3996
+ Superstep 21 GiraphComputation (ms)=4006
+ Superstep 22 GiraphComputation (ms)=4002
+ Superstep 23 GiraphComputation (ms)=3998
+ Superstep 24 GiraphComputation (ms)=4003
+ Superstep 25 GiraphComputation (ms)=4001
+ Superstep 26 GiraphComputation (ms)=4003
+ Superstep 27 GiraphComputation (ms)=4005
+ Superstep 28 GiraphComputation (ms)=4002
+ Superstep 29 GiraphComputation (ms)=4001
+ Superstep 3 GiraphComputation (ms)=3988
+ Superstep 30 GiraphComputation (ms)=4248
+ Superstep 4 GiraphComputation (ms)=4010
+ Superstep 5 GiraphComputation (ms)=3998
+ Superstep 6 GiraphComputation (ms)=3996
+ Superstep 7 GiraphComputation (ms)=4005
+ Superstep 8 GiraphComputation (ms)=4009
+ Superstep 9 GiraphComputation (ms)=3994
+ Total (ms)=138788
+ File Input Format Counters
+ Bytes Read=0
+ File Output Format Counters
+ Bytes Written=0
+$ hdfs dfs -cat output/~g/*
+{"id":1,"label":"person","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":39,"value":0.15000000000000002}],"name":[{"id":0,"value":"marko"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":10,"value":3.0}],"age":[{"id":1,"value":29}]}}
+{"id":5,"label":"software","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":35,"value":0.23181250000000003}],"name":[{"id":8,"value":"ripple"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":6,"value":0.0}],"lang":[{"id":9,"value":"java"}]}}
+{"id":3,"label":"software","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":39,"value":0.4018125}],"name":[{"id":4,"value":"lop"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":10,"value":0.0}],"lang":[{"id":5,"value":"java"}]}}
+{"id":4,"label":"person","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":39,"value":0.19250000000000003}],"name":[{"id":6,"value":"josh"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":10,"value":2.0}],"age":[{"id":7,"value":32}]}}
+{"id":2,"label":"person","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":35,"value":0.19250000000000003}],"name":[{"id":2,"value":"vadas"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":6,"value":0.0}],"age":[{"id":3,"value":27}]}}
+{"id":6,"label":"person","properties":{"gremlin.pageRankVertexProgram.pageRank":[{"id":35,"value":0.15000000000000002}],"name":[{"id":10,"value":"peter"}],"gremlin.pageRankVertexProgram.edgeCount":[{"id":6,"value":1.0}],"age":[{"id":11,"value":35}]}}
+----
+
+Vertex 4 ("josh") is isolated below:
+
+[source,js]
+----
+{
+ "id":4,
+ "label":"person",
+ "properties": {
+ "gremlin.pageRankVertexProgram.pageRank":[{"id":39,"value":0.19250000000000003}],
+ "name":[{"id":6,"value":"josh"}],
+ "gremlin.pageRankVertexProgram.edgeCount":[{"id":10,"value":2.0}],
+ "age":[{"id":7,"value":32}]}
+ }
+}
+----
+
+Hadoop-Gremlin for Graph System Providers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Hadoop-Gremlin is centered around `InputFormats` and `OutputFormats`. If a 3rd-party graph system provider wishes to
+leverage Hadoop-Gremlin (and its respective `GraphComputer` engines), then they need to provide, at minimum, a
+Hadoop2 `InputFormat<NullWritable,VertexWritable>` for their graph system. If the provider wishes to persist computed
+results back to their graph system (and not just to HDFS via a `FileOutputFormat`), then a graph system specific
+`OutputFormat<NullWritable,VertexWritable>` must be developed as well.
+
+Conceptually, `HadoopGraph` is a wrapper around a `Configuration` object. There is no "data" in the `HadoopGraph` as
+the `InputFormat` specifies where and how to get the graph data at OLAP (and OLTP) runtime. Thus, `HadoopGraph` is a
+small object with little overhead. Graph system providers should realize `HadoopGraph` as the gateway to the OLAP
+features offered by Hadoop-Gremlin. For example, a graph system specific `Graph.compute(Class<? extends GraphComputer>
+graphComputerClass)`-method may look as follows:
+
+[source,java]
+----
+public <C extends GraphComputer> C compute(final Class<C> graphComputerClass) throws IllegalArgumentException {
+ try {
+ if (AbstractHadoopGraphComputer.class.isAssignableFrom(graphComputerClass))
+ r
<TRUNCATED>
[18/22] incubator-tinkerpop git commit: Expanded the section on
"documentation" in developer docs.
Posted by sp...@apache.org.
Expanded the section on "documentation" in developer docs.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/3b7d7c92
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/3b7d7c92
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/3b7d7c92
Branch: refs/heads/master
Commit: 3b7d7c920ddda5253ee49572e23dab980f2e1a71
Parents: 7d20789
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Fri Nov 20 14:07:05 2015 -0500
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Fri Nov 20 14:07:05 2015 -0500
----------------------------------------------------------------------
docs/src/dev/developer/contributing.asciidoc | 18 +++++++++++++++---
1 file changed, 15 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/3b7d7c92/docs/src/dev/developer/contributing.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/dev/developer/contributing.asciidoc b/docs/src/dev/developer/contributing.asciidoc
index deaccfa..23be871 100644
--- a/docs/src/dev/developer/contributing.asciidoc
+++ b/docs/src/dev/developer/contributing.asciidoc
@@ -53,6 +53,7 @@ a successful `mvn clean install`, do `mvn verify -DskipIntegrationTests=false -p
Once a pull request is submitted it must go through <<rtc,review>> and will be merged once three TinkerPop committers
offer positive vote and achieve Apache consensus.
+[[building-testing]]
Building and Testing
--------------------
@@ -365,7 +366,18 @@ contains a NOTICE of its own. If so, include that NOTICE in the TinkerPop NOTICE
Documentation
~~~~~~~~~~~~~
-The TinkerPop docs are splitted into several subdirectories, each having its own index.asciidoc file. If a new AsciiDoc
-file is added, then it should also be included in the index.asciidoc file, otherwise the preprocessor will ignore it.
-Likewise, if a whole new section (subdirectory) is added, it must include an index.asciidoc file in order to be
+The documentation for TinkerPop is stored in the git repository in `docs/src/` and are then split into several
+subdirectories, each representing a "book" (or its own publishable body of work). If a new AsciiDoc file is added to
+a book, then it should also be included in the `index.asciidoc` file for that book, otherwise the preprocessor will
+ignore it. Likewise, if a whole new book (subdirectory) is added, it must include an `index.asciidoc` file to be
recognized by the AsciiDoc preprocessor.
+
+Adding a book also requires a change to the root `pom.xml` file. Find the "asciidoc" Maven profile and add a new
+`<execution>` to the `asciidoctor-maven-plugin` configuration. For each book in `docs/src/`, there should be a
+related `<execution>` that generates the HTML from the AsciiDoc. Follows the patterns already established by
+the existing `<execution>` entries, paying special attention to the pathing of the '<sourceDirectory>',
+`<outputDirectory>` and `<imagesdir>`. Note that the `<outputDirectory>` represents where the book will exist when
+uploaded to the server and should preserve the directory structure in git as referenced in `<sourceDirectory>`.
+
+Please see the <<building-testing,Building and Testing>> section for more information on how to generate the
+documentation.
\ No newline at end of file
[03/22] incubator-tinkerpop git commit: Made subdirectories for
various "books" in the docs.
Posted by sp...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/tutorials-getting-started.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/tutorials-getting-started.asciidoc b/docs/src/tutorials-getting-started.asciidoc
deleted file mode 100644
index 8787e57..0000000
--- a/docs/src/tutorials-getting-started.asciidoc
+++ /dev/null
@@ -1,571 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-
-image::apache-tinkerpop-logo.png[width=500]
-
-Getting Started
-===============
-
-link:http://tinkerpop.com[Apache TinkerPop] is an open source Graph Computing Framework. Within itself, TinkerPop
-represents a large collection of capabilities and technologies and, in its wider ecosystem, an additionally extended
-world of link:http://tinkerpop.incubator.apache.org/#graph-systems[third-party contributed] graph libraries and
-systems. TinkerPop's ecosystem can appear complex to newcomers of all experience, especially when glancing at the
-link:http://tinkerpop.incubator.apache.org/docs/x.y.z/index.html[reference documentation] for the first time.
-
-So, where do you get started with TinkerPop? How do you dive in quickly and get productive? Well - Gremlin, the
-most recognizable citizen of The TinkerPop, is here to help with this thirty minute tutorial. That's right - in just
-thirty short minutes, you too can be fit to start building graph applications with TinkerPop. Welcome to _The
-TinkerPop Workout - by Gremlin_!
-
-image::gremlin-gym.png[width=1024]
-
-The First Five Minutes
-----------------------
-
-It is quite possible to learn a lot in just five minutes with TinkerPop, but before doing so, a proper introduction of
-your trainer is in order. Meet Gremlin!
-
-image:gremlin-standing.png[width=125]
-
-Gremlin helps you navigate the vertices and edges of a graph. He is essentially your query language to graph
-databases, as link:http://sql2gremlin.com/[SQL] is the query language to relational databases. To tell Gremlin how
-he should "traverse" the graph (i.e. what you want your query to do) you need a way to provide him commands in the
-language he understands - and, of course, that language is called "Gremlin". For this task, you need one of
-TinkerPop's most important tools: link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#gremlin-console[The Gremlin Console].
-
-Download the console, unpackage it and start it:
-
-[source,text]
-----
-$ curl -L -O https://www.apache.org/dist/incubator/tinkerpop/x.y.z/apache-gremlin-console-x.y.z-bin.zip
-$ unzip apache-gremlin-console-x.y.z-bin.zip
-$ cd apache-gremlin-console-x.y.z-bin.zip
-$ bin/gremlin.sh
-
- \,,,/
- (o o)
------oOOo-(3)-oOOo-----
-plugin activated: tinkerpop.server
-plugin activated: tinkerpop.utilities
-plugin activated: tinkerpop.tinkergraph
-gremlin>
-----
-
-TIP: Windows users may use the included `bin/gremlin.bat` file to start the Gremlin Console.
-
-The Gremlin Console is a link:http://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop[REPL environment],
-which provides a nice way to learn Gremlin as you get immediate feedback for the code that you enter. This eliminates
-the more complex need to "create a project" to try things out. The console is not just for "getting started" however.
-You will find yourself using it for a variety of TinkerPop-related activities, such as loading data, administering
-graphs, working out complex traversals, etc.
-
-To get Gremlin to traverse a graph, you need a `Graph` instance, which holds the
-link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#_the_graph_structure[structure] and data of the
-graph. TinkerPop is a graph abstraction layer over different graph databases and different graph processors, so there
-are many `Graph` instances you can choose from to instantiate in the console. The best `Graph` instance to start with
-however is link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#tinkergraph-gremlin[TinkerGraph]. TinkerGraph
-is a fast, in-memory graph database with a small handful of configuration options, making it a good choice for beginners.
-
-TIP: TinkerGraph is not just a toy for beginners. It is useful in analyzing subgraphs taken from a large graph,
-working with a small static graph that doesn't change much, writing unit tests and other use cases where the graph
-can fit in memory.
-
-TIP: Resist the temptation to "get started" with more complex databases like link:http://thinkaurelius.github.io/titan/[Titan]
-or to delve into how to get link:http://tinkerpop.incubator.apache.org/docs/x.y.zg/#gremlin-server[Gremlin Server]
-working properly. Focusing on the basics, presented in this guide, builds a good foundation for all the other things
-TinkerPop offers.
-
-To make your process even easier, start with one of TinkerPop's "toy" graphs. These are "small" graphs designed to
-provide a quick start into querying. It is good to get familiar with them, as almost all TinkerPop documentation is based
-on them and when you need help and have to come to the link:http://groups.google.com/group/gremlin-users[mailing list],
-a failing example put in the context of the toy graphs can usually get you a fast answer to your problem.
-
-For your first graph, use the "Modern" graph which looks like this:
-
-image:tinkerpop-modern.png[width=500]
-
-It can be instantiated in the console this way:
-
-[gremlin-groovy]
-----
-graph = TinkerFactory.createModern()
-g = graph.traversal()
-----
-
-The first command creates a `Graph` instance named `graph`, which thus provides a reference to the data you want
-Gremlin to traverse. Unfortunately, just having `graph` doesn't provide Gremlin enough context to do his job. You
-also need something called a `TraversalSource`, which is generated by the second command. The `TraversalSource`
-provides additional information to Gremlin (such as the link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#traversalstrategy[traversal strategies]
-to apply and the link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#graphcomputer[traversal engine] to use) which
-provides him guidance on how to execute his trip around the `Graph`.
-
-With your `TraversalSource` `g` available it is now possible to ask Gremlin to traverse the `Graph`:
-
-[gremlin-groovy,modern]
-----
-g.V() <1>
-g.V(1) <2>
-g.V(1).values('name') <3>
-g.V(1).outE('knows') <4>
-g.V(1).outE('knows').inV().values('name') <5>
-g.V(1).out('knows').values('name') <6>
-g.V(1).out('knows').has('age', gt(30)).values('name') <7>
-----
-
-<1> Get all the vertices in the `Graph`.
-<2> Get a vertex with the unique identifier of "1".
-<3> Get the value of the `name` property on vertex with the unique identifier of "1".
-<4> Get the edges with the label "knows" for the vertex with the unique identifier of "1".
-<5> Get the names of the people that the vertex with the unique identifier of "1" "knows".
-<6> Note that when one uses `outE().inV()` as shown in the previous command, this can be shortened to just `out()`
-(similar to `inE().inV()` and `in` for incoming edges).
-<7> Get the names of the people vertex "1" knows who are over the age of 30.
-
-IMPORTANT: A `Traversal` is essentially an `Iterator` so if you have code like `x = g.V()`, the `x` does not contain
-the results of the `g.V()` query. Rather, that statement assigns an `Iterator` value to `x`. To get your results,
-you would then need to iterate through `x`. This understanding is *important* because in the context of the console
-typing `g.V()` instantly returns a value. The console does some magic for you by noticing that `g.V()` returns
-an `Iterator` and then automatically iterates the results. In short, when writing Gremlin outside of the console
-always remember that you must iterate your `Traversal` manually in some way for it to do anything.
-
-In this first five minutes with Gremlin, you've gotten the Gremlin Console installed, instantiated a `Graph` and
-`TraversalSource`, wrote some traversals and hopefully learned something about TinkerPop in general. You've only
-scratched the surface of what there is to know, but those accomplishments will help enable your understanding of the
-detailed sections to come.
-
-The Next Fifteen Minutes
-------------------------
-
-In the first five minutes of _The TinkerPop Workout - by Gremlin_, you learned some basics for traversing graphs. Of
-course, there wasn't much discussion about what a graph is. A graph is a collection of vertices (i.e. nodes, dots)
-and edges (i.e. relationships, lines), where a vertex is an entity which represents some domain object (e.g. a person,
-a place, etc.) and an edge represents the relationship between two vertices.
-
-image:modern-edge-1-to-3-1.png[width=300]
-
-The diagram above shows a graph with two vertices, one with a unique identifier of "1" and another with a unique
-identifier of "3". There is a edge connecting the two with a unique identifier of "9". It is important to consider
-that the edge has a direction which goes _out_ from vertex "1" and _in_ to vertex "3'.
-
-IMPORTANT: Most TinkerPop implementations do not allow for identifier assignment. They will rather assign
-their own identifiers and ignore assigned identifiers that you attempt to assign to them.
-
-A graph with elements that just have identifiers does not make for much of a database. To give some meaning to
-this basic structure, vertices and edges can each be given labels to categorize them.
-
-image:modern-edge-1-to-3-2.png[width=300]
-
-You can now see that a vertex "1" is a "person" and vertex "3" is a "software" vertex. They are joined by a "created"
-edge which allows you to see that a "person created software". The "label" and the "id" are reserved attributes of
-vertices and edges, but you can add your own arbitrary properties as well:
-
-image:modern-edge-1-to-3-3.png[width=325]
-
-This model is referred to as a _property graph_ and it provides a flexible and intuitive way in which to model your
-data.
-
-Creating a Graph
-^^^^^^^^^^^^^^^^
-
-As intuitive as it is to you, it is perhaps more intuitive to Gremlin himself, as vertices, edges and properties make
-up the very elements of his existence. It is indeed helpful to think of our friend, Gremlin, moving about a graph when
-developing traversals, as picturing his position as the link:http://tinkerpop.incubator.apache.org/docs/3.0.2-incubating/#_the_traverser[traverser]
-helps orient where you need him to go next. Let's use the two vertex, one edge graph we've been discussing above
-as an example. First, you need to create this graph:
-
-[gremlin-groovy]
-----
-graph = TinkerGraph.open()
-v1 = graph.addVertex(id, 1, label, "person", "name", "marko", "age", 29)
-v2 = graph.addVertex(id, 3, label, "software", "name", "lop", "lang", "java")
-v1.addEdge("created", v2, id, 9, "weight", 0.4)
-----
-
-There are a number of important things to consider in the above code. First, recall that `id` and `label` are
-"reserved" for special usage in TinkerPop. Those "keys" supplied to the creation method are statically imported to
-the console. You would normally refer to them as `T.id` and `T.label`.
-
-NOTE: The fully qualified name for `T` is `org.apache.tinkerpop.gremlin.structure.T`.
-
-Second, don't forget that you are working with TinkerGraph which allows for identifier assignment. That is _not_ the
-case with most graph databases.
-
-Finally, the label for an `Edge` is required and is thus part of the method signature of `addEdge()`. It is the first
-parameter supplied, followed by the `Vertex` to which `v1` should be connected. Therefore, this usage of `addEdge` is
-creating an edge that goes _out_ of `v1` and into `v2` with a label of "created".
-
-Graph Traversal - Staying Simple
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Now that Gremlin knows where the graph data is, you can ask him to get you some data from it by doing a traversal,
-which you can think of as executing some link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#_the_graph_process[process]
-over the structure of the graph. We can form our question in English and then translate it to Gremlin. For this
-initial example, let's ask Gremlin: "What software has Marko created?"
-
-To answer this question, we would want Gremlin to:
-
-. Find "marko" in the graph
-. Walk along the "created" edges to "software" vertices
-. Select the "name" property of the "software" vertices
-
-The English-based steps above largely translate to Gremlin's position in the graph and to the steps we need to take
-to ask him to answer our question. By stringing these steps together, we form a `Traversal` or the sequence of programmatic
-link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#graph-traversal-steps[steps] Gremlin needs to perform
-in order to get you an answer.
-
-Let's start with finding "marko". This operation is a filtering step as it searches the full set of vertices to match
-those that have the "name" property value of "marko". This can be done with the
-link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#has-step[has()] step as follows:
-
-[gremlin-groovy,modern]
-----
-g.V().has('name','marko')
-----
-
-NOTE: The variable `g` is the `TraversalSource`, which was introduced in the "The First Five Minutes". The
-`TraversalSource` is created with `graph.traversal()` and is the object used to spawn new traversals.
-
-We can picture this traversal in our little graph with Gremlin sitting on vertex "1".
-
-image:modern-edge-1-to-3-1-gremlin.png[width=325]
-
-When Gremlin is on a vertex or an edge, he has access to all the properties that are available to that element.
-
-IMPORTANT: The above query iterates all the vertices in the graph to get its answer. That's fine for our little example,
-but for multi-million or billion edge graphs that is a big problem. To solve this problem, you should look to use
-indices. TinkerPop does not provide an abstraction for index management. You should consult the documentation of the
-graph you have chosen and utilize its native API to create indices which will then speed up these types of lookups. Your
-traversals will remain unchanged however, as the indices will be used transparently at execution time.
-
-Now that Gremlin has found "marko", he can now consider the next step in the traversal where we ask him to "walk"
-along "created" edges to "software" vertices. As described earlier, edges have direction, so we have to tell Gremlin
-what direction to follow. In this case, we want him to traverse on outgoing edges from the "marko" vertex. For this,
-we use the link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#vertex-steps[outE] step.
-
-[gremlin-groovy,modern]
-----
-g.V().has('name','marko').outE('created')
-----
-
-At this point, you can picture Gremlin moving from the "marko" vertex to the "created" edge.
-
-image:modern-edge-1-to-3-2-gremlin.png[width=325]
-
-To get to the vertex on the other end of the edge, you need to tell Gremlin to move from the edge to the incoming
-vertex with `inV()`.
-
-[gremlin-groovy,modern]
-----
-g.V().has('name','marko').outE('created').inV()
-----
-
-You can now picture Gremlin on the "software" vertex as follows:
-
-image:modern-edge-1-to-3-3-gremlin.png[width=325]
-
-As you are not asking Gremlin to do anything with the properties of the "created" edge, you can simplify the
-statement above with:
-
-[gremlin-groovy,modern]
-----
-g.V().has('name','marko').out('created')
-----
-
-Finally, now that Gremlin has reached the "software that Marko created", he has access to the properties of the
-"software" vertex and you can therefore ask Gremlin to extract the value of the "name" property as follows:
-
-[gremlin-groovy,modern]
-----
-g.V().has('name','marko').out('created').values('name')
-----
-
-You should now be able to see the connection Gremlin has to the structure of the graph and how Gremlin maneuvers from
-vertices to edges and so on. Your ability to string together steps to ask Gremlin to do more complex things, depends
-on your understanding of these basic concepts.
-
-Graph Traversal - Increasing Complexity
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-Armed with the knowledge from the previous section, let's ask Gremlin to perform some more difficult traversal tasks.
-There's not much more that can be done with the "baby" graph we had, so let's return to the "modern" toy graph from
-the "five minutes section". Recall that you can create this `Graph` and establish a `TraversalSource` with:
-
-[gremlin-groovy]
-----
-graph = TinkerFactory.createModern()
-g = graph.traversal()
-----
-
-Earlier we'd used the `has()` step to tell Gremlin how to find the "marko" vertex. Let's look at some other ways to
-use `has()`. What if we wanted Gremlin to find the "age" values of both "vadas" and "marko"? In this case we could
-use the `within` comparator with `has()` as follows:
-
-[gremlin-groovy,modern]
-----
-g.V().has('name',within('vadas','marko')).values('age')
-----
-
-It is worth noting that `within` is statically imported from `P` to the Gremlin Console (much like `T` is, as described
-earlier).
-
-NOTE: The fully qualified name for `P` is `org.apache.tinkerpop.gremlin.process.traversal.P`.
-
-If we wanted to ask Gremlin the average age of "vadas" and "marko" we could use the
-link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#mean-step[mean()] step as follows:
-
-[gremlin-groovy,modern]
-----
-g.V().has('name',within('vadas','marko')).values('age').mean()
-----
-
-Another method of filtering is seen in the use of the link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#where-step[where]
-step. We know how to find the "software" that "marko" created:
-
-[gremlin-groovy,modern]
-----
-g.V().has('name','marko').out('created')
-----
-
-image:gremlin-on-software-vertex.png[width=350,float=right] Let's extend on that query to try to learn who "marko"
-collaborates with when it comes to the software he created. In other words, let's try to answer the question of: "Who
-are the people that marko develops software with?" To do that, we should first picture Gremlin where we left him in
-the previous query. He was standing on the "software" vertex. To find out who "created" that "software", we need to
-have Gremlin traverse back _in_ along the "created" edges to find the "person" vertices tied to it.
-
-[gremlin-groovy,modern]
-----
-g.V().has('name','marko').out('created').in('created').values('name')
-----
-
-So that's nice, we can see that "peter", "josh" and "marko" are both responsible for creating "lop". Of course, we already
-know about the involvement of "marko" and it seems strange to say that "marko" collaborates with himself, so excluding
-"marko" from the results seems logical. The following traversal handles that exclusion:
-
-[gremlin-groovy,modern]
-----
-g.V().has('name','marko').as('exclude').out('created').in('created').where(neq('exclude')).values('name')
-----
-
-We made two additions to the traversal to make it exclude "marko" from the results. First, we added the
-link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#as-step[as()] step. The `as()` step is not really a "step",
-but a "step modulator" - something that adds features to a step or the traversal. Here, the `as('exclude')` labels
-the `has()` step with the name "exclude" and all values that pass through that step are held in that label for later
-use. In this case, the "marko" vertex is the only vertex to pass through that point, so it is held in "exclude".
-
-The other addition that was made was the `where()` step which is a filter step like `has()`. The `where()` is
-positioned after the `in()` step that has "person" vertices, which means that the `where()` filter is occurring
-on the list of "marko" collaborators. The `where()` specifies that the "person" vertices passing through it should
-not equal (i.e. `neq()`) the contents of the "exclude" label. As it just contains the "marko" vertex, the `where()`
-filters out the "marko" that we get when we traverse back _in_ on the "created" edges.
-
-You will find many uses of `as()`. Here it is in combination with link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#select-step[select]:
-
-[gremlin-groovy,modern]
-----
-g.V().as('a').out().as('b').out().as('c').select('a','b','c')
-----
-
-In the above example, we tell Gremlin to iterate through all vertices and traverse _out_ twice from each. Gremlin
-will label each vertex in that path with "a", "b" and "c", respectively. We can then use `select` to extract the
-contents of that label.
-
-Another common but important step is the link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#group-step[group()]
-step and its related step modulator called link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#by-step[by()]. If
-we wanted to ask Gremlin to group all the vertices in the graph by their vertex label we could do:
-
-[gremlin-groovy,modern]
-----
-g.V().group().by(label)
-----
-
-The use of `by()` here provides the mechanism by which to do the grouping. In this case, we've asked Gremlin to
-use the `label` (which, again, is an automatic static import from `T` in the console). We can't really tell much
-about our distribution though because we just have vertex unique identifiers as output. To make that nicer we
-could ask Gremlin to get us the value of the "name" property from those vertices, by supplying another `by()`
-modulator to `group()` to transform the values.
-
-[gremlin-groovy,modern]
-----
-g.V().group().by(label).by('name')
-----
-
-In this section, you have learned a bit more about what property graphs are and how Gremlin interacts with them.
-You also learned how to envision Gremlin moving about a graph and how to use some of the more complex, but commonly
-utilized traversal steps. You are now ready to think about TinkerPop in terms of its wider applicability to
-graph computing.
-
-The Final Ten Minutes
----------------------
-
-In these final ten minutes of _The TinkerPop Workout - by Gremlin_ we'll look at TinkerPop from a higher level and
-introduce different features of the stack in order to orient you with what it offers. In this way, you can
-identify areas of interest and dig into the details from there.
-
-Why TinkerPop?
-^^^^^^^^^^^^^^
-
-image:provider-integration.png[float=right,width=350] The goal of TinkerPop, as a Graph Computing Framework, is to make it
-easy for developers to create graph applications by providing APIs and tools that simplify their endeavors. One of
-the fundamental aspects to what TinkerPop offers in this area lies in the fact that TinkerPop is an abstraction layer
-over different graph databases and different graph processors. As an abstraction layer, TinkerPop provides a way to
-avoid vendor lock-in to a specific database or processor. This capability provides immense value to developers who
-are thus afforded options in their architecture and development because:
-
-* They can try different implementations using the same code to decide which is best for their environment.
-* They can grow into a particular implementation if they so desire - start with a graph that is designed to scale
-within a single machine and then later switch to a graph that is designed to scale horizontally.
-* They can feel more confident in graph technology choices, as advances in the state of different provider
-implementations are behind TinkerPop APIs, which open the possibility to switch providers with limited impact.
-
-TinkerPop has always had the vision of being an abstraction over different graph databases. That much
-is not new and dates back to TinkerPop 1.x. It is in TinkerPop 3.x however that we see the introduction of the notion
-that TinkerPop is also an abstraction over different graph processors like link:http://spark.apache.org[Spark] and
-link:http://giraph.apache.org/[Giraph]. The scope of this tutorial does not permit it to delve into
-"graph processors", but the short story is that the same Gremlin statement we wrote in the examples above can be
-executed to run in distributed fashion over Spark or Hadoop. The changes required to the code to do this are not
-in the traversal itself, but in the definition of the `TraversalSource`. You can again see why we encourage, graph
-operations to be executed through that class as opposed to just using `Graph`. You can read more about these
-features in this section on link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#hadoop-gremlin[hadoop-gremlin].
-
-TIP: To maintain an abstraction over `Graph` creation use `GraphFactory.open()` to construct new instances. See
-the documentation for individual `Graph` implementations to learn about the configuration options to provide.
-
-Loading Data
-^^^^^^^^^^^^
-
-image:gremlin-to-the-7.png[width=100,float=left] There are many strategies for getting data into your graph. As you are just
-getting started, let's look the more simple methods aimed at "smaller" graphs. A "small" graph, in this context, is
-one that has less than ten million edges. The most direct way to load this data is to write a Groovy script that can
-be executed in the Gremlin Console, a tool that you should be well familiar with at this point. For our example, let's
-use the link:http://snap.stanford.edu/data/wiki-Vote.html[Wikipedia Vote Network] data set which contains 7,115
-vertices and 103,689 edges.
-
-[source,text]
-----
-$ curl -L -O http://snap.stanford.edu/data/wiki-Vote.txt.gz
-$ gunzip wiki-Vote.txt.gz
-----
-
-The data is contained in a tab-delimited structure where vertices are Wikipedia users and edges from one user to
-another implies a "vote" relationship. Here is the script to parse the file and generate the `Graph` instance using
-TinkerGraph:
-
-[source,groovy]
-----
-graph = TinkerGraph.open()
-graph.createIndex('userId', Vertex.class) <1>
-
-g = graph.traversal()
-
-getOrCreate = { id ->
- g.V().has('userId', id).tryNext().orElseGet{ g.addV('userId', id).next() }
-}
-
-new File('wiki-Vote.txt').eachLine {
- if (!it.startsWith("#")){
- (fromVertex, toVertex) = it.split('\t').collect(getOrCreate) <2>
- fromVertex.addEdge('votesFor', toVertex)
- }
-}
-----
-
-<1> To ensure fast lookups of vertices, we need an index. The `createIndex()` method is a method native to
-TinkerGraph. Please consult your graph databases documentation for their index creation approaches.
-<2> We are iterating each line of the `wiki-Vote.txt` file and this line splits the line on the delimiter, then
-uses some neat Groovy syntax to apply the `getOrCreate()` function to each of the two `userId` fields encountered in
-the line and stores those vertices in the `fromVertex` and `toVertex` variables respectively.
-
-NOTE: While this is a tab-delimited structure, this same pattern can be applied
-to any data source you require and Groovy tends to have nice libraries that can help making working with data
-link:http://thinkaurelius.com/2013/02/04/polyglot-persistence-and-query-with-gremlin/[quite enjoyable].
-
-WARNING: Take care if using a `Graph` implementation that supports
-link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#transactions[transactions]. As TinkerGraph does not, there is
-no need to `commit()`. If your `Graph` does support transactions, intermediate commits during load will need to be
-applied.
-
-To load larger data sets you should read about the
-link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#bulkloadervertexprogram[BulkLoaderVertexProgram] (BLVP), which
-provides a generalized method for loading graphs of virtually any size.
-
-Gremlin Server
-^^^^^^^^^^^^^^
-
-image:gremlin-server-protocol.png[width=325,float=right] link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#gremlin-server[Gremlin Server]
-provides a way to remotely execute Gremlin scripts against one or more `Graph` instances hosted within it. It does
-this by exposing different endpoints, such as link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#_connecting_via_rest[REST]
-and link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#_connecting_via_java[websockets], which allow a request
-containing a Gremlin script to be processed with results returned.
-
-[source,text]
-----
-$ curl -L -O https://www.apache.org/dist/incubator/tinkerpop/x.y.z/apache-gremlin-server-x.y.z-bin.zip
-$ unzip apache-gremlin-server-x.y.z-bin.zip
-$ cd apache-gremlin-server-x.y.z-bin.zip
-$ bin/gremlin-server.sh conf/gremlin-server-modern.yaml
-[INFO] GremlinServer -
- \,,,/
- (o o)
------oOOo-(3)-oOOo-----
-
-[INFO] GremlinServer - Configuring Gremlin Server from conf/gremlin-server-rest-modern.yaml
-...
-[INFO] GremlinServer$1 - Channel started at port 8182.
-----
-
-[source,text]
-$ curl -X POST -d "{\"gremlin\":\"g.V(x).out().values('name')\", \"language\":\"gremlin-groovy\", \"bindings\":{\"x\":1}}" "http://localhost:8182"
-
-[source,json]
-----
-{
- "requestId": "abe3be05-1e86-481a-85e0-c59ad8a37c6b",
- "status": {
- "message": "",
- "code": 200,
- "attributes": {}
- },
- "result": {
- "data": [
- "lop",
- "vadas",
- "josh"
- ],
- "meta": {}
- }
-}
-----
-
-IMPORTANT: Take careful note of the use of "bindings" in the arguments on the request. These are variables that are
-applied to the script on execution and is essentially a way to parameterize your scripts. This "parameterization" is
-critical to link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#_best_practices[performance]. Whenever
-possible, parameterize your queries.
-
-As mentioned earlier, Gremlin Server can also be configured with a websockets endpoint. This endpoint has an
-embedded link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#_developing_a_driver[subprotocol] that allow a
-compliant driver to communicate with it. TinkerPop supplies a
-link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#_connecting_via_java[reference driver] written in Java, but
-there are drivers developed by third-parties for other link:http://tinkerpop.incubator.apache.org/#graph-libraries[languages]
-such as Python, Javascript, etc. Gremlin Server therefore represents the method by which non-JVM languages can
-interact with TinkerPop.
-
-Conclusion
-^^^^^^^^^^
-
-...and that is the end of _The TinkerPop Workout - by Gremlin_. You are hopefully feeling more confident in your
-TinkerPop skills and have a good overview of what the stack has to offer, as well as some entry points to further
-research within the reference documentation. Welcome to The TinkerPop!
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/tutorials/getting-started/index.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/tutorials/getting-started/index.asciidoc b/docs/src/tutorials/getting-started/index.asciidoc
new file mode 100644
index 0000000..8787e57
--- /dev/null
+++ b/docs/src/tutorials/getting-started/index.asciidoc
@@ -0,0 +1,571 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+
+image::apache-tinkerpop-logo.png[width=500]
+
+Getting Started
+===============
+
+link:http://tinkerpop.com[Apache TinkerPop] is an open source Graph Computing Framework. Within itself, TinkerPop
+represents a large collection of capabilities and technologies and, in its wider ecosystem, an additionally extended
+world of link:http://tinkerpop.incubator.apache.org/#graph-systems[third-party contributed] graph libraries and
+systems. TinkerPop's ecosystem can appear complex to newcomers of all experience, especially when glancing at the
+link:http://tinkerpop.incubator.apache.org/docs/x.y.z/index.html[reference documentation] for the first time.
+
+So, where do you get started with TinkerPop? How do you dive in quickly and get productive? Well - Gremlin, the
+most recognizable citizen of The TinkerPop, is here to help with this thirty minute tutorial. That's right - in just
+thirty short minutes, you too can be fit to start building graph applications with TinkerPop. Welcome to _The
+TinkerPop Workout - by Gremlin_!
+
+image::gremlin-gym.png[width=1024]
+
+The First Five Minutes
+----------------------
+
+It is quite possible to learn a lot in just five minutes with TinkerPop, but before doing so, a proper introduction of
+your trainer is in order. Meet Gremlin!
+
+image:gremlin-standing.png[width=125]
+
+Gremlin helps you navigate the vertices and edges of a graph. He is essentially your query language to graph
+databases, as link:http://sql2gremlin.com/[SQL] is the query language to relational databases. To tell Gremlin how
+he should "traverse" the graph (i.e. what you want your query to do) you need a way to provide him commands in the
+language he understands - and, of course, that language is called "Gremlin". For this task, you need one of
+TinkerPop's most important tools: link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#gremlin-console[The Gremlin Console].
+
+Download the console, unpackage it and start it:
+
+[source,text]
+----
+$ curl -L -O https://www.apache.org/dist/incubator/tinkerpop/x.y.z/apache-gremlin-console-x.y.z-bin.zip
+$ unzip apache-gremlin-console-x.y.z-bin.zip
+$ cd apache-gremlin-console-x.y.z-bin.zip
+$ bin/gremlin.sh
+
+ \,,,/
+ (o o)
+-----oOOo-(3)-oOOo-----
+plugin activated: tinkerpop.server
+plugin activated: tinkerpop.utilities
+plugin activated: tinkerpop.tinkergraph
+gremlin>
+----
+
+TIP: Windows users may use the included `bin/gremlin.bat` file to start the Gremlin Console.
+
+The Gremlin Console is a link:http://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop[REPL environment],
+which provides a nice way to learn Gremlin as you get immediate feedback for the code that you enter. This eliminates
+the more complex need to "create a project" to try things out. The console is not just for "getting started" however.
+You will find yourself using it for a variety of TinkerPop-related activities, such as loading data, administering
+graphs, working out complex traversals, etc.
+
+To get Gremlin to traverse a graph, you need a `Graph` instance, which holds the
+link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#_the_graph_structure[structure] and data of the
+graph. TinkerPop is a graph abstraction layer over different graph databases and different graph processors, so there
+are many `Graph` instances you can choose from to instantiate in the console. The best `Graph` instance to start with
+however is link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#tinkergraph-gremlin[TinkerGraph]. TinkerGraph
+is a fast, in-memory graph database with a small handful of configuration options, making it a good choice for beginners.
+
+TIP: TinkerGraph is not just a toy for beginners. It is useful in analyzing subgraphs taken from a large graph,
+working with a small static graph that doesn't change much, writing unit tests and other use cases where the graph
+can fit in memory.
+
+TIP: Resist the temptation to "get started" with more complex databases like link:http://thinkaurelius.github.io/titan/[Titan]
+or to delve into how to get link:http://tinkerpop.incubator.apache.org/docs/x.y.zg/#gremlin-server[Gremlin Server]
+working properly. Focusing on the basics, presented in this guide, builds a good foundation for all the other things
+TinkerPop offers.
+
+To make your process even easier, start with one of TinkerPop's "toy" graphs. These are "small" graphs designed to
+provide a quick start into querying. It is good to get familiar with them, as almost all TinkerPop documentation is based
+on them and when you need help and have to come to the link:http://groups.google.com/group/gremlin-users[mailing list],
+a failing example put in the context of the toy graphs can usually get you a fast answer to your problem.
+
+For your first graph, use the "Modern" graph which looks like this:
+
+image:tinkerpop-modern.png[width=500]
+
+It can be instantiated in the console this way:
+
+[gremlin-groovy]
+----
+graph = TinkerFactory.createModern()
+g = graph.traversal()
+----
+
+The first command creates a `Graph` instance named `graph`, which thus provides a reference to the data you want
+Gremlin to traverse. Unfortunately, just having `graph` doesn't provide Gremlin enough context to do his job. You
+also need something called a `TraversalSource`, which is generated by the second command. The `TraversalSource`
+provides additional information to Gremlin (such as the link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#traversalstrategy[traversal strategies]
+to apply and the link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#graphcomputer[traversal engine] to use) which
+provides him guidance on how to execute his trip around the `Graph`.
+
+With your `TraversalSource` `g` available it is now possible to ask Gremlin to traverse the `Graph`:
+
+[gremlin-groovy,modern]
+----
+g.V() <1>
+g.V(1) <2>
+g.V(1).values('name') <3>
+g.V(1).outE('knows') <4>
+g.V(1).outE('knows').inV().values('name') <5>
+g.V(1).out('knows').values('name') <6>
+g.V(1).out('knows').has('age', gt(30)).values('name') <7>
+----
+
+<1> Get all the vertices in the `Graph`.
+<2> Get a vertex with the unique identifier of "1".
+<3> Get the value of the `name` property on vertex with the unique identifier of "1".
+<4> Get the edges with the label "knows" for the vertex with the unique identifier of "1".
+<5> Get the names of the people that the vertex with the unique identifier of "1" "knows".
+<6> Note that when one uses `outE().inV()` as shown in the previous command, this can be shortened to just `out()`
+(similar to `inE().inV()` and `in` for incoming edges).
+<7> Get the names of the people vertex "1" knows who are over the age of 30.
+
+IMPORTANT: A `Traversal` is essentially an `Iterator` so if you have code like `x = g.V()`, the `x` does not contain
+the results of the `g.V()` query. Rather, that statement assigns an `Iterator` value to `x`. To get your results,
+you would then need to iterate through `x`. This understanding is *important* because in the context of the console
+typing `g.V()` instantly returns a value. The console does some magic for you by noticing that `g.V()` returns
+an `Iterator` and then automatically iterates the results. In short, when writing Gremlin outside of the console
+always remember that you must iterate your `Traversal` manually in some way for it to do anything.
+
+In this first five minutes with Gremlin, you've gotten the Gremlin Console installed, instantiated a `Graph` and
+`TraversalSource`, wrote some traversals and hopefully learned something about TinkerPop in general. You've only
+scratched the surface of what there is to know, but those accomplishments will help enable your understanding of the
+detailed sections to come.
+
+The Next Fifteen Minutes
+------------------------
+
+In the first five minutes of _The TinkerPop Workout - by Gremlin_, you learned some basics for traversing graphs. Of
+course, there wasn't much discussion about what a graph is. A graph is a collection of vertices (i.e. nodes, dots)
+and edges (i.e. relationships, lines), where a vertex is an entity which represents some domain object (e.g. a person,
+a place, etc.) and an edge represents the relationship between two vertices.
+
+image:modern-edge-1-to-3-1.png[width=300]
+
+The diagram above shows a graph with two vertices, one with a unique identifier of "1" and another with a unique
+identifier of "3". There is a edge connecting the two with a unique identifier of "9". It is important to consider
+that the edge has a direction which goes _out_ from vertex "1" and _in_ to vertex "3'.
+
+IMPORTANT: Most TinkerPop implementations do not allow for identifier assignment. They will rather assign
+their own identifiers and ignore assigned identifiers that you attempt to assign to them.
+
+A graph with elements that just have identifiers does not make for much of a database. To give some meaning to
+this basic structure, vertices and edges can each be given labels to categorize them.
+
+image:modern-edge-1-to-3-2.png[width=300]
+
+You can now see that a vertex "1" is a "person" and vertex "3" is a "software" vertex. They are joined by a "created"
+edge which allows you to see that a "person created software". The "label" and the "id" are reserved attributes of
+vertices and edges, but you can add your own arbitrary properties as well:
+
+image:modern-edge-1-to-3-3.png[width=325]
+
+This model is referred to as a _property graph_ and it provides a flexible and intuitive way in which to model your
+data.
+
+Creating a Graph
+^^^^^^^^^^^^^^^^
+
+As intuitive as it is to you, it is perhaps more intuitive to Gremlin himself, as vertices, edges and properties make
+up the very elements of his existence. It is indeed helpful to think of our friend, Gremlin, moving about a graph when
+developing traversals, as picturing his position as the link:http://tinkerpop.incubator.apache.org/docs/3.0.2-incubating/#_the_traverser[traverser]
+helps orient where you need him to go next. Let's use the two vertex, one edge graph we've been discussing above
+as an example. First, you need to create this graph:
+
+[gremlin-groovy]
+----
+graph = TinkerGraph.open()
+v1 = graph.addVertex(id, 1, label, "person", "name", "marko", "age", 29)
+v2 = graph.addVertex(id, 3, label, "software", "name", "lop", "lang", "java")
+v1.addEdge("created", v2, id, 9, "weight", 0.4)
+----
+
+There are a number of important things to consider in the above code. First, recall that `id` and `label` are
+"reserved" for special usage in TinkerPop. Those "keys" supplied to the creation method are statically imported to
+the console. You would normally refer to them as `T.id` and `T.label`.
+
+NOTE: The fully qualified name for `T` is `org.apache.tinkerpop.gremlin.structure.T`.
+
+Second, don't forget that you are working with TinkerGraph which allows for identifier assignment. That is _not_ the
+case with most graph databases.
+
+Finally, the label for an `Edge` is required and is thus part of the method signature of `addEdge()`. It is the first
+parameter supplied, followed by the `Vertex` to which `v1` should be connected. Therefore, this usage of `addEdge` is
+creating an edge that goes _out_ of `v1` and into `v2` with a label of "created".
+
+Graph Traversal - Staying Simple
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Now that Gremlin knows where the graph data is, you can ask him to get you some data from it by doing a traversal,
+which you can think of as executing some link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#_the_graph_process[process]
+over the structure of the graph. We can form our question in English and then translate it to Gremlin. For this
+initial example, let's ask Gremlin: "What software has Marko created?"
+
+To answer this question, we would want Gremlin to:
+
+. Find "marko" in the graph
+. Walk along the "created" edges to "software" vertices
+. Select the "name" property of the "software" vertices
+
+The English-based steps above largely translate to Gremlin's position in the graph and to the steps we need to take
+to ask him to answer our question. By stringing these steps together, we form a `Traversal` or the sequence of programmatic
+link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#graph-traversal-steps[steps] Gremlin needs to perform
+in order to get you an answer.
+
+Let's start with finding "marko". This operation is a filtering step as it searches the full set of vertices to match
+those that have the "name" property value of "marko". This can be done with the
+link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#has-step[has()] step as follows:
+
+[gremlin-groovy,modern]
+----
+g.V().has('name','marko')
+----
+
+NOTE: The variable `g` is the `TraversalSource`, which was introduced in the "The First Five Minutes". The
+`TraversalSource` is created with `graph.traversal()` and is the object used to spawn new traversals.
+
+We can picture this traversal in our little graph with Gremlin sitting on vertex "1".
+
+image:modern-edge-1-to-3-1-gremlin.png[width=325]
+
+When Gremlin is on a vertex or an edge, he has access to all the properties that are available to that element.
+
+IMPORTANT: The above query iterates all the vertices in the graph to get its answer. That's fine for our little example,
+but for multi-million or billion edge graphs that is a big problem. To solve this problem, you should look to use
+indices. TinkerPop does not provide an abstraction for index management. You should consult the documentation of the
+graph you have chosen and utilize its native API to create indices which will then speed up these types of lookups. Your
+traversals will remain unchanged however, as the indices will be used transparently at execution time.
+
+Now that Gremlin has found "marko", he can now consider the next step in the traversal where we ask him to "walk"
+along "created" edges to "software" vertices. As described earlier, edges have direction, so we have to tell Gremlin
+what direction to follow. In this case, we want him to traverse on outgoing edges from the "marko" vertex. For this,
+we use the link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#vertex-steps[outE] step.
+
+[gremlin-groovy,modern]
+----
+g.V().has('name','marko').outE('created')
+----
+
+At this point, you can picture Gremlin moving from the "marko" vertex to the "created" edge.
+
+image:modern-edge-1-to-3-2-gremlin.png[width=325]
+
+To get to the vertex on the other end of the edge, you need to tell Gremlin to move from the edge to the incoming
+vertex with `inV()`.
+
+[gremlin-groovy,modern]
+----
+g.V().has('name','marko').outE('created').inV()
+----
+
+You can now picture Gremlin on the "software" vertex as follows:
+
+image:modern-edge-1-to-3-3-gremlin.png[width=325]
+
+As you are not asking Gremlin to do anything with the properties of the "created" edge, you can simplify the
+statement above with:
+
+[gremlin-groovy,modern]
+----
+g.V().has('name','marko').out('created')
+----
+
+Finally, now that Gremlin has reached the "software that Marko created", he has access to the properties of the
+"software" vertex and you can therefore ask Gremlin to extract the value of the "name" property as follows:
+
+[gremlin-groovy,modern]
+----
+g.V().has('name','marko').out('created').values('name')
+----
+
+You should now be able to see the connection Gremlin has to the structure of the graph and how Gremlin maneuvers from
+vertices to edges and so on. Your ability to string together steps to ask Gremlin to do more complex things, depends
+on your understanding of these basic concepts.
+
+Graph Traversal - Increasing Complexity
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Armed with the knowledge from the previous section, let's ask Gremlin to perform some more difficult traversal tasks.
+There's not much more that can be done with the "baby" graph we had, so let's return to the "modern" toy graph from
+the "five minutes section". Recall that you can create this `Graph` and establish a `TraversalSource` with:
+
+[gremlin-groovy]
+----
+graph = TinkerFactory.createModern()
+g = graph.traversal()
+----
+
+Earlier we'd used the `has()` step to tell Gremlin how to find the "marko" vertex. Let's look at some other ways to
+use `has()`. What if we wanted Gremlin to find the "age" values of both "vadas" and "marko"? In this case we could
+use the `within` comparator with `has()` as follows:
+
+[gremlin-groovy,modern]
+----
+g.V().has('name',within('vadas','marko')).values('age')
+----
+
+It is worth noting that `within` is statically imported from `P` to the Gremlin Console (much like `T` is, as described
+earlier).
+
+NOTE: The fully qualified name for `P` is `org.apache.tinkerpop.gremlin.process.traversal.P`.
+
+If we wanted to ask Gremlin the average age of "vadas" and "marko" we could use the
+link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#mean-step[mean()] step as follows:
+
+[gremlin-groovy,modern]
+----
+g.V().has('name',within('vadas','marko')).values('age').mean()
+----
+
+Another method of filtering is seen in the use of the link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#where-step[where]
+step. We know how to find the "software" that "marko" created:
+
+[gremlin-groovy,modern]
+----
+g.V().has('name','marko').out('created')
+----
+
+image:gremlin-on-software-vertex.png[width=350,float=right] Let's extend on that query to try to learn who "marko"
+collaborates with when it comes to the software he created. In other words, let's try to answer the question of: "Who
+are the people that marko develops software with?" To do that, we should first picture Gremlin where we left him in
+the previous query. He was standing on the "software" vertex. To find out who "created" that "software", we need to
+have Gremlin traverse back _in_ along the "created" edges to find the "person" vertices tied to it.
+
+[gremlin-groovy,modern]
+----
+g.V().has('name','marko').out('created').in('created').values('name')
+----
+
+So that's nice, we can see that "peter", "josh" and "marko" are both responsible for creating "lop". Of course, we already
+know about the involvement of "marko" and it seems strange to say that "marko" collaborates with himself, so excluding
+"marko" from the results seems logical. The following traversal handles that exclusion:
+
+[gremlin-groovy,modern]
+----
+g.V().has('name','marko').as('exclude').out('created').in('created').where(neq('exclude')).values('name')
+----
+
+We made two additions to the traversal to make it exclude "marko" from the results. First, we added the
+link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#as-step[as()] step. The `as()` step is not really a "step",
+but a "step modulator" - something that adds features to a step or the traversal. Here, the `as('exclude')` labels
+the `has()` step with the name "exclude" and all values that pass through that step are held in that label for later
+use. In this case, the "marko" vertex is the only vertex to pass through that point, so it is held in "exclude".
+
+The other addition that was made was the `where()` step which is a filter step like `has()`. The `where()` is
+positioned after the `in()` step that has "person" vertices, which means that the `where()` filter is occurring
+on the list of "marko" collaborators. The `where()` specifies that the "person" vertices passing through it should
+not equal (i.e. `neq()`) the contents of the "exclude" label. As it just contains the "marko" vertex, the `where()`
+filters out the "marko" that we get when we traverse back _in_ on the "created" edges.
+
+You will find many uses of `as()`. Here it is in combination with link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#select-step[select]:
+
+[gremlin-groovy,modern]
+----
+g.V().as('a').out().as('b').out().as('c').select('a','b','c')
+----
+
+In the above example, we tell Gremlin to iterate through all vertices and traverse _out_ twice from each. Gremlin
+will label each vertex in that path with "a", "b" and "c", respectively. We can then use `select` to extract the
+contents of that label.
+
+Another common but important step is the link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#group-step[group()]
+step and its related step modulator called link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#by-step[by()]. If
+we wanted to ask Gremlin to group all the vertices in the graph by their vertex label we could do:
+
+[gremlin-groovy,modern]
+----
+g.V().group().by(label)
+----
+
+The use of `by()` here provides the mechanism by which to do the grouping. In this case, we've asked Gremlin to
+use the `label` (which, again, is an automatic static import from `T` in the console). We can't really tell much
+about our distribution though because we just have vertex unique identifiers as output. To make that nicer we
+could ask Gremlin to get us the value of the "name" property from those vertices, by supplying another `by()`
+modulator to `group()` to transform the values.
+
+[gremlin-groovy,modern]
+----
+g.V().group().by(label).by('name')
+----
+
+In this section, you have learned a bit more about what property graphs are and how Gremlin interacts with them.
+You also learned how to envision Gremlin moving about a graph and how to use some of the more complex, but commonly
+utilized traversal steps. You are now ready to think about TinkerPop in terms of its wider applicability to
+graph computing.
+
+The Final Ten Minutes
+---------------------
+
+In these final ten minutes of _The TinkerPop Workout - by Gremlin_ we'll look at TinkerPop from a higher level and
+introduce different features of the stack in order to orient you with what it offers. In this way, you can
+identify areas of interest and dig into the details from there.
+
+Why TinkerPop?
+^^^^^^^^^^^^^^
+
+image:provider-integration.png[float=right,width=350] The goal of TinkerPop, as a Graph Computing Framework, is to make it
+easy for developers to create graph applications by providing APIs and tools that simplify their endeavors. One of
+the fundamental aspects to what TinkerPop offers in this area lies in the fact that TinkerPop is an abstraction layer
+over different graph databases and different graph processors. As an abstraction layer, TinkerPop provides a way to
+avoid vendor lock-in to a specific database or processor. This capability provides immense value to developers who
+are thus afforded options in their architecture and development because:
+
+* They can try different implementations using the same code to decide which is best for their environment.
+* They can grow into a particular implementation if they so desire - start with a graph that is designed to scale
+within a single machine and then later switch to a graph that is designed to scale horizontally.
+* They can feel more confident in graph technology choices, as advances in the state of different provider
+implementations are behind TinkerPop APIs, which open the possibility to switch providers with limited impact.
+
+TinkerPop has always had the vision of being an abstraction over different graph databases. That much
+is not new and dates back to TinkerPop 1.x. It is in TinkerPop 3.x however that we see the introduction of the notion
+that TinkerPop is also an abstraction over different graph processors like link:http://spark.apache.org[Spark] and
+link:http://giraph.apache.org/[Giraph]. The scope of this tutorial does not permit it to delve into
+"graph processors", but the short story is that the same Gremlin statement we wrote in the examples above can be
+executed to run in distributed fashion over Spark or Hadoop. The changes required to the code to do this are not
+in the traversal itself, but in the definition of the `TraversalSource`. You can again see why we encourage, graph
+operations to be executed through that class as opposed to just using `Graph`. You can read more about these
+features in this section on link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#hadoop-gremlin[hadoop-gremlin].
+
+TIP: To maintain an abstraction over `Graph` creation use `GraphFactory.open()` to construct new instances. See
+the documentation for individual `Graph` implementations to learn about the configuration options to provide.
+
+Loading Data
+^^^^^^^^^^^^
+
+image:gremlin-to-the-7.png[width=100,float=left] There are many strategies for getting data into your graph. As you are just
+getting started, let's look the more simple methods aimed at "smaller" graphs. A "small" graph, in this context, is
+one that has less than ten million edges. The most direct way to load this data is to write a Groovy script that can
+be executed in the Gremlin Console, a tool that you should be well familiar with at this point. For our example, let's
+use the link:http://snap.stanford.edu/data/wiki-Vote.html[Wikipedia Vote Network] data set which contains 7,115
+vertices and 103,689 edges.
+
+[source,text]
+----
+$ curl -L -O http://snap.stanford.edu/data/wiki-Vote.txt.gz
+$ gunzip wiki-Vote.txt.gz
+----
+
+The data is contained in a tab-delimited structure where vertices are Wikipedia users and edges from one user to
+another implies a "vote" relationship. Here is the script to parse the file and generate the `Graph` instance using
+TinkerGraph:
+
+[source,groovy]
+----
+graph = TinkerGraph.open()
+graph.createIndex('userId', Vertex.class) <1>
+
+g = graph.traversal()
+
+getOrCreate = { id ->
+ g.V().has('userId', id).tryNext().orElseGet{ g.addV('userId', id).next() }
+}
+
+new File('wiki-Vote.txt').eachLine {
+ if (!it.startsWith("#")){
+ (fromVertex, toVertex) = it.split('\t').collect(getOrCreate) <2>
+ fromVertex.addEdge('votesFor', toVertex)
+ }
+}
+----
+
+<1> To ensure fast lookups of vertices, we need an index. The `createIndex()` method is a method native to
+TinkerGraph. Please consult your graph databases documentation for their index creation approaches.
+<2> We are iterating each line of the `wiki-Vote.txt` file and this line splits the line on the delimiter, then
+uses some neat Groovy syntax to apply the `getOrCreate()` function to each of the two `userId` fields encountered in
+the line and stores those vertices in the `fromVertex` and `toVertex` variables respectively.
+
+NOTE: While this is a tab-delimited structure, this same pattern can be applied
+to any data source you require and Groovy tends to have nice libraries that can help making working with data
+link:http://thinkaurelius.com/2013/02/04/polyglot-persistence-and-query-with-gremlin/[quite enjoyable].
+
+WARNING: Take care if using a `Graph` implementation that supports
+link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#transactions[transactions]. As TinkerGraph does not, there is
+no need to `commit()`. If your `Graph` does support transactions, intermediate commits during load will need to be
+applied.
+
+To load larger data sets you should read about the
+link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#bulkloadervertexprogram[BulkLoaderVertexProgram] (BLVP), which
+provides a generalized method for loading graphs of virtually any size.
+
+Gremlin Server
+^^^^^^^^^^^^^^
+
+image:gremlin-server-protocol.png[width=325,float=right] link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#gremlin-server[Gremlin Server]
+provides a way to remotely execute Gremlin scripts against one or more `Graph` instances hosted within it. It does
+this by exposing different endpoints, such as link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#_connecting_via_rest[REST]
+and link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#_connecting_via_java[websockets], which allow a request
+containing a Gremlin script to be processed with results returned.
+
+[source,text]
+----
+$ curl -L -O https://www.apache.org/dist/incubator/tinkerpop/x.y.z/apache-gremlin-server-x.y.z-bin.zip
+$ unzip apache-gremlin-server-x.y.z-bin.zip
+$ cd apache-gremlin-server-x.y.z-bin.zip
+$ bin/gremlin-server.sh conf/gremlin-server-modern.yaml
+[INFO] GremlinServer -
+ \,,,/
+ (o o)
+-----oOOo-(3)-oOOo-----
+
+[INFO] GremlinServer - Configuring Gremlin Server from conf/gremlin-server-rest-modern.yaml
+...
+[INFO] GremlinServer$1 - Channel started at port 8182.
+----
+
+[source,text]
+$ curl -X POST -d "{\"gremlin\":\"g.V(x).out().values('name')\", \"language\":\"gremlin-groovy\", \"bindings\":{\"x\":1}}" "http://localhost:8182"
+
+[source,json]
+----
+{
+ "requestId": "abe3be05-1e86-481a-85e0-c59ad8a37c6b",
+ "status": {
+ "message": "",
+ "code": 200,
+ "attributes": {}
+ },
+ "result": {
+ "data": [
+ "lop",
+ "vadas",
+ "josh"
+ ],
+ "meta": {}
+ }
+}
+----
+
+IMPORTANT: Take careful note of the use of "bindings" in the arguments on the request. These are variables that are
+applied to the script on execution and is essentially a way to parameterize your scripts. This "parameterization" is
+critical to link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#_best_practices[performance]. Whenever
+possible, parameterize your queries.
+
+As mentioned earlier, Gremlin Server can also be configured with a websockets endpoint. This endpoint has an
+embedded link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#_developing_a_driver[subprotocol] that allow a
+compliant driver to communicate with it. TinkerPop supplies a
+link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#_connecting_via_java[reference driver] written in Java, but
+there are drivers developed by third-parties for other link:http://tinkerpop.incubator.apache.org/#graph-libraries[languages]
+such as Python, Javascript, etc. Gremlin Server therefore represents the method by which non-JVM languages can
+interact with TinkerPop.
+
+Conclusion
+^^^^^^^^^^
+
+...and that is the end of _The TinkerPop Workout - by Gremlin_. You are hopefully feeling more confident in your
+TinkerPop skills and have a good overview of what the stack has to offer, as well as some entry points to further
+research within the reference documentation. Welcome to The TinkerPop!
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/upgrade-release-3.0.x-incubating.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/upgrade-release-3.0.x-incubating.asciidoc b/docs/src/upgrade-release-3.0.x-incubating.asciidoc
deleted file mode 100644
index 4bb1473..0000000
--- a/docs/src/upgrade-release-3.0.x-incubating.asciidoc
+++ /dev/null
@@ -1,180 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-
-TinkerPop 3.0.0
-===============
-
-image::https://raw.githubusercontent.com/apache/incubator-tinkerpop/master/docs/static/images/gremlin-hindu.png[width=225]
-
-*A Gremlin Rāga in 7/16 Time*
-
-TinkerPop 3.0.2
----------------
-
-*Release Date: October 19, 2015*
-
-Please see the link:https://github.com/apache/incubator-tinkerpop/blob/3.0.2-incubating/CHANGELOG.asciidoc#tinkerpop-302-release-date-october-19-2015[changelog] for a complete list of all the modifications that are part of this release.
-
-Upgrading for Users
-~~~~~~~~~~~~~~~~~~~
-
-BulkLoaderVertexProgram (BLVP)
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-`BulkLoaderVertexProgram` now supports arbitrary inputs (i addition to `HadoopGraph`, which was already supported in
-version 3.0.1-incubating). It can now also read from any TP3 enabled graph, like `TinkerGraph`
-or `Neo4jGraph`.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-814[TINKERPOP3-319],
-link:http://tinkerpop.incubator.apache.org/docs/3.0.2-incubating/#bulkloadervertexprogram[Reference Documentation - BLVP]
-
-TinkerGraph
-^^^^^^^^^^^
-
-TinkerGraph can now be configured to support persistence, where TinkerGraph will try to load a graph from a specified
-location and calls to `close()` will save the graph data to that location.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-828[TINKERPOP3-828],
-link:http://tinkerpop.incubator.apache.org/docs/3.0.2-incubating/#_configuration[Reference Documentation - TinkerGraph]
-
-Gremlin Driver and Server
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-There were a number of fixes to `gremlin-driver` that prevent protocol desynchronization when talking to Gremlin
-Server.
-
-On the Gremlin Server side, Websocket sub-protocol introduces a new "close" operation to explicitly close sessions.
-Prior to this change, sessions were closed in a more passive fashion (i.e. session timeout). There were also so
-bug fixes around the protocol as it pertained to third-party drivers (e.g. python) using JSON for authentication.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-814[TINKERPOP3-814],
-link:https://issues.apache.org/jira/browse/TINKERPOP3-816[TINKERPOP3-816],
-link:https://issues.apache.org/jira/browse/TINKERPOP3-817[TINKERPOP3-817],
-link:https://issues.apache.org/jira/browse/TINKERPOP3-855[TINKERPOP3-855],
-link:https://issues.apache.org/jira/browse/TINKERPOP3-870[TINKERPOP3-870],
-link:https://issues.apache.org/jira/browse/TINKERPOP3-877[TINKERPOP3-877]
-
-Upgrading for Providers
-~~~~~~~~~~~~~~~~~~~~~~~
-
-Graph Driver Providers
-^^^^^^^^^^^^^^^^^^^^^^
-
-Gremlin Server close Operation
-++++++++++++++++++++++++++++++
-
-It is important to note that this feature of the sub-protocol applies to the `SessionOpProcessor` (i.e. for
-session-based requests). Prior to this change, there was no way to explicitly close a session. Sessions would get
-closed by the server after timeout of activity. This new "op" gives drivers the ability to close the session
-explicitly and as needed.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-849[TINKERPOP3-849],
-link:http://tinkerpop.incubator.apache.org/docs/3.0.2-incubating/#_opprocessors_arguments[Reference Documentation - OpProcessor]
-
-TinkerPop 3.0.1
----------------
-
-*Release Date: September 2, 2015*
-
-Please see the link:https://github.com/apache/incubator-tinkerpop/blob/3.0.1-incubating/CHANGELOG.asciidoc#tinkerpop-301-release-date-september-2-2015[changelog] for a complete list of all the modifications that are part of this release.
-
-Upgrading for Users
-~~~~~~~~~~~~~~~~~~~
-
-Gremlin Server
-^^^^^^^^^^^^^^
-
-Gremlin Server now supports a link:https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer[SASL-based]
-(Simple Authentication and Security Layer) authentication model and a default `SimpleAuthenticator` which implements
-the `PLAIN` SASL mechanism (i.e. plain text) to authenticate requests. This gives Gremlin Server some basic security
-capabilities, especially when combined with its built-in SSL feature.
-
-There have also been changes in how global variable bindings in Gremlin Server are established via initialization
-scripts. The initialization scripts now allow for a `Map` of values that can be returned from those scripts.
-That `Map` will be used to set global bindings for the server. See this
-link:https://github.com/apache/incubator-tinkerpop/blob/3.0.1-incubating/gremlin-server/scripts/generate-modern.groovy[sample script]
-for an example.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-576[TINKERPOP3-576]
-
-Neo4j
-^^^^^
-
-Problems related to using `:install` to get the Neo4j plugin operating in Gremlin Console on Windows have been
-resolved.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-804[TINKERPOP3-804]
-
-Upgrading for Providers
-~~~~~~~~~~~~~~~~~~~~~~~
-
-Graph System Providers
-^^^^^^^^^^^^^^^^^^^^^^
-
-GraphFactoryClass Annotation
-++++++++++++++++++++++++++++
-
-Providers can consider the use of the new `GraphFactoryClass` annotation to specify the factory class that `GraphFactory` will use to open a new `Graph` instance. This is an optional feature and will generally help implementations that have an interface extending `Graph`. If that is the case, then this annotation can be used in the following fashion:
-
-[source,java]
-----
-@GraphFactory(MyGraphFactory.class)
-public interface MyGraph extends Graph{
-}
-----
-
-`MyGraphFactory` must contain the static `open` method that is normally expected by `GraphFactory`.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-778[TINKERPOP3-778]
-
-GraphProvider.Descriptor Annotation
-+++++++++++++++++++++++++++++++++++
-
-There was a change that affected providers who implemented `GraphComputer` related tests such as the `ProcessComputerSuite`. If the provider runs those tests, then edit the `GraphProvider` implementation for those suites to include the `GraphProvider.Descriptor` annotation as follows:
-
-[source,java]
-----
-@GraphProvider.Descriptor(computer = GiraphGraphComputer.class)
-public final class HadoopGiraphGraphProvider extends HadoopGraphProvider {
-
- public GraphTraversalSource traversal(final Graph graph) {
- return GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(GiraphGraphComputer.class)).create(graph);
- }
-}
-----
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-690[TINKERPOP3-690] for more information.
-
-Semantics of Transaction.close()
-++++++++++++++++++++++++++++++++
-
-There were some adjustments to the test suite with respect to how `Transaction.close()` was being validated. For most providers, this will generally mean checking `OptOut` annotations for test renaming problems. The error that occurs when running the test suite should make it apparent that a test name is incorrect in an `OptOut` if there are issues there.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-764[TINKERPOP3-764] for more information.
-
-Graph Driver Providers
-^^^^^^^^^^^^^^^^^^^^^^
-
-Authentication
-++++++++++++++
-
-Gremlin Server now supports SASL-based authentication. By default, Gremlin Server is not configured with
-authentication turned on and authentication is not required, so existing drivers should still work without any
-additional change. Drivers should however consider implementing this feature as it is likely that many users will
-want the security capabilities that it provides.
-
-See: link:http://tinkerpop.incubator.apache.org/docs/3.0.1-incubating/#_authentication[Reference Documentation - Gremlin Server Authentication]
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/upgrade-release-3.1.x-incubating.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/upgrade-release-3.1.x-incubating.asciidoc b/docs/src/upgrade-release-3.1.x-incubating.asciidoc
deleted file mode 100644
index 32d23f0..0000000
--- a/docs/src/upgrade-release-3.1.x-incubating.asciidoc
+++ /dev/null
@@ -1,322 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-
-TinkerPop 3.1.0
-===============
-
-image::https://raw.githubusercontent.com/apache/incubator-tinkerpop/master/docs/static/images/gremlin-gangster.png[width=225]
-
-*A 187 On The Undercover Gremlinz*
-
-TinkerPop 3.1.0
----------------
-
-*Release Date: November 16, 2015*
-
-Please see the link:https://github.com/apache/incubator-tinkerpop/blob/3.1.0-incubating/CHANGELOG.asciidoc#tinkerpop-310-release-date-november-16-2015[changelog] for a complete list of all the modifications that are part of this release.
-
-Additional upgrade information can be found here:
-
-* <<_tinkerpop_3_0_2,TinkerPop 3.0.2>>
-* <<_tinkerpop_3_0_1,TinkerPop 3.0.1>>
-
-Upgrading for Users
-~~~~~~~~~~~~~~~~~~~
-
-Shading Jackson
-^^^^^^^^^^^^^^^
-
-The Jackson library is now shaded to `gremlin-shaded`, which will allow Jackson to version independently without
-breaking compatibility with dependent libraries or with those who depend on TinkerPop. The downside is that if a
-library depends on TinkerPop and uses the Jackson classes, those classes will no longer exist with the standard
-Jackson package naming. They will have to shifted as follows:
-
-* `org.objenesis` becomes `org.apache.tinkerpop.shaded.objenesis`
-* `com.esotericsoftware.minlog` becomes `org.apache.tinkerpop.shaded.minlog`
-* `com.fasterxml.jackson` becomes `org.apache.tinkerpop.shaded.jackson`
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-835[TINKERPOP3-835]
-
-PartitionStrategy and VertexProperty
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-`PartitionStrategy` now supports partitioning within `VertexProperty`. The `Graph` needs to be able to support
-meta-properties for this feature to work.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-333[TINKERPOP3-333]
-
-Gremlin Server and Epoll
-^^^^^^^^^^^^^^^^^^^^^^^^
-
-Gremlin Server provides a configuration option to turn on support for Netty
-link:http://netty.io/wiki/native-transports.html[native transport] on Linux, which has been shown to help improve
-performance.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-901[TINKERPOP3-901]
-
-Rebindings Deprecated
-^^^^^^^^^^^^^^^^^^^^^
-
-The notion of "rebindings" has been deprecated in favor of the term "aliases". Alias is a better and more intuitive
-term than rebindings which should make it easier for newcomers to understand what they are for.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-913[TINKERPOP3-913],
-link:http://tinkerpop.incubator.apache.org/docs/3.1.0-incubating/#_aliases[Reference Documentation - Aliases]
-
-Configurable Driver Channelizer
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The Gremlin Driver now allows the `Channerlizer` to be supplied as a configuration, which means that custom
-implementations may be supplied.
-
-See: https://issues.apache.org/jira/browse/TINKERPOP3-680[TINKERPOP3-680]
-
-GraphSON and Strict Option
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The `GraphMLReader` now has a `strict` option on the `Builder` so that if a data type for a value is invalid in some
-way, GraphMLReader will simply skip that problem value. In that way, it is a bit more forgiving than before especially
-with empty data.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-756[TINKERPOP3-756]
-
-Transaction.close() Default Behavior
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The default behavior of `Transaction.close()` is to rollback the transaction. This is in contrast to previous versions
-where the default behavior was commit. Using rollback as the default should be thought of as a like a safer approach
-to closing where a user must now explicitly call `commit()` to persist their mutations.
-
-See link:https://issues.apache.org/jira/browse/TINKERPOP3-805[TINKERPOP3-805] for more information.
-
-ThreadLocal Transaction Settings
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The `Transaction.onReadWrite()` and `Transaction.onClose()` settings now need to be set for each thread (if another
-behavior than the default is desired). For gremlin-server users that may be changing these settings via scripts.
-If the settings are changed for a sessionless request they will now only apply to that one request. If the settings are
-changed for an in-session request they will now only apply to all future requests made in the scope of that session.
-
-See link:https://issues.apache.org/jira/browse/TINKERPOP3-885[TINKERPOP3-885]
-
-Hadoop-Gremlin
-^^^^^^^^^^^^^^
-
-* Hadoop1 is no longer supported. Hadoop2 is now the only supported Hadoop version in TinkerPop.
-* Spark and Giraph have been split out of Hadoop-Gremlin into their own respective packages (Spark-Gremlin and Giraph-Gremlin).
-* The directory where application jars are stored in HDFS is now `hadoop-gremlin-x.y.z-libs`.
-** This versioning is important so that cross-version TinkerPop use does not cause jar conflicts.
-
-See link:https://issues.apache.org/jira/browse/TINKERPOP3-616
-
-Spark-Gremlin
-^^^^^^^^^^^^^
-
-* Providers that wish to reuse a graphRDD can leverage the new `PersistedInputRDD` and `PersistedOutputRDD`.
-** This allows the graphRDD to avoid serialization into HDFS for reuse. Be sure to enabled persisted `SparkContext` (see documentation).
-
-See link:https://issues.apache.org/jira/browse/TINKERPOP3-868,
-link:https://issues.apache.org/jira/browse/TINKERPOP3-925
-
-TinkerGraph Serialization
-^^^^^^^^^^^^^^^^^^^^^^^^^
-
-TinkerGraph is serializable over Gryo, which means that it can shipped over the wire from Gremlin Server. This
-feature can be useful when working with remote subgraphs.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-728[TINKERPOP3-728]
-
-Deprecation in TinkerGraph
-^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The `public static String` configurations have been renamed. The old `public static` variables have been deprecated.
-If the deprecated variables were being used, then convert to the replacements as soon as possible.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-926[TINKERPOP3-926]
-
-Deprecation in Gremlin-Groovy
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The closure wrappers classes `GFunction`, `GSupplier`, `GConsumer` have been deprecated. In Groovy, a closure can be
-specified using `as Function` and thus, these wrappers are not needed. Also, the `GremlinExecutor.promoteBindings()`
-method which was previously deprecated has been removed.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-879[TINKERPOP3-879],
-link:https://issues.apache.org/jira/browse/TINKERPOP3-897[TINKERPOP3-897]
-
-Gephi Traversal Visualization
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The process for visualizing a traversal has been simplified. There is no longer a need to "name" steps that will
-represent visualization points for Gephi. It is possible to just "configure" a `visualTraversal` in the console:
-
-[source,text]
-gremlin> :remote config visualTraversal graph vg
-
-which creates a special `TraversalSource` from `graph` called `vg`. The traversals created from `vg` can be used
-to `:submit` to Gephi.
-
-See: link:http://tinkerpop.incubator.apache.org/docs/3.1.0-SNAPSHOT/#gephi-plugin[Reference Documentation - Gephi]
-
-Alterations to GraphTraversal
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-There were a number of changes to `GraphTraversal`. Many of the changes came by way of deprecation, but some semantics
-have changed as well:
-
-* `ConjunctionStrategy` has been renamed to `ConnectiveStrategy` (no other behaviors changed).
-* `ConjunctionP` has been renamed to `ConnectiveP` (no other behaviors changed).
-* `DedupBijectionStrategy` has been renamed (and made more effective) as `FilterRankingStrategy`.
-* The `GraphTraversal` mutation API has change significantly with all previous methods being supported but deprecated.
-** The general pattern used now is `addE('knows').from(select('a')).to(select('b')).property('weight',1.0)`.
-* The `GraphTraversal` sack API has changed with all previous methods being supported but deprecated.
-** The old `sack(mult,'weight')` is now `sack(mult).by('weight')`.
-* `GroupStep` has been redesigned such that there is now only a key- and value-traversal. No more reduce-traversal.
-** The previous `group()`-methods have been renamed to `groupV3d0()`. To immediately upgrade, rename all your `group()`-calls to `groupV3d0()`.
-** To migrate to the new `group()`-methods, what was `group().by('age').by(outE()).by(sum(local))` is now `group().by('age').by(outE().sum())`.
-* There was a bug in `fold()`, where if a bulked traverser was provided, the traverser was only represented once.
-** This bug fix might cause a breaking change to a user query if the non-bulk behavior was being counted on. If so, used `dedup()` prior to `fold()`.
-* Both `GraphTraversal().mapKeys()` and `GraphTraversal.mapValues()` has been deprecated.
-** Use `select(keys)` and `select(columns)`. However, note that `select()` will not unroll the keys/values. Thus, `mapKeys()` => `select(keys).unfold()`.
-* The data type of `Operator` enums will now always be the highest common data type of the two given numbers, rather than the data type of the first number, as it's been before.
-
-Aliasing Remotes in the Console
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-The `:remote` command in Gremlin Console has a new `alias` configuration option. This `alias` option allows
-specification of a set of key/value alias/binding pairs to apply to the remote. In this way, it becomes possible
-to refer to a variable on the server as something other than what it is referred to for purpose of the submitted
-script. For example once a `:remote` is created, this command:
-
-[source,text]
-:remote alias x g
-
-would allow "g" on the server to be referred to as "x".
-
-[source,text]
-:> x.E().label().groupCount()
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-914[TINKERPOP3-914]
-
-Upgrading for Providers
-~~~~~~~~~~~~~~~~~~~~~~~
-
-IMPORTANT: It is recommended that providers also review all the upgrade instructions specified for users. Many of the
-changes there may prove important for the provider's implementation.
-
-All providers should be aware that Jackson is now shaded to `gremlin-shaded` and could represent breaking change if
-there was usage of the dependency by way of TinkerPop, a direct dependency to Jackson may be required on the
-provider's side.
-
-Graph System Providers
-^^^^^^^^^^^^^^^^^^^^^^
-
-GraphStep Alterations
-+++++++++++++++++++++
-
-* `GraphStep` is no longer in `sideEffect`-package, but now in `map`-package as traversals support mid-traversal `V()`.
-* Traversals now support mid-traversal `V()`-steps. Graph system providers should ensure that a mid-traversal `V()` can leverage any suitable index.
-
-See link:https://issues.apache.org/jira/browse/TINKERPOP3-762
-
-Decomposition of AbstractTransaction
-++++++++++++++++++++++++++++++++++++
-
-The `AbstractTransaction` class has been abstracted into two different classes supporting two different modes of
-operation: `AbstractThreadLocalTransaction` and `AbstractThreadedTransaction`, where the former should be used when
-supporting `ThreadLocal` transactions and the latter for threaded transactions. Of course, providers may still
-choose to build their own implementation on `AbstractTransaction` itself or simply implement the `Transaction`
-interface.
-
-The `AbstractTransaction` gains the following methods to potentially implement (though default implementations
-are supplied in `AbstractThreadLocalTransaction` and `AbstractThreadedTransaction`):
-
-* `doReadWrite` that should execute the read-write consumer.
-* `doClose` that should execute the close consumer.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-765[TINKERPOP3-765],
-link:https://issues.apache.org/jira/browse/TINKERPOP3-885[TINKERPOP3-885]
-
-Transaction.close() Default Behavior
-++++++++++++++++++++++++++++++++++++
-
-The default behavior for `Transaction.close()` is to rollback the transaction and is enforced by tests, which
-previously asserted the opposite (i.e. commit on close). These tests have been renamed to suite the new semantics:
-
-* `shouldCommitOnCloseByDefault` became `shouldCommitOnCloseWhenConfigured`
-* `shouldRollbackOnCloseWhenConfigured` became `shouldRollbackOnCloseByDefault`
-
-If these tests were referenced in an `OptOut`, then their names should be updated.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-805[TINKERPOP3-805]
-
-Graph Traversal Updates
-+++++++++++++++++++++++
-
-There were numerous changes to the `GraphTraversal` API. Nearly all changes are backwards compatible with respective
-"deprecated" annotations. Please review the respective updates specified in the "Graph System Users" section.
-
-* `GraphStep` is no longer in `sideEffect` package. Now in `map` package.
-* Make sure mid-traversal `GraphStep` calls are folding `HasContainers` in for index-lookups.
-* Think about copying `TinkerGraphStepStrategyTest` for your implementation so you know folding is happening correctly.
-
-Element Removal
-+++++++++++++++
-
-`Element.Exceptions.elementAlreadyRemoved` has been deprecated and test enforcement for consistency have been removed.
- Providers are free to deal with deleted elements as they see fit.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-297[TINKERPOP3-297]
-
-VendorOptimizationStrategy Rename
-+++++++++++++++++++++++++++++++++
-
-The `VendorOptimizationStrategy` has been renamed to `ProviderOptimizationStrategy`. This renaming is consistent
-with revised terminology for what were formerly referred to as "vendors".
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-876[TINKERPOP3-876]
-
-GraphComputer Updates
-+++++++++++++++++++++
-
-`GraphComputer.configure(String key, Object value)` is now a method (with default implementation).
-This allows the user to specify engine-specific parameters to the underlying OLAP system. These parameters are not intended
-to be cross engine supported. Moreover, if there are not parameters that can be altered (beyond the standard `GraphComputer`
-methods), then the provider's `GraphComputer` implementation should simply return and do nothing.
-
-Driver Providers
-^^^^^^^^^^^^^^^^
-
-Aliases Parameter
-+++++++++++++++++
-
-The "rebindings" argument to the "standard" `OpProcessor` has been renamed to "aliases". While "rebindings" is still
-supported it is recommended that the upgrade to "aliases" be made as soon as possible as support will be removed in
-the future. Gremlin Server will not accept both parameters at the same time - a request must contain either one
-parameter or the other if either is supplied.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-913[TINKERPOP3-913]
-
-ThreadLocal Transaction Settings
-++++++++++++++++++++++++++++++++
-
-If a driver configures the `Transaction.onReadWrite()` or `Transaction.onClose()` settings, note that these settings no
-longer apply to all future requests. If the settings are changed for a sessionless request they will only apply to
-that one request. If the settings are changed from an in-session request they will only apply to all future requests
-made in the scope of that session.
-
-See: link:https://issues.apache.org/jira/browse/TINKERPOP3-885[TINKERPOP3-885]
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/upgrade.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/upgrade.asciidoc b/docs/src/upgrade.asciidoc
deleted file mode 100644
index 86dd6b6..0000000
--- a/docs/src/upgrade.asciidoc
+++ /dev/null
@@ -1,39 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-image::apache-tinkerpop-logo.png[width=500]
-
-:toc-position: left
-
-TinkerPop Upgrade Information
-=============================
-
-This document helps users of TinkerPop to understand the changes that come with each software release. It outlines
-new features, how to resolve breaking changes and other information specific to a release. This document is useful
-to end-users who are building applications on TinkerPop, but it is equally useful to TinkerPop providers, who
-build libraries and other systems on the the core APIs and protocols that TinkerPop exposes.
-
-These providers include:
-
-* Graph System Provider
-** Graph Database Provider
-** Graph Processor Provider
-* Graph Driver Provider
-* Graph Language Provider
-
-include::upgrade-release-3.1.x-incubating.asciidoc[]
-
-include::upgrade-release-3.0.x-incubating.asciidoc[]
\ No newline at end of file
[07/22] incubator-tinkerpop git commit: Made subdirectories for
various "books" in the docs.
Posted by sp...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/reference/the-graphcomputer.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/reference/the-graphcomputer.asciidoc b/docs/src/reference/the-graphcomputer.asciidoc
new file mode 100644
index 0000000..af65f07
--- /dev/null
+++ b/docs/src/reference/the-graphcomputer.asciidoc
@@ -0,0 +1,475 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+[[graphcomputer]]
+The GraphComputer
+=================
+
+image:graphcomputer-puffers.png[width=350,float=right] TinkerPop3 provides two primary means of interacting with a
+graph: link:http://en.wikipedia.org/wiki/Online_transaction_processing[online transaction processing] (OLTP) and
+link:http://en.wikipedia.org/wiki/Online_analytical_processing[online analytical processing] (OLAP). OTLP-based
+graph systems allow the user to query the graph in real-time. However, typically, real-time performance is only
+possible when a local traversal is enacted. A local traversal is one that starts at a particular vertex (or small set
+of vertices) and touches a small set of connected vertices (by any arbitrary path of arbitrary length). In short, OLTP
+queries interact with a limited set of data and respond on the order of milliseconds or seconds. On the other hand,
+with OLAP graph processing, the entire graph is processed and thus, every vertex and edge is analyzed (some times
+more than once for iterative, recursive algorithms). Due to the amount of data being processed, the results are
+typically not returned in real-time and for massive graphs (i.e. graphs represented across a cluster of machines),
+results can take on the order of minutes or hours.
+
+ * *OLTP*: real-time, limited data accessed, random data access, sequential processing, querying
+ * *OLAP*: long running, entire data set accessed, sequential data access, parallel processing, batch processing
+
+image::oltp-vs-olap.png[width=600]
+
+The image above demonstrates the difference between Gremlin OLTP and Gremlin OLAP. With Gremlin OLTP, the graph is
+walked by moving from vertex-to-vertex via incident edges. With Gremlin OLAP, all vertices are provided a
+`VertexProgram`. The programs send messages to one another with the topological structure of the graph acting as the
+communication network (though random message passing possible). In many respects, the messages passed are like
+the OLTP traversers moving from vertex-to-vertex. However, all messages are moving independent of one another, in
+parallel. Once a vertex program is finished computing, TinkerPop3's OLAP engine supports any number
+link:http://en.wikipedia.org/wiki/MapReduce[`MapReduce`] jobs over the resultant graph.
+
+IMPORTANT: `GraphComputer` was designed from the start to be used within a multi-JVM, distributed environment --
+in other words, a multi-machine compute cluster. As such, all the computing objects must be able to be migrated
+between JVMs. The pattern promoted is to store state information in a `Configuration` object to later be regenerated
+by a loading process. It is important to realize that `VertexProgram`, `MapReduce`, and numerous particular instances
+rely heavily on the state of the computing classes (not the structure, but the processes) to be stored in a
+`Configuration`.
+
+[[vertexprogram]]
+VertexProgram
+-------------
+
+image:bsp-diagram.png[width=400,float=right] GraphComputer takes a `VertexProgram`. A VertexProgram can be thought of
+as a piece of code that is executed at each vertex in logically parallel manner until some termination condition is
+met (e.g. a number of iterations have occurred, no more data is changing in the graph, etc.). A submitted
+`VertexProgram` is copied to all the workers in the graph. A worker is not an explicit concept in the API, but is
+assumed of all `GraphComputer` implementations. At minimum each vertex is a worker (though this would be inefficient
+due to the fact that each vertex would maintain a VertexProgram). In practice, the workers partition the vertex set
+and and are responsible for the execution of the VertexProgram over all the vertices within their sphere of influence.
+The workers orchestrate the execution of the `VertexProgram.execute()` method on all their vertices in an
+link:http://en.wikipedia.org/wiki/Bulk_synchronous_parallel[bulk synchronous parallel] (BSP) fashion. The vertices
+are able to communicate with one another via messages. There are two kinds of messages in Gremlin OLAP:
+`MessageScope.Local` and `MessageScope.Global`. A local message is a message to an adjacent vertex. A global
+message is a message to any arbitrary vertex in the graph. Once the VertexProgram has completed its execution,
+any number of `MapReduce` jobs are evaluated. MapReduce jobs are provided by the user via `GraphComputer.mapReduce()`
+ or by the `VertexProgram` via `VertexProgram.getMapReducers()`.
+
+image::graphcomputer.png[width=500]
+
+The example below demonstrates how to submit a VertexProgram to a graph's GraphComputer. `GraphComputer.submit()`
+yields a `Future<ComputerResult>`. The `ComputerResult` has the resultant computed graph which can be a full copy
+of the original graph (see <<hadoop-gremlin,Hadoop-Gremlin>>) or a view over the original graph (see
+<<tinkergraph,TinkerGraph>>). The ComputerResult also provides access to computational side-effects called `Memory`
+(which includes, for example, runtime, number of iterations, results of MapReduce jobs, and VertexProgram-specific
+memory manipulations).
+
+[gremlin-groovy,modern]
+----
+result = graph.compute().program(PageRankVertexProgram.build().create()).submit().get()
+result.memory().runtime
+g = result.graph().traversal(standard())
+g.V().valueMap('name',PageRankVertexProgram.PAGE_RANK)
+----
+
+NOTE: This model of "vertex-centric graph computing" was made popular by Google's
+link:http://googleresearch.blogspot.com/2009/06/large-scale-graph-computing-at-google.html[Pregel] graph engine.
+In the open source world, this model is found in OLAP graph computing systems such as link:https://giraph.apache.org/[Giraph],
+link:https://hama.apache.org/[Hama], and link:http://faunus.thinkaurelius.com[Faunus]. TinkerPop3 extends the
+popularized model with integrated post-processing <<mapreduce,MapReduce>> jobs over the vertex set.
+
+[[mapreduce]]
+MapReduce
+---------
+
+The BSP model proposed by Pregel stores the results of the computation in a distributed manner as properties on the
+elements in the graph. In many situations, it is necessary to aggregate those resultant properties into a single
+result set (i.e. a statistic). For instance, assume a VertexProgram that computes a nominal cluster for each vertex
+(i.e. link:http://en.wikipedia.org/wiki/Community_structure[a graph clustering algorithm]). At the end of the
+computation, each vertex will have a property denoting the cluster it was assigned to. TinkerPop3 provides the
+ability to answer global questions about the clusters. For instance, in order to answer the following questions,
+`MapReduce` jobs are required:
+
+ * How many vertices are in each cluster? (*presented below*)
+ * How many unique clusters are there? (*presented below*)
+ * What is the average age of each vertex in each cluster?
+ * What is the degree distribution of the vertices in each cluster?
+
+A compressed representation of the `MapReduce` API in TinkerPop3 is provided below. The key idea is that the
+`map`-stage processes all vertices to emit key/value pairs. Those values are aggregated on their respective key
+for the `reduce`-stage to do its processing to ultimately yield more key/value pairs.
+
+[source,java]
+public interface MapReduce<MK, MV, RK, RV, R> {
+ public void map(final Vertex vertex, final MapEmitter<MK, MV> emitter);
+ public void reduce(final MK key, final Iterator<MV> values, final ReduceEmitter<RK, RV> emitter);
+ // there are more methods
+}
+
+IMPORTANT: The vertex that is passed into the `MapReduce.map()` method does not contain edges. The vertex only
+contains original and computed vertex properties. This reduces the amount of data required to be loaded and ensures
+that MapReduce is used for post-processing computed results. All edge-based computing should be accomplished in the
+`VertexProgram`.
+
+image::mapreduce.png[width=650]
+
+The `MapReduce` extension to GraphComputer is made explicit when examining the
+<<peerpressurevertexprogram,`PeerPressureVertexProgram`>> and corresponding `ClusterPopulationMapReduce`.
+In the code below, the GraphComputer result returns the computed on `Graph` as well as the `Memory` of the
+computation (`ComputerResult`). The memory maintain the results of any MapReduce jobs. The cluster population
+MapReduce result states that there are 5 vertices in cluster 1 and 1 vertex in cluster 6. This can be verified
+(in a serial manner) by looking at the `PeerPressureVertexProgram.CLUSTER` property of the resultant graph. Notice
+that the property is "hidden" unless it is directly accessed via name.
+
+[gremlin-groovy,modern]
+----
+graph = TinkerFactory.createModern()
+result = graph.compute().program(PeerPressureVertexProgram.build().create()).mapReduce(ClusterPopulationMapReduce.build().create()).submit().get()
+result.memory().get('clusterPopulation')
+g = result.graph().traversal(standard())
+g.V().values(PeerPressureVertexProgram.CLUSTER).groupCount().next()
+g.V().valueMap()
+----
+
+If there are numerous statistics desired, then its possible to register as many MapReduce jobs as needed. For
+instance, the `ClusterCountMapReduce` determines how many unique clusters were created by the peer pressure algorithm.
+Below both `ClusterCountMapReduce` and `ClusterPopulationMapReduce` are computed over the resultant graph.
+
+[gremlin-groovy,modern]
+----
+result = graph.compute().program(PeerPressureVertexProgram.build().create()).
+ mapReduce(ClusterPopulationMapReduce.build().create()).
+ mapReduce(ClusterCountMapReduce.build().create()).submit().get()
+result.memory().clusterPopulation
+result.memory().clusterCount
+----
+
+IMPORTANT: The MapReduce model of TinkerPop3 does not support MapReduce chaining. Thus, the order in which the
+MapReduce jobs are executed is irrelevant. This is made apparent when realizing that the `map()`-stage takes a
+`Vertex` as its input and the `reduce()`-stage yields key/value pairs. Thus, the results of reduce can not feed back
+into map.
+
+A Collection of VertexPrograms
+------------------------------
+
+TinkerPop3 provides a collection of VertexPrograms that implement common algorithms. This section discusses the various implementations.
+
+IMPORTANT: The vertex programs presented are what are provided as of TinkerPop x.y.z. Over time, with future releases,
+more algorithms will be added.
+
+[[pagerankvertexprogram]]
+PageRankVertexProgram
+~~~~~~~~~~~~~~~~~~~~~
+
+image:gremlin-pagerank.png[width=400,float=right] link:http://en.wikipedia.org/wiki/PageRank[PageRank] is perhaps the
+most popular OLAP-oriented graph algorithm. This link:http://en.wikipedia.org/wiki/Centrality[eigenvector centrality]
+variant was developed by Brin and Page of Google. PageRank defines a centrality value for all vertices in the graph,
+where centrality is defined recursively where a vertex is central if it is connected to central vertices. PageRank is
+an iterative algorithm that converges to a link:http://en.wikipedia.org/wiki/Ergodicity[steady state distribution]. If
+the pageRank values are normalized to 1.0, then the pageRank value of a vertex is the probability that a random walker
+will be seen that that vertex in the graph at any arbitrary moment in time. In order to help developers understand the
+methods of a `VertexProgram`, the PageRankVertexProgram code is analyzed below.
+
+[source,java]
+----
+public class PageRankVertexProgram implements VertexProgram<Double> { <1>
+
+ private MessageScope.Local<Double> incidentMessageScope = MessageScope.Local.of(__::outE); <2>
+ private MessageScope.Local<Double> countMessageScope = MessageScope.Local.of(new MessageScope.Local.ReverseTraversalSupplier(this.incidentMessageScope));
+
+ public static final String PAGE_RANK = "gremlin.pageRankVertexProgram.pageRank"; <3>
+ public static final String EDGE_COUNT = "gremlin.pageRankVertexProgram.edgeCount";
+
+ private static final String VERTEX_COUNT = "gremlin.pageRankVertexProgram.vertexCount";
+ private static final String ALPHA = "gremlin.pageRankVertexProgram.alpha";
+ private static final String TOTAL_ITERATIONS = "gremlin.pageRankVertexProgram.totalIterations";
+ private static final String INCIDENT_TRAVERSAL_SUPPLIER = "gremlin.pageRankVertexProgram.incidentTraversalSupplier";
+
+ private ConfigurationTraversal<Vertex, Edge> configurationTraversal;
+ private double vertexCountAsDouble = 1.0d;
+ private double alpha = 0.85d;
+ private int totalIterations = 30;
+
+ private static final Set<String> COMPUTE_KEYS = new HashSet<>(Arrays.asList(PAGE_RANK, EDGE_COUNT));
+
+ private PageRankVertexProgram() {}
+
+ @Override
+ public void loadState(final Graph graph, final Configuration configuration) { <4>
+ if (configuration.containsKey(TRAVERSAL_SUPPLIER)) {
+ this.configurationTraversal = ConfigurationTraversal.loadState(graph, configuration, TRAVERSAL_SUPPLIER);
+ this.incidentMessageScope = MessageScope.Local.of(this.configurationTraversal);
+ this.countMessageScope = MessageScope.Local.of(new MessageScope.Local.ReverseTraversalSupplier(this.incidentMessageScope));
+ }
+ this.vertexCountAsDouble = configuration.getDouble(VERTEX_COUNT, 1.0d);
+ this.alpha = configuration.getDouble(ALPHA, 0.85d);
+ this.totalIterations = configuration.getInt(TOTAL_ITERATIONS, 30);
+ }
+
+ @Override
+ public void storeState(final Configuration configuration) {
+ configuration.setProperty(VERTEX_PROGRAM, PageRankVertexProgram.class.getName());
+ configuration.setProperty(VERTEX_COUNT, this.vertexCountAsDouble);
+ configuration.setProperty(ALPHA, this.alpha);
+ configuration.setProperty(TOTAL_ITERATIONS, this.totalIterations);
+ if (null != this.traversalSupplier) {
+ this.traversalSupplier.storeState(configuration);
+ }
+ }
+
+ @Override
+ public Set<String> getElementComputeKeys() { <5>
+ return COMPUTE_KEYS;
+ }
+
+ @Override
+ public Optional<MessageCombiner<Double>> getMessageCombiner() {
+ return (Optional) PageRankMessageCombiner.instance();
+ }
+
+ @Override
+ public Set<MessageScope> getMessageScopes(final int iteration) {
+ final Set<MessageScope> set = new HashSet<>();
+ set.add(0 == iteration ? this.countMessageScope : this.incidentMessageScope);
+ return set;
+ }
+
+ @Override
+ public void setup(final Memory memory) {
+
+ }
+
+ @Override
+ public void execute(final Vertex vertex, Messenger<Double> messenger, final Memory memory) { <6>
+ if (memory.isInitialIteration()) { <7>
+ messenger.sendMessage(this.countMessageScope, 1.0d);
+ } else if (1 == memory.getIteration()) { <8>
+ double initialPageRank = 1.0d / this.vertexCountAsDouble;
+ double edgeCount = IteratorUtils.reduce(messenger.receiveMessages(), 0.0d, (a, b) -> a + b);
+ vertex.property(PAGE_RANK, initialPageRank);
+ vertex.property(EDGE_COUNT, edgeCount);
+ messenger.sendMessage(this.incidentMessageScope, initialPageRank / edgeCount);
+ } else { <9>
+ double newPageRank = IteratorUtils.reduce(messenger.receiveMessages(), 0.0d, (a, b) -> a + b);
+ newPageRank = (this.alpha * newPageRank) + ((1.0d - this.alpha) / this.vertexCountAsDouble);
+ vertex.property(PAGE_RANK, newPageRank);
+ messenger.sendMessage(this.incidentMessageScope, newPageRank / vertex.<Double>value(EDGE_COUNT));
+ }
+ }
+
+ @Override
+ public boolean terminate(final Memory memory) { <10>
+ return memory.getIteration() >= this.totalIterations;
+ }
+
+ @Override
+ public String toString() {
+ return StringFactory.vertexProgramString(this, "alpha=" + this.alpha + ",iterations=" + this.totalIterations);
+ }
+}
+----
+
+<1> `PageRankVertexProgram` implements `VertexProgram<Double>` because the messages it sends are Java doubles.
+<2> The default path of energy propagation is via outgoing edges from the current vertex.
+<3> The resulting PageRank values for the vertices are stored as a hidden property.
+<4> A vertex program is constructed using an Apache `Configuration` to ensure easy dissemination across a cluster of JVMs.
+<5> A vertex program must define the "compute keys" that are the properties being operated on during the computation.
+<6> The "while"-loop of the vertex program.
+<7> In order to determine how to distribute the energy to neighbors, a "1"-count is used to determine how many incident vertices exist for the `MessageScope`.
+<8> Initially, each vertex is provided an equal amount of energy represented as a double.
+<9> Energy is aggregated, computed on according to the PageRank algorithm, and then disseminated according to the defined `MessageScope.Local`.
+<10> The computation is terminated after a pre-defined number of iterations.
+
+The above `PageRankVertexProgram` is used as follows.
+
+[gremlin-groovy,modern]
+----
+result = graph.compute().program(PageRankVertexProgram.build().create()).submit().get()
+result.memory().runtime
+g = result.graph().traversal(standard())
+g.V().valueMap('name',PageRankVertexProgram.PAGE_RANK)
+----
+
+
+[[peerpressurevertexprogram]]
+PeerPressureVertexProgram
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The `PeerPressureVertexProgram` is a clustering algorithm that assigns a nominal value to each vertex in the graph.
+The nominal value represents the vertex's cluster. If two vertices have the same nominal value, then they are in the
+same cluster. The algorithm proceeds in the following manner.
+
+ . Every vertex assigns itself to a unique cluster ID (initially, its vertex ID).
+ . Every vertex determines its per neighbor vote strength as 1.0d / incident edges count.
+ . Every vertex sends its cluster ID and vote strength to its adjacent vertices as a `Pair<Serializable,Double>`
+ . Every vertex generates a vote energy distribution of received cluster IDs and changes its current cluster ID to the most frequent cluster ID.
+ .. If there is a tie, then the cluster with the lowest `toString()` comparison is selected.
+ . Steps 3 and 4 repeat until either a max number of iterations has occurred or no vertex has adjusted its cluster anymore.
+
+[[bulkloadervertexprogram]]
+BulkLoaderVertexProgram
+~~~~~~~~~~~~~~~~~~~~~~~
+
+The `BulkLoaderVertexProgram` provides a generalized way for loading graphs of any size (preferably large sized graphs)
+into a persistent `Graph`. The input can be any existing `Graph` database supporting TinkerPop3 or any of the Hadoop
+GraphInputFormats (e.g. `GraphSONInputFormat`, `GryoInputFormat` or `ScriptInputFormat`). The following example
+demonstrates how to load data from one TinkerGraph to another:
+
+[gremlin-groovy]
+----
+writeGraphConf = new BaseConfiguration()
+writeGraphConf.setProperty("gremlin.graph", "org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph")
+writeGraphConf.setProperty("gremlin.tinkergraph.graphFormat", "gryo")
+writeGraphConf.setProperty("gremlin.tinkergraph.graphLocation", "/tmp/tinkergraph.kryo")
+modern = TinkerFactory.createModern()
+blvp = BulkLoaderVertexProgram.build().
+ keepOriginalIds(false).
+ writeGraph(writeGraphConf).create(modern)
+modern.compute().workers(1).program(blvp).submit().get()
+graph = GraphFactory.open(writeGraphConf)
+g = graph.traversal()
+g.V().valueMap()
+graph.close()
+----
+
+.Available configuration options
+[width="800px",options="header"]
+|========================================
+|Builder Method |Purpose | Default Value
+|`bulkLoader(Class\|String)` | Sets the class of the bulk loader implementation. | `IncrementalBulkLoader`
+|`vertexIdProperty(String)` | Sets the name of the property in the target graph that holds the vertex id from the
+source graph. | `bulkLoader.vertex.id`
+|`keepOriginalIds(boolean)` |Whether to keep the id's from the source graph in the target graph or not. It's
+recommended to keep them if it's planned to do further bulk loads using the same datasources. | `true`
+|`userSuppliedIds(boolean)` |Whether to use the id's from the source graph as id's in the target graph. If set to
+`true`, `vertexIdProperty` will be ignored. Note, that the target graph must support user supplied identifiers. | `false`
+|`intermediateBatchSize(int)` |Sets the batch size for intermediate transactions. This is per thread in a
+multi-threaded environment. +0+ means that transactions will only be committed at the end of an iteration cycle.
+It's recommended to tune this property for the target graph and not use the default value of +0+. | `0`
+|`writeGraph(String)` | Sets the path to a `GraphFactory` compatible configuration file for the target graph. | _none_
+|========================================
+
+NOTE: `BulkLoaderVertexProgram` comes with a default `BulkLoader` implementation, namely `IncrementalBulkLoader`. It
+will work for the most use-cases, but has one limitation though: It doesn't support multi-valued properties.
+`IncrementalBulkLoader` will handle every property as a single-valued property. A custom `BulkLoader` implementation
+has to be used if the default behavior is not sufficient.
+
+NOTE: A custom `BulkLoader` implementation for incremental loading should use `GraphTraversal` methods to create/update
+elements (e.g. `g.addV()` instead of `graph.addVertex()`). This way the `BulkLoaderVertexProgram` is able to efficiently
+track changes in the underlying graph and can apply several optimization techniques.
+
+[[traversalvertexprogram]]
+TraversalVertexProgram
+~~~~~~~~~~~~~~~~~~~~~~
+
+image:traversal-vertex-program.png[width=250,float=left] The `TraversalVertexProgram` is a "special" VertexProgram in
+that it can be executed via a `GraphTraversal` with a `ComputerTraversalEngine`. In Gremlin, it is possible to have
+the same traversal executed using either the standard OTLP-engine or the `GraphComputer` OLAP-engine. The difference
+being where the traversal is submitted.
+
+NOTE: This model of graph traversal in a BSP system was first implemented by the
+link:http://faunus.thinkaurelius.com[Faunus] graph analytics engine and originally described in
+link:http://markorodriguez.com/2011/04/19/local-and-distributed-traversal-engines/[Local and Distributed Traversal Engines].
+
+[gremlin-groovy,modern]
+----
+g = graph.traversal(standard())
+g.V().both().hasLabel('person').values('age').groupCount().next() // OLTP
+g = graph.traversal(computer())
+g.V().both().hasLabel('person').values('age').groupCount().next() // OLAP
+----
+
+image::olap-traversal.png[width=650]
+
+In the OLAP example above, a `TraversalVertexProgram` is (logically) sent to each vertex in the graph. Each instance
+evaluation requires (logically) 5 BSP iterations and each iteration is interpreted as such:
+
+ . `g.V()`: Put a traverser on each vertex in the graph.
+ . `both()`: Propagate each traverser to the vertices `both`-adjacent to its current vertex.
+ . `hasLabel('person')`: If the vertex is not a person, kill the traversers at that vertex.
+ . `values('age')`: Have all the traversers reference the integer age of their current vertex.
+ . `groupCount()`: Count how many times a particular age has been seen.
+
+While 5 iterations were presented, in fact, `TraversalVertexProgram` will execute the traversal in only
+3 iterations. The reason being is that `hasLabel('person').values('age').groupCount()` can all be executed in a
+single iteration as any message sent would simply be to the current executing vertex. Thus, a simple optimization
+exists in Gremlin OLAP called "reflexive message passing" which simulates non-message-passing BSP iterations within a
+single BSP iteration.
+
+When the computation is complete a <<mapreduce,MapReduce>> job executes which aggregates all the `groupCount()`
+sideEffect Map (i.e. "`HashMap`") objects on each vertex into a single local representation (thus, turning the
+distributed Map representation into a local Map representation).
+
+////
+The same OLAP traversal can be executed using the standard `g.compute()` model, though at the expense of verbosity.
+`TraversalVertexProgram` provides a fluent `Builder` for constructing a `TraversalVertexProgram`. The specified
+`traversal()` can be either a `Supplier<Traversal>` object, a `Supplier<Traversal>` class, or a
+link:http://en.wikipedia.org/wiki/Scripting_for_the_Java_Platform[JSR-223] script that will generate (i.e. supply) a
+`Traversal`. If `traversal()` is supplied a single string, it is assumed that "gremlin-groovy" is the `ScriptEngine`
+to use. If two strings are supplied, then the first string denotes the `ScriptEngine` to evaluate the second string
+script with in order to generate (i.e. supply) the `Traversal`.
+
+[gremlin-groovy,modern]
+----
+//g.engine(computer())
+//result = g.compute().program(TraversalVertexProgram.build().traversal(g.V().both().hasLabel('person').values('age').groupCount('a')).create()).submit().get()
+//result.memory().a
+//result.memory().iteration
+//result.memory().runtime
+----
+////
+
+[[distributed-gremlin-gotchas]]
+Distributed Gremlin Gotchas
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Gremlin OLTP is not identical to Gremlin OLAP.
+
+IMPORTANT: There are two primary theoretical differences between Gremlin OLTP and Gremlin OLAP. First, Gremlin OLTP
+(via `Traversal`) leverages a link:http://en.wikipedia.org/wiki/Depth-first_search[depth-first] execution engine.
+Depth-first execution has a limited memory footprint due to link:http://en.wikipedia.org/wiki/Lazy_evaluation[lazy evaluation].
+On the other hand, Gremlin OLAP (via `TraversalVertexProgram`) leverages a
+link:http://en.wikipedia.org/wiki/Breadth-first_search[breadth-first] execution engine which maintains a larger memory
+footprint, but a better time complexity due to vertex-local traversers being able to be merged. The second difference
+is that Gremlin OLTP is executed in a serial fashion, while Gremlin OLAP is executed in a parallel fashion. These two
+fundamental differences lead to the behaviors enumerated below.
+
+image::gremlin-without-a-cause.png[width=200,float=right]
+
+. Traversal sideEffects are represented as a distributed data structure across the graph's vertex set. It is not
+possible to get a global view of a sideEffect until it is aggregated via a <<mapreduce,MapReduce>> job. In some
+situations, the local vertex representation of the sideEffect is sufficient to ensure the intended semantics of the
+traversal are respected. However, this is not generally true so be wary of traversals that require global views of a
+sideEffect.
+. When evaluating traversals that rely on path information (i.e. the history of the traversal), practical
+computational limits can easily be reached due the link:http://en.wikipedia.org/wiki/Combinatorial_explosion[combinatoric explosion]
+of data. With path computing enabled, every traverser is unique and thus, must be enumerated as opposed to being
+counted/merged. The difference being a collection of paths vs. a single 64-bit long at a single vertex. For more
+information on this concept, please see link:http://thinkaurelius.com/2012/11/11/faunus-provides-big-graph-data-analytics/[Faunus Provides Big Graph Data].
+. When traversals of the form `x.as('a').y.someSideEffectStep('a').z` are evaluated, the `a` object is stored in the
+path information of the traverser and thus, such traversals (may) turn on path calculations when executed on a
+`GraphComputer`
+. Steps that are concerned with the global ordering of traversers do not have a meaningful representation in
+OLAP. For example, what does <<order-step,`order()`>>-step mean when all traversers are being processed in parallel?
+Even if the traversers were aggregated and ordered, then at the next step they would return to being executed in
+parallel and thus, in an unpredictable order. When `order()`-like steps are executed at the end of a traversal (i.e
+the final step), the `TraverserMapReduce` job ensures the resultant serial representation is ordered accordingly.
+. Steps that are concerned with providing a global aggregate to the next step of computation do not have a correlate
+in OLAP. For example, <<fold-step,`fold()`>>-step can only fold up the objects at each executing vertex. Next, even
+if a global fold was possible, where would it go? Which vertex would be the host of the data structure? The
+`fold()`-step only makes sense as an end-step whereby a MapReduce job can generate the proper global-to-local data
+reduction.
[10/22] incubator-tinkerpop git commit: Made subdirectories for
various "books" in the docs.
Posted by sp...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/reference/gremlin-applications.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/reference/gremlin-applications.asciidoc b/docs/src/reference/gremlin-applications.asciidoc
new file mode 100644
index 0000000..7ef7a71
--- /dev/null
+++ b/docs/src/reference/gremlin-applications.asciidoc
@@ -0,0 +1,1760 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+[[gremlin-applications]]
+Gremlin Applications
+====================
+
+Gremlin applications represent tools that are built on top of the core APIs to help expose common functionality to
+users when working with graphs. There are two key applications:
+
+. Gremlin Console - A link:http://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop[REPL] environment for
+interactive development and analysis
+. Gremlin Server - A server that hosts script engines thus enabling remote Gremlin execution
+
+image:gremlin-lab-coat.png[width=310,float=left] Gremlin is designed to be extensible, making it possible for users
+and graph system/language providers to customize it to their needs. Such extensibility is also found in the Gremlin
+Console and Server, where a universal plugin system makes it possible to extend their capabilities. One of the
+important aspects of the plugin system is the ability to help the user install the plugins through the command line
+thus automating the process of gathering dependencies and other error prone activities.
+
+The process of plugin installation is handled by link:http://groovy.codehaus.org/Grape[Grape], which helps resolve
+dependencies into the classpath. It is therefore important to ensure that Grape is properly configured in order to
+use the automated capabilities of plugin installation. Grape is configured by `~/.groovy/grapeConfig.xml` and
+generally speaking, if that file is not present, the default settings will suffice. However, they will not suffice
+if a required dependency is not in one of the default configured repositories. Please see the
+link:http://groovy.codehaus.org/Grape[Custom Ivy Settings] section of the Grape documentation for more details on
+the defaults. TinkerPop recommends the following configuration in that file:
+
+[source,xml]
+<ivysettings>
+ <settings defaultResolver="downloadGrapes"/>
+ <resolvers>
+ <chain name="downloadGrapes">
+ <filesystem name="cachedGrapes">
+ <ivy pattern="${user.home}/.groovy/grapes/[organisation]/[module]/ivy-[revision].xml"/>
+ <artifact pattern="${user.home}/.groovy/grapes/[organisation]/[module]/[type]s/[artifact]-[revision].[ext]"/>
+ </filesystem>
+ <ibiblio name="codehaus" root="http://repository.codehaus.org/" m2compatible="true"/>
+ <ibiblio name="central" root="http://central.maven.org/maven2/" m2compatible="true"/>
+ <ibiblio name="java.net2" root="http://download.java.net/maven/2/" m2compatible="true"/>
+ <ibiblio name="hyracs-releases" root="http://obelix.ics.uci.edu/nexus/content/groups/hyracks-public-releases/" m2compatible="true"/>
+ </chain>
+ </resolvers>
+</ivysettings>
+
+Note that if the intention is to work with TinkerPop snapshots then the file should also include:
+
+[source,xml]
+<ibiblio name="apache-snapshots" root="http://repository.apache.org/snapshots/" m2compatible="true"/>
+
+Additionally, the Graph configuration can also be modified to include the local system's Maven `.m2` directory by including:
+
+[source,xml]
+<ibiblio name="local" root="file:${user.home}/.m2/repository/" m2compatible="true"/>
+
+This configuration is useful during development (i.e. if one is working with locally built artifacts) of TinkerPop
+Plugins. Consider adding the "local" reference first in the set of `<ibilio>` resolvers, as putting it after
+"apache-snapshots" will likely resolve dependencies from that repository before looking locally. If it does that,
+then it's possible that the artifact from the newer local build will not be used.
+
+CAUTION: If building TinkerPop from source, be sure to clear TinkerPop-related jars from the `~/.groovy/grapes`
+directory as they can become stale on some systems and not re-import properly from the local `.m2` after fresh rebuilds.
+
+[[gremlin-console]]
+Gremlin Console
+---------------
+
+image:gremlin-console.png[width=325,float=right] The Gremlin Console is an interactive terminal or
+link:http://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop[REPL] that can be used to traverse graphs
+and interact with the data that they contain. It represents the most common method for performing ad-hoc graph
+analysis, small to medium sized data loading projects and other exploratory functions. The Gremlin Console is
+highly extensible, featuring a rich plugin system that allows new tools, commands,
+link:http://en.wikipedia.org/wiki/Domain-specific_language[DSLs], etc. to be exposed to users.
+
+To start the Gremlin Console, run `gremlin.sh` or `gremlin.bat`:
+
+[source,text]
+----
+$ bin/gremlin.sh
+
+ \,,,/
+ (o o)
+-----oOOo-(3)-oOOo-----
+plugin loaded: tinkerpop.server
+plugin loaded: tinkerpop.utilities
+plugin loaded: tinkerpop.tinkergraph
+gremlin>
+----
+
+NOTE: If the above plugins are not loaded then they will need to be enabled or else certain examples will not work.
+If using the standard Gremlin Console distribution, then the plugins should be enabled by default. See below for
+more information on the `:plugin use` command to manually enable plugins. These plugins, with the exception of
+`tinkerpop.tinkergraph`, cannot be removed from the Console as they are a part of the `gremlin-console.jar` itself.
+These plugins can only be deactivated.
+
+The Gremlin Console is loaded and ready for commands. Recall that the console hosts the Gremlin-Groovy language.
+Please review link:http://groovy.codehaus.org/[Groovy] for help on Groovy-related constructs. In short, Groovy is a
+superset of Java. What works in Java, works in Groovy. However, Groovy provides many shorthands to make it easier
+to interact with the Java API. Moreoever, Gremlin provides many neat shorthands to make it easier to express paths
+through a property graph.
+
+[gremlin-groovy]
+----
+i = 'goodbye'
+j = 'self'
+i + " " + j
+"${i} ${j}"
+----
+
+The "toy" graph provides a way to get started with Gremlin quickly.
+
+[gremlin-groovy]
+----
+g = TinkerFactory.createModern().traversal(standard())
+g.V()
+g.V().values('name')
+g.V().has('name','marko').out('knows').values('name')
+----
+
+TIP: When using Gremlin-Groovy in a Groovy class file, add `static { GremlinLoader.load() }` to the head of the file.
+
+Console Commands
+~~~~~~~~~~~~~~~~
+
+In addition to the standard commands of the link:http://groovy.codehaus.org/Groovy+Shell[Groovy Shell], Gremlin adds
+some other useful operations. The following table outlines the most commonly used commands:
+
+[width="100%",cols="3,^2,10",options="header"]
+|=========================================================
+|Command |Alias |Description
+|:help |:? |Displays list of commands and descriptions. When followed by a command name, it will display more specific help on that particular item.
+|:exit |:x |Ends the Console session.
+|import |:i |Import a class into the Console session.
+|:clear |:c |Sometimes the Console can get into a state where the command buffer no longer understands input (e.g. a misplaced `(` or `}`). Use this command to clear that buffer.
+|:load |:l |Load a file or URL into the command buffer for execution.
+|:install |:+ |Imports a maven library and its dependencies into the Console.
+|:uninstall |:- |Removes a maven library and its dependencies. A restart of the console is required for removal to fully take effect.
+|:plugin |:pin |Plugin management functions to list, activate and deactivate available plugins.
+|:remote |:rem |Configures a "remote" context where Gremlin or results of Gremlin will be processed via usage of `:submit`.
+|:submit |:> |Submit Gremlin to the currently active context defined by `:remote`.
+|=========================================================
+
+Gremlin Console adds a special `max-iteration` preference that can be configured with the standard `:set` command
+from the Groovy Shell. Use this setting to control the maximum number of results that the Console will display.
+Consider the following usage:
+
+[gremlin-groovy]
+----
+:set max-iteration 10
+(0..200)
+:set max-iteration 5
+(0..200)
+----
+
+If this setting is not present, the console will default the maximum to 100 results.
+
+Dependencies and Plugin Usage
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The Gremlin Console can dynamically load external code libraries and make them available to the user. Furthermore,
+those dependencies may contain Gremlin plugins which can expand the language, provide useful functions, etc. These
+important console features are managed by the `:install` and `:plugin` commands.
+
+The following Gremlin Console session demonstrates the basics of these features:
+
+[source,groovy]
+----
+gremlin> :plugin list <1>
+==>tinkerpop.server[active]
+==>tinkerpop.gephi
+==>tinkerpop.utilities[active]
+==>tinkerpop.sugar
+==>tinkerpop.tinkergraph[active]
+gremlin> :plugin use tinkerpop.sugar <2>
+==>tinkerpop.sugar activated
+gremlin> :install org.apache.tinkerpop neo4j-gremlin x.y.z <3>
+==>loaded: [org.apache.tinkerpop, neo4j-gremlin, x.y.z]
+gremlin> :plugin list <4>
+==>tinkerpop.server[active]
+==>tinkerpop.gephi
+==>tinkerpop.utilities[active]
+==>tinkerpop.sugar
+==>tinkerpop.tinkergraph[active]
+==>tinkerpop.neo4j
+gremlin> :plugin use tinkerpop.neo4j <5>
+==>tinkerpop.neo4j activated
+gremlin> :plugin list <6>
+==>tinkerpop.server[active]
+==>tinkerpop.gephi
+==>tinkerpop.sugar[active]
+==>tinkerpop.utilities[active]
+==>tinkerpop.neo4j[active]
+==>tinkerpop.tinkergraph[active]
+----
+
+<1> Show a list of "available" plugins. The list of "available" plugins is determined by the classes available on
+the Console classpath. Plugins need to be "active" for their features to be available.
+<2> To make a plugin "active" execute the `:plugin use` command and specify the name of the plugin to enable.
+<3> Sometimes there are external dependencies that would be useful within the Console. To bring those in, execute
+`:install` and specify the Maven coordinates for the dependency.
+<4> Note that there is a "tinkerpop.neo4j" plugin available, but it is not yet "active".
+<5> Again, to use the "tinkerpop.neo4j" plugin, it must be made "active" with `:plugin use`.
+<6> Now when the plugin list is displayed, the "tinkerpop.neo4j" plugin is displayed as "active".
+
+CAUTION: Plugins must be compatible with the version of the Gremlin Console (or Gremlin Server) being used. Attempts
+to use incompatible versions cannot be guaranteed to work. Moreover, be prepared for dependency conflicts in
+third-party plugins, that may only be resolved via manual jar removal from the `ext/{plugin}` directory.
+
+TIP: It is possible to manage plugin activation and deactivation by manually editing the `ext/plugins.txt` file which
+contains the class names of the "active" plugins. It is also possible to clear dependencies added by `:install` by
+deleting them from the `ext` directory.
+
+Script Executor
+~~~~~~~~~~~~~~~
+
+For automated tasks and batch executions of Gremlin, it can be useful to execute Gremlin scripts from the command
+line. Consider the following file named `gremlin.groovy`:
+
+[source,groovy]
+----
+import org.apache.tinkerpop.gremlin.tinkergraph.structure.*
+graph = TinkerFactory.createModern()
+g = graph.traversal()
+g.V().each { println it }
+----
+
+This script creates the toy graph and then iterates through all its vertices printing each to the system out. Note
+that under this approach, "imports" need to be explicitly defined (except for "core" TinkerPop classes). In addition,
+plugins and other dependencies should already be "installed" via console commands which cannot be used with this mode
+of execution. To execute this script from the command line, `gremlin.sh` has the `-e` option used as follows:
+
+[source,bash]
+----
+$ bin/gremlin.sh -e gremlin.groovy
+v[1]
+v[2]
+v[3]
+v[4]
+v[5]
+v[6]
+----
+
+It is also possible to pass arguments to scripts. Any parameters following the file name specification are treated
+as arguments to the script. They are collected into a list and passed in as a variable called "args". The following
+Gremlin script is exactly like the previous one, but it makes use of the "args" option to filter the vertices printed
+to system out:
+
+[source,groovy]
+----
+import org.apache.tinkerpop.gremlin.tinkergraph.structure.*
+graph = TinkerFactory.createModern()
+g = graph.traversal()
+g.V().has('name',args[0]).each { println it }
+----
+
+When executed from the command line a parameter can be supplied:
+
+[source,bash]
+----
+$ bin/gremlin.sh -e gremlin.groovy marko
+v[1]
+$ bin/gremlin.sh -e gremlin.groovy vadas
+v[2]
+----
+
+[[gremlin-server]]
+Gremlin Server
+--------------
+
+image:gremlin-server.png[width=400,float=right] Gremlin Server provides a way to remotely execute Gremlin scripts
+against one or more `Graph` instances hosted within it. The benefits of using Gremlin Server include:
+
+* Allows any Gremlin Structure-enabled graph to exist as a standalone server, which in turn enables the ability for
+multiple clients to communicate with the same graph database.
+* Enables execution of ad-hoc queries through remotely submitted Gremlin scripts.
+* Allows for the hosting of Gremlin-based DSLs (Domain Specific Language) that expand the Gremlin language to match
+the language of the application domain, which will help support common graph use cases such as searching, ranking,
+and recommendation.
+* Provides a method for Non-JVM languages (e.g. Python, Javascript, etc.) to communicate with the TinkerPop stack.
+* Exposes numerous methods for extension and customization to include serialization options, remote commands, etc.
+
+NOTE: Gremlin Server is the replacement for link:http://rexster.tinkerpop.com[Rexster].
+
+By default, communication with Gremlin Server occurs over link:http://en.wikipedia.org/wiki/WebSocket[WebSockets] and
+exposes a custom sub-protocol for interacting with the server.
+
+[[connecting-via-console]]
+Connecting via Console
+~~~~~~~~~~~~~~~~~~~~~~
+
+The most direct way to get started with Gremlin Server is to issue it some remote Gremlin scripts from the Gremlin
+Console. To do that, first start Gremlin Server:
+
+[source,text]
+----
+$ bin/gremlin-server.sh conf/gremlin-server-modern.yaml
+[INFO] GremlinServer -
+ \,,,/
+ (o o)
+-----oOOo-(3)-oOOo-----
+
+[INFO] GremlinServer - Configuring Gremlin Server from conf/gremlin-server-modern.yaml
+[INFO] MetricManager - Configured Metrics Slf4jReporter configured with interval=180000ms and loggerName=org.apache.tinkerpop.gremlin.server.Settings$Slf4jReporterMetrics
+[INFO] Graphs - Graph [graph] was successfully configured via [conf/tinkergraph-empty.properties].
+[INFO] ServerGremlinExecutor - Initialized Gremlin thread pool. Threads in pool named with pattern gremlin-*
+[INFO] ScriptEngines - Loaded gremlin-groovy ScriptEngine
+[INFO] GremlinExecutor - Initialized gremlin-groovy ScriptEngine with scripts/generate-modern.groovy
+[INFO] ServerGremlinExecutor - Initialized GremlinExecutor and configured ScriptEngines.
+[INFO] ServerGremlinExecutor - A GraphTraversalSource is now bound to [g] with graphtraversalsource[tinkergraph[vertices:0 edges:0], standard]
+[INFO] GremlinServer - Executing start up LifeCycleHook
+[INFO] Logger$info - Loading 'modern' graph data.
+[INFO] AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo with org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0
+[INFO] AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo-stringd with org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0
+[INFO] GremlinServer$1 - Gremlin Server configured with worker thread pool of 1, gremlin pool of 8 and boss thread pool of 1.
+[INFO] GremlinServer$1 - Channel started at port 8182.
+----
+
+Gremlin Server is configured by the provided link:http://www.yaml.org/[YAML] file `conf/gremlin-server-modern.yaml`.
+That file tells Gremlin Server many things such as:
+
+* The host and port to serve on
+* Thread pool sizes
+* Where to report metrics gathered by the server
+* The serializers to make available
+* The Gremlin `ScriptEngine` instances to expose and external dependencies to inject into them
+* `Graph` instances to expose
+
+The log messages that printed above show a number of things, but most importantly, there is a `Graph` instance named
+`graph` that is exposed in Gremlin Server. This graph is an in-memory TinkerGraph and was empty at the start of the
+server. An initialization script at `scripts/generate-modern.groovy` was executed during startup. It's contents are
+as follows:
+
+[source,groovy]
+----
+include::{basedir}/gremlin-server/scripts/generate-modern.groovy[]
+----
+
+The script above initializes a `Map` and assigns two key/values to it. The first, assigned to "hook", defines a
+`LifeCycleHook` for Gremlin Server. The "hook" provides a way to tie script code into the Gremlin Server startup and
+shutdown sequences. The `LifeCycleHook` has two methods that can be implemented: `onStartUp` and `onShutDown`.
+These events are called once at Gremlin Server start and once at Gremlin Server stop. This is an important point
+because code outside of the "hook" is executed for each `ScriptEngine` creation (multiple may be created when
+"sessions" are enabled) and therefore the `LifeCycleHook` provides a way to ensure that a script is only executed a
+single time. In this case, the startup hook loads the "modern" graph into the empty TinkerGraph instance, preparing
+it for use. The second key/value pair assigned to the `Map`, named "g", defines a `TraversalSource` from the `Graph`
+bound to the "graph" variable in the YAML configuration file. This variable `g`, as well as any other variable
+assigned to the `Map`, will be made available as variables for future remote script executions. In more general
+terms, any key/value pairs assigned to a `Map` returned from the init script will become variables that are global
+to all requests. In addition, any functions that are defined will be cached for future use.
+
+With Gremlin Server running it is now possible to issue some scripts to it for processing. Start Gremlin Console as follows:
+
+[source,text]
+----
+$ bin/gremlin.sh
+
+ \,,,/
+ (o o)
+-----oOOo-(3)-oOOo-----
+gremlin>
+----
+
+The console has the notion of a "remote", which represents a place a script will be sent from the console to be
+evaluated elsewhere in some other context (e.g. Gremlin Server, Hadoop, etc.). To create a remote in the console,
+do the following:
+
+[gremlin-groovy]
+----
+:remote connect tinkerpop.server conf/remote.yaml
+----
+
+The `:remote` command shown above displays the current status of the remote connection. This command can also be
+used to configure a new connection and change other related settings. To actually send a script to the server a
+different command is required:
+
+[gremlin-groovy]
+----
+:> g.V().values('name')
+:> g.V().has('name','marko').out('created').values('name')
+:> g.E().label().groupCount()
+result
+:remote close
+----
+
+The `:>` command, which is a shorthand for `:submit`, sends the script to the server to execute there. Results are
+wrapped in an `Result` object which is a just a holder for each individual result. The `class` shows the data type
+for the containing value. Note that the last script sent was supposed to return a `Map`, but its `class` is
+`java.lang.String`. By default, the connection is configured to only return text results. In other words,
+Gremlin Server is using `toString` to serialize all results back to the console. This enables virtually any
+object on the server to be returned to the console, but it doesn't allow the opportunity to work with this data
+in any way in the console itself. A different configuration of the `:remote` is required to get the results back
+as "objects":
+
+[gremlin-groovy]
+----
+:remote connect tinkerpop.server conf/remote-objects.yaml <1>
+:remote list <2>
+:> g.E().label().groupCount() <3>
+m = result[0].object <4>
+m.sort {it.value}
+script = """
+ matthias = graph.addVertex('name','matthias')
+ matthias.addEdge('co-creator',g.V().has('name','marko').next())
+ """
+:> @script <5>
+:> g.V().has('name','matthias').out('co-creator').values('name')
+:remote close
+----
+
+<1> This configuration file specifies that results should be deserialized back into an `Object` in the console with
+the caveat being that the server and console both know how to serialize and deserialize the result to be returned.
+<2> There are now two configured remote connections. The one marked by an asterisk is the one that was just created
+and denotes the current one that `:sumbit` will react to.
+<3> When the script is executed again, the `class` is no longer shown to be a `java.lang.String`. It is instead a `java.util.HashMap`.
+<4> The last result of a remote script is always stored in the reserved variable `result`, which allows access to
+the `Result` and by virtue of that, the `Map` itself.
+<5> If the submission requires multiple-lines to express, then a multi-line string can be created. The `:>` command
+realizes that the user is referencing a variable via `@` and submits the string script.
+
+TIP: In Groovy, `""" text """` is a convenient way to create a multi-line string and works well in concert with
+`:> @variable`. Note that this model of submitting a string variable works for all `:>` based plugins, not just Gremlin Server.
+
+WARNING: Not all values that can be returned from a Gremlin script end up being serializable. For example,
+submitting `:> graph` will return a `Graph` instance and in most cases those are not serializable by Gremlin Server
+and will return a serialization error. It should be noted that `TinkerGraph`, as a convenience for shipping around
+small sub-graphs, is serializable from Gremlin Server.
+
+The Gremlin Server `:remote` command has the following configuration options:
+
+[width="100%",cols="3,10a",options="header"]
+|=========================================================
+|Command |Description
+|alias |
+[width="100%",cols="3,10",options="header"]
+!=========================================================
+!Option !Description
+! _pairs_ !A set of key/value alias/binding pairs to apply to requests.
+!`reset` !Clears any aliases that were supplied in previous configurations of the remote.
+!`show` !Shows the current set of aliases which is returned as a `Map`
+!=========================================================
+|timeout |Specifies the length of time in milliseconds that the remote will wait for a response from the server.
+|=========================================================
+
+The `alias` configuration command for the Gremlin Server `:remote` can be useful in situations where there are
+multiple `Graph` or `TraversalSource` instances on the server, as it becomes possible to rename them from the client
+for purposes of execution within the context of a script. Therefore, it becomes possible to submit commands this way:
+
+[gremlin-groovy]
+----
+:remote connect tinkerpop.server conf/remote-objects.yaml
+:remote config alias x g
+:> x.E().label().groupCount()
+----
+
+Connecting via Java
+~~~~~~~~~~~~~~~~~~~
+
+[source,xml]
+----
+<dependency>
+ <groupId>org.apache.tinkerpop</groupId>
+ <artifactId>gremlin-driver</artifactId>
+ <version>x.y.z</version>
+</dependency>
+----
+
+image:gremlin-java.png[width=175,float=left] TinkerPop3 comes equipped with a reference client for Java-based
+applications. It is referred to as Gremlin Driver, which enables applications to send requests to Gremlin Server
+and get back results.
+
+Gremlin code is sent to the server from a `Client` instance. A `Client` is created as follows:
+
+[source,java]
+----
+Cluster cluster = Cluster.open(); <1>
+Client client = cluster.connect(); <2>
+----
+
+<1> Opens a reference to `localhost` - note that there are many configuration options available in defining a `Cluster` object.
+<2> Creates a `Client` given the configuration options of the `Cluster`.
+
+Once a `Client` instance is ready, it is possible to issue some Gremlin:
+
+[source,java]
+----
+ResultSet results = client.submit("[1,2,3,4]"); <1>
+results.stream().map(i -> i.get(Integer.class) * 2); <2>
+
+CompletableFuture<List<Result>> results = client.submit("[1,2,3,4]").all(); <3>
+
+CompletableFuture<ResultSet> future = client.submitAsync("[1,2,3,4]"); <4>
+
+Map<String,Object> params = new HashMap<>()
+params.put("x",4)
+client.submit("[1,2,3,x]", params); <5>
+----
+
+<1> Submits a script that simply returns a `List` of integers. This method blocks until the request is written to
+the server and a `ResultSet` is constructed.
+<2> Even though the `ResultSet` is constructed, it does not mean that the server has sent back the results (or even
+evaluated the script potentially). The `ResultSet` is just a holder that is awaiting the results from the server.
+In this case, they are streamed from the server as they arrive.
+<3> Submit a script, get a `ResultSet`, then return a `CompletableFuture` that will be called when all results have been returned.
+<4> Submit a script asynchronously without waiting for the request to be written to the server.
+<5> Parameterized request are considered the most efficient way to send Gremlin to the server as they can be cached,
+which will boost performance and reduce resources required on the server.
+
+Aliases
+^^^^^^^
+
+Scripts submitted to Gremlin Server automatically have the globally configured `Graph` and `TraversalSource` instances
+made available to them. Therefore, if Gremlin Server configures two `TraversalSource` instances called "g1" and "g2"
+a script can simply reference them directly as:
+
+[source,java]
+client.submit("g1.V()")
+client.submit("g2.V()")
+
+While this is an acceptable way to submit scripts, it has the downside of forcing the client to encode the server-side
+variable name directly into the script being sent. If the server configuration ever changed such that "g1" became
+"g100", the client-side code might have to see a significant amount of change. Decoupling the script code from the
+server configuration can be managed by the `alias` method on `Client` as follows:
+
+[source,java]
+Client g1Client = client.alias("g1")
+Client g2Client = client.alias("g2")
+g1Client.submit("g.V()")
+g2Client.submit("g.V()")
+
+The above code demonstrates how the `alias` method can be used such that the script need only contain a reference
+to "g" and "g1" and "g2" are automatically rebound into "g" on the server-side.
+
+Serialization
+^^^^^^^^^^^^^
+
+When using Gryo serialization (the default serializer for the driver), it is important that the client and server
+have the same serializers configured or else one or the other will experience serialization exceptions and fail to
+always communicate. Discrepancy in serializer registration between client and server can happen fairly easily as
+graphs will automatically include serializers on the server-side, thus leaving the client to be configured manually.
+This can be done manually as follows:
+
+[source,java]
+GryoMapper kryo = GryoMapper.build().addRegistry(TitanIoRegistry.INSTANCE).create();
+MessageSerializer serializer = new GryoMessageSerializerV1d0(kryo);
+Cluster cluster = Cluster.build()
+ .serializer(serializer)
+ .create();
+Client client = cluster.connect().init();
+
+The above code demonstrates using the `TitanIoRegistry` which is an `IoRegistry` instance. It tells the serializer
+what classes (from Titan in this case) to auto-register during serialization. Gremlin Server roughly uses this same
+approach when it configures it's serializers, so using this same model will ensure compatibility when making requests.
+
+Connecting via REST
+~~~~~~~~~~~~~~~~~~~
+
+image:gremlin-rexster.png[width=225,float=left] While the default behavior for Gremlin Server is to provide a
+WebSockets-based connection, it can also be configured to support link:http://en.wikipedia.org/wiki/Representational_state_transfer[REST].
+The REST endpoint provides for a communication protocol familiar to most developers, with a wide support of
+programming languages, tools and libraries for accessing it. As a result, REST provides a fast way to get started
+with Gremlin Server. It also may represent an easier upgrade path from link:http://rexster.tinkerpop.com/[Rexster]
+as the API for the endpoint is very similar to Rexster's link:https://github.org/apache/tinkerpop/rexster/wiki/Gremlin-Extension[Gremlin Extension].
+
+Gremlin Server provides for a single REST endpoint - a Gremlin evaluator - which allows the submission of a Gremlin
+script as a request. For each request, it returns a response containing the serialized results of that script.
+To enable this endpoint, Gremlin Server needs to be configured with the `HttpChannelizer`, which replaces the default
+`WebSocketChannelizer`, in the configuration file:
+
+[source,yaml]
+channelizer: org.apache.tinkerpop.gremlin.server.channel.HttpChannelizer
+
+This setting is already configured in the `gremlin-server-rest-modern.yaml` file that is packaged with the Gremlin
+Server distribution. To utilize it, start Gremlin Server as follows:
+
+[source,text]
+bin/gremlin-server.sh conf/gremlin-server-rest-modern.yaml
+
+Once the server has started, issue a request. Here's an example with link:http://curl.haxx.se/[cURL]:
+
+[source,text]
+$ curl "http://localhost:8182?gremlin=100-1"
+
+which returns:
+
+[source,js]
+{
+ "result":{"data":99,"meta":{}},
+ "requestId":"0581cdba-b152-45c4-80fa-3d36a6eecf1c",
+ "status":{"code":200,"attributes":{},"message":""}
+}
+
+The above example showed a `GET` operation, but the preferred method for this endpoint is `POST`:
+
+[source,text]
+curl -X POST -d "{\"gremlin\":\"100-1\"}" "http://localhost:8182"
+
+which returns:
+
+[source,js]
+{
+ "result":{"data":99,"meta":{}},
+ "requestId":"ef2fe16c-441d-4e13-9ddb-3c7b5dfb10ba",
+ "status":{"code":200,"attributes":{},"message":""}
+}
+
+It is also preferred that Gremlin scripts be parameterized when possible via `bindings`:
+
+[source,text]
+curl -X POST -d "{\"gremlin\":\"100-x\", \"bindings\":{\"x\":1}}" "http://localhost:8182"
+
+The `bindings` argument is a `Map` of variables where the keys become available as variables in the Gremlin script.
+Note that parameterization of requests is critical to performance, as repeated script compilation can be avoided on
+each request.
+
+NOTE: It is possible to pass bindings via `GET` based requests. Query string arguments prefixed with "bindings." will
+be treated as parameters, where that prefix will be removed and the value following the period will become the
+parameter name. In other words, `bindings.x` will create a parameter named "x" that can be referenced in the submitted
+Gremlin script. The caveat is that these arguments will always be treated as `String` values. To ensure that data
+types are preserved or to pass complex objects such as lists or maps, use `POST` which will at least support the
+allowed JSON data types.
+
+Finally, as Gremlin Server can host multiple `ScriptEngine` instances (e.g. `gremlin-groovy`, `nashorn`), it is
+possible to define the language to utilize to process the request:
+
+[source,text]
+curl -X POST -d "{\"gremlin\":\"100-x\", \"language\":\"gremlin-groovy\", \"bindings\":{\"x\":1}}" "http://localhost:8182"
+
+By default this value is set to `gremlin-groovy`. If using a `GET` operation, this value can be set as a query
+string argument with by setting the `language` key.
+
+CAUTION: Consider the size of the result of a submitted script being returned from the REST endpoint. A script
+that iterates thousands of results will serialize each of those in memory into a single JSON result set. It is
+quite possible that such a script will generate `OutOfMemoryError` exceptions on the server. Consider the default
+WebSockets configuration, which supports streaming, if that type of use case is required.
+
+Configuring
+~~~~~~~~~~~
+
+As mentioned earlier, Gremlin Server is configured though a YAML file. By default, Gremlin Server will look for a
+file called `config/gremlin-server.yaml` to configure itself on startup. To override this default, supply the file
+to use to `bin/gremlin-server.sh` as in:
+
+[source,text]
+----
+bin/gremlin-server.sh conf/gremlin-server-min.yaml
+----
+
+The `gremlin-server.sh` file also serves a second purpose. It can be used to "install" dependencies to the Gremlin
+Server path. For example, to be able to configure and use other `Graph` implementations, the dependencies must be
+made available to Gremlin Server. To do this, use the `-i` switch and supply the Maven coordinates for the dependency
+to "install". For example, to use Neo4j in Gremlin Server:
+
+[source,text]
+----
+bin/gremlin-server.sh -i org.apache.tinkerpop neo4j-gremlin x.y.z
+----
+
+This command will "grab" the appropriate dependencies and copy them to the `ext` directory of Gremlin Server, which
+will then allow them to be "used" the next time the server is started. To uninstall dependencies, simply delete them
+from the `ext` directory.
+
+The following table describes the various configuration options that Gremlin Server expects:
+
+[width="100%",cols="3,10,^2",options="header"]
+|=========================================================
+|Key |Description |Default
+|authentication.className |The fully qualified classname of an `Authenticator` implementation to use. If this setting is not present, then authentication is effectively disabled. |`AllowAllAuthenticator`
+|authentication.config |A `Map` of configuration settings to be passes to the `Authenticator` when it is constructed. The settings available are dependent on the implementation. |_none_
+|channelizer |The fully qualified classname of the `Channelizer` implementation to use. A `Channelizer` is a "channel initializer" which Gremlin Server uses to define the type of processing pipeline to use. By allowing different `Channelizer` implementations, Gremlin Server can support different communication protocols (e.g. Websockets, Java NIO, etc.). |WebSocketChannelizer
+|graphs |A `Map` of `Graph` configuration files where the key of the `Map` becomes the name to which the `Graph` will be bound and the value is the file name of a `Graph` configuration file. |_none_
+|gremlinPool |The number of "Gremlin" threads available to execute actual scripts in a `ScriptEngine`. This pool represents the workers available to handle blocking operations in Gremlin Server. |8
+|host |The name of the host to bind the server to. |localhost
+|useEpollEventLoop |try to use epoll event loops (works only on Linux os) instead of netty NIO. |false
+|maxAccumulationBufferComponents |Maximum number of request components that can be aggregated for a message. |1024
+|maxChunkSize |The maximum length of the content or each chunk. If the content length exceeds this value, the transfer encoding of the decoded request will be converted to 'chunked' and the content will be split into multiple `HttpContent` objects. If the transfer encoding of the HTTP request is 'chunked' already, each chunk will be split into smaller chunks if the length of the chunk exceeds this value. |8192
+|maxContentLength |The maximum length of the aggregated content for a message. Works in concert with `maxChunkSize` where chunked requests are accumulated back into a single message. A request exceeding this size will return a `413 - Request Entity Too Large` status code. A response exceeding this size will raise an internal exception. |65536
+|maxHeaderSize |The maximum length of all headers. |8192
+|maxInitialLineLength |The maximum length of the initial line (e.g. "GET / HTTP/1.0") processed in a request, which essentially controls the maximum length of the submitted URI. |4096
+|metrics.consoleReporter.enabled |Turns on console reporting of metrics. |false
+|metrics.consoleReporter.interval |Time in milliseconds between reports of metrics to console. |180000
+|metrics.csvReporter.enabled |Turns on CSV reporting of metrics. |false
+|metrics.csvReporter.fileName |The file to write metrics to. |_none_
+|metrics.csvReporter.interval |Time in milliseconds between reports of metrics to file. |180000
+|metrics.gangliaReporter.addressingMode |Set to `MULTICAST` or `UNICAST`. |_none_
+|metrics.gangliaReporter.enabled |Turns on Ganglia reporting of metrics. |false
+|metrics.gangliaReporter.host |Define the Ganglia host to report Metrics to. |localhost
+|metrics.gangliaReporter.interval |Time in milliseconds between reports of metrics for Ganglia. |180000
+|metrics.gangliaReporter.port |Define the Ganglia port to report Metrics to. |8649
+|metrics.graphiteReporter.enabled |Turns on Graphite reporting of metrics. |false
+|metrics.graphiteReporter.host |Define the Graphite host to report Metrics to. |localhost
+|metrics.graphiteReporter.interval |Time in milliseconds between reports of metrics for Graphite. |180000
+|metrics.graphiteReporter.port |Define the Graphite port to report Metrics to. |2003
+|metrics.graphiteReporter.prefix |Define a "prefix" to append to metrics keys reported to Graphite. |_none_
+|metrics.jmxReporter.enabled |Turns on JMX reporting of metrics. |false
+|metrics.slf4jReporter.enabled |Turns on SLF4j reporting of metrics. |false
+|metrics.slf4jReporter.interval |Time in milliseconds between reports of metrics to SLF4j. |180000
+|plugins |A list of plugins that should be activated on server startup in the available script engines. It assumes that the plugins are in Gremlin Server's classpath. |_none_
+|port |The port to bind the server to. |8182
+|processors |A `List` of `Map` settings, where each `Map` represents a `OpProcessor` implementation to use along with its configuration. |_none_
+|processors[X].className |The full class name of the `OpProcessor` implementation. |_none_
+|processors[X].config |A `Map` containing `OpProcessor` specific configurations. |_none_
+|resultIterationBatchSize |Defines the size in which the result of a request is "batched" back to the client. In other words, if set to `1`, then a result that had ten items in it would get each result sent back individually. If set to `2` the same ten results would come back in five batches of two each. |64
+|scriptEngines |A `Map` of `ScriptEngine` implementations to expose through Gremlin Server, where the key is the name given by the `ScriptEngine` implementation. The key must match the name exactly for the `ScriptEngine` to be constructed. The value paired with this key is itself a `Map` of configuration for that `ScriptEngine`. |_none_
+|scriptEngines.<name>.imports |A comma separated list of classes/packages to make available to the `ScriptEngine`. |_none_
+|scriptEngines.<name>.staticImports |A comma separated list of "static" imports to make available to the `ScriptEngine`. |_none_
+|scriptEngines.<name>.scripts |A comma separated list of script files to execute on `ScriptEngine` initialization. `Graph` and `TraversalSource` instance references produced from scripts will be stored globally in Gremlin Server, therefore it is possible to use initialization scripts to add Traversal Strategies or create entirely new `Graph` instances all together. Instantiating a `LifeCycleHook` in a script provides a way to execute scripts when Gremlin Server starts and stops.|_none_
+|scriptEngines.<name>.config |A `Map` of configuration settings for the `ScriptEngine`. These settings are dependent on the `ScriptEngine` implementation being used. |_none_
+|scriptEvaluationTimeout |The amount of time in milliseconds before a script evaluation times out. The notion of "script evaluation" refers to the time it takes for the `ScriptEngine` to do its work and *not* any additional time it takes for the result of the evaluation to be iterated and serialized. |30000
+|serializers |A `List` of `Map` settings, where each `Map` represents a `MessageSerializer` implementation to use along with its configuration. |_none_
+|serializers[X].className |The full class name of the `MessageSerializer` implementation. |_none_
+|serializers[X].config |A `Map` containing `MessageSerializer` specific configurations. |_none_
+|serializedResponseTimeout |The amount of time in milliseconds before a response serialization times out. The notion of "response serialization" refers to the time it takes for Gremlin Server to iterate an entire result after the script is evaluated in the `ScriptEngine`. |30000
+|ssl.enabled |Determines if SSL is turned on or not. |false
+|ssl.keyCertChainFile |The X.509 certificate chain file in PEM format. If this value is not present and `ssl.enabled` is `true` a self-signed certificate will be used (not suitable for production). |_none_
+|ssl.keyFile |The `PKCS#8` private key file in PEM format. If this value is not present and `ssl.enabled` is `true` a self-signed certificate will be used (not suitable for production). |_none_
+|ssl.keyPassword |The password of the `keyFile` if it's not password-protected |_none_
+|ssl.trustCertChainFile |Trusted certificates for verifying the remote endpoint's certificate. The file should contain an X.509 certificate chain in PEM format. A system default will be used if this setting is not present. |_none_
+|threadPoolBoss |The number of threads available to Gremlin Server for accepting connections. Should always be set to `1`. |1
+|threadPoolWorker |The number of threads available to Gremlin Server for processing non-blocking reads and writes. |1
+|writeBufferHighWaterMark | If the number of bytes in the network send buffer exceeds this value then the channel is no longer writeable, accepting no additional writes until buffer is drained and the `writeBufferLowWaterMark` is met. |65536
+|writeBufferLowWaterMark | Once the number of bytes queued in the network send buffer exceeds the `writeBufferHighWaterMark`, the channel will not become writeable again until the buffer is drained and it drops below this value. |65536
+|=========================================================
+
+NOTE: Configuration of link:http://ganglia.sourceforge.net/[Ganglia] requires an additional library that is not
+packaged with Gremlin Server due to its LGPL licensing that conflicts with the TinkerPop's Apache 2.0 License. To
+run Gremlin Server with Ganglia monitoring, download the `org.acplt:oncrpc` jar from
+link:http://repo1.maven.org/maven2/org/acplt/oncrpc/1.0.7/[here] and copy it to the Gremlin Server `/lib` directory
+before starting the server.
+
+Security
+^^^^^^^^
+
+image:gremlin-server-secure.png[width=175,float=right] Gremlin Server provides for several features that aid in the
+security of the graphs that it exposes. It has built in SSL support and a pluggable authentication framework using
+link:https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer[SASL] (Simple Authentication and
+Security Layer). SSL options are described in the configuration settings table above, so this section will focus on
+authentication.
+
+By default, Gremlin Server is configured to allow all requests to be processed (i.e. no authentication). To enable
+authentication, Gremlin Server must be configured with an `Authenticator` implementation in its YAML file. Gremlin
+Server comes packaged with an implementation called `SimpleAuthenticator`. The `SimpleAuthenticator` implements the
+`PLAIN` SASL mechanism (i.e. plain text) to authenticate a request. It validates username/password pairs against a
+graph database, which must be provided to it as part of the configuration.
+
+[source,yaml]
+authentication: {
+ className: org.apache.tinkerpop.gremlin.server.auth.SimpleAuthenticator,
+ config: {
+ credentialsDb: conf/credential-graph.properties}}
+
+Quick Start
++++++++++++
+
+A quick way to get started with the `SimpleAuthenticator` is to use TinkerGraph for the "credentials graph" and the
+"sample" credential graph that is packaged with the server. Recall that TinkerGraph is an in-memory graph and
+therefore always starts as "empty" when opened by `GraphFactory`. To allow TinkerGraph to be used in this "getting
+started" capacity, Gremlin Server allows for a TinkerGraph-only configuration option called `credentialsDbLocation`.
+The following snippet comes from the `conf/gremlin-server-secure.yaml` file packaged with the server:
+
+[source,yaml]
+authentication: {
+ className: org.apache.tinkerpop.gremlin.server.auth.SimpleAuthenticator,
+ config: {
+ credentialsDb: conf/tinkergraph-empty.properties,
+ credentialsDbLocation: data/credentials.kryo}}
+
+This added configuration tells Gremlin Server to look for a gryo file at that location containing the data for the
+graph which it loads via standard `io` methods. The limitation is that this read is only performed at the
+initialization of the server so therefore credentials remain static for the life of the server. In this case,
+`data/credentials.kryo` contains a single user named "stephen" with the imaginative password of "password".
+
+[source,text]
+----
+$ bin/gremlin-server.sh conf/gremlin-server-secure.yaml
+[INFO] GremlinServer -
+ \,,,/
+ (o o)
+-----oOOo-(3)-oOOo-----
+
+[INFO] GremlinServer - Configuring Gremlin Server from conf/gremlin-server-secure.yaml
+...
+[WARN] AbstractChannelizer - Enabling SSL with self-signed certificate (NOT SUITABLE FOR PRODUCTION)
+[INFO] AbstractChannelizer - SSL enabled
+[INFO] SimpleAuthenticator - Initializing authentication with the org.apache.tinkerpop.gremlin.server.auth.SimpleAuthenticator
+[INFO] SimpleAuthenticator - CredentialGraph initialized at CredentialGraph{graph=tinkergraph[vertices:1 edges:0]}
+[INFO] GremlinServer$1 - Gremlin Server configured with worker thread pool of 1, gremlin pool of 8 and boss thread pool of 1.
+[INFO] GremlinServer$1 - Channel started at port 8182.
+----
+
+In addition to configuring the authenticator, `gremlin-server-secure.yaml` also enables SSL with a self-signed
+certificate. As SSL is enabled on the server it must also be enabled on the client when connecting. To connect to
+Gremlin Server with `gremlin-driver`, set the `credentials` and `enableSsl` when constructing the `Cluster`.
+
+[source,java]
+Cluster cluster = Cluster.build().credentials("stephen", "password")
+ .enableSsl(true).create();
+
+If connecting with Gremlin Console, which utilizes `gremlin-driver` for remote script execution, use the provided
+`config/remote-secure.yaml` file when defining the remote. That file contains configuration for the username and
+password as well as enablement of SSL from the client side.
+
+Similarly, Gremlin Server can be configured for REST and security.
+
+[source,text]
+----
+$ bin/gremlin-server.sh conf/gremlin-server-rest-secure.yaml
+[INFO] GremlinServer -
+ \,,,/
+ (o o)
+-----oOOo-(3)-oOOo-----
+
+[INFO] GremlinServer - Configuring Gremlin Server from conf/gremlin-server-secure.yaml
+...
+[WARN] AbstractChannelizer - Enabling SSL with self-signed certificate (NOT SUITABLE FOR PRODUCTION)
+[INFO] AbstractChannelizer - SSL enabled
+[INFO] SimpleAuthenticator - Initializing authentication with the org.apache.tinkerpop.gremlin.server.auth.SimpleAuthenticator
+[INFO] SimpleAuthenticator - CredentialGraph initialized at CredentialGraph{graph=tinkergraph[vertices:1 edges:0]}
+[INFO] GremlinServer$1 - Gremlin Server configured with worker thread pool of 1, gremlin pool of 8 and boss thread pool of 1.
+[INFO] GremlinServer$1 - Channel started at port 8182.
+----
+
+Once the server has started, issue a request passing the credentials with an `Authentication` header, as described in link:http://tools.ietf.org/html/rfc2617#section-2[RFC2617]. Here's a HTTP Basic authentication example with cURL:
+
+[source,text]
+curl -X POST --insecure -u stephen:password -d "{\"gremlin\":\"100-1\"}" "https://localhost:8182"
+
+[[credentials-dsl]]
+Credentials Graph DSL
++++++++++++++++++++++
+
+The "credentials graph", which has been mentioned in previous sections, is used by Gremlin Server to hold the list of
+users who can authenticate to the server. It is possible to use virtually any `Graph` instance for this task as long
+as it complies to a defined schema. The credentials graph stores users as vertices with the `label` of "user". Each
+"user" vertex has two properties: `username` and `password`. Naturally, these are both `String` values. The password
+must not be stored in plain text and should be hashed.
+
+IMPORTANT: Be sure to define an index on the `username` property, as this will be used for lookups. If supported by
+the `Graph`, consider specifying a unique constraint as well.
+
+To aid with the management of a credentials graph, Gremlin Server provides a Gremlin Console plugin which can be
+used to add and remove users so as to ensure that the schema is adhered to, thus ensuring compatibility with Gremlin
+ Server. In addition, as it is a plugin, it works naturally in the Gremlin Console as an extension of its
+ capabilities (though one could use it programmatically, if desired). This plugin is distributed with the Gremlin
+ Console so it does not have to be "installed". It does however need to be activated:
+
+[source,groovy]
+gremlin> :plugin use tinkerpop.credentials
+==>tinkerpop.credentials activated
+
+Please see the example usage as follows:
+
+[gremlin-groovy]
+----
+graph = TinkerGraph.open()
+graph.createIndex("username",Vertex.class)
+credentials = credentials(graph)
+credentials.createUser("stephen","password")
+credentials.createUser("daniel","better-password")
+credentials.createUser("marko","rainbow-dash")
+credentials.findUser("marko").properties()
+credentials.countUsers()
+credentials.removeUser("daniel")
+credentials.countUsers()
+----
+
+[[script-execution]]
+Script Execution
+++++++++++++++++
+
+It is important to remember that Gremlin Server exposes a `ScriptEngine` instance that allows for remote execution
+of arbitrary code on the server. Obviously, this situation can represent a security risk or, more minimally, provide
+ways for "bad" scripts to be inadvertently executed. A simple example of a "valid" Gremlin script that would cause
+some problems would be, `while(true) {}`, which would consume a thread in the Gremlin pool indefinitely, thus
+preventing it from serving other requests. Sending enough of these kinds of scripts would eventually consume all
+available threads and Gremlin Server would stop responding.
+
+Gremlin Server (more specifically the `GremlinGroovyScriptEngine`) provides methods to protect itself from these
+kinds of troublesome scripts. A user can configure the script engine with different `CompilerCustomizerProvider`
+implementations. Consider the basic configuration from the Gremlin Server YAML file:
+
+[source,yaml]
+scriptEngines: {
+ gremlin-groovy: {
+ imports: [java.lang.Math],
+ staticImports: [java.lang.Math.PI],
+ scripts: [scripts/empty-sample.groovy]}}
+
+This configuration can be extended to include a `config` key as follows:
+
+[source,yaml]
+scriptEngines: {
+ gremlin-groovy: {
+ imports: [java.lang.Math],
+ staticImports: [java.lang.Math.PI],
+ scripts: [scripts/empty-sample.groovy],
+ config: {
+ compilerCustomizerProviders: {
+ "org.apache.tinkerpop.gremlin.groovy.jsr223.customizer.TimedInterruptCustomizerProvider":[10000] }}}
+
+This configuration sets up the script engine with a `CompilerCustomizerProvider` implementation. The
+`TimedInterruptCustomizerProvider` injects checks that ensure that loops (like `while`) can only execute for `10000`
+milliseconds. With this configuration in place, a remote execution as follows, now times out rather than consuming
+the thread continuously:
+
+[source,groovy]
+gremlin> :remote connect tinkerpop.server conf/remote.yaml
+==>Connected - localhost/127.0.0.1:8182
+gremlin> :> while(true) { }
+Execution timed out after 10000 units. Start time: Fri Jul 24 11:04:52 EDT 2015
+
+There are a number of pre-packaged `CustomizerProvider` implementations:
+
+[width="100%",cols="3,10a",options="header"]
+|=========================================================
+|Customizer |Description
+|`CompileStaticCustomizerProvider` |Applies `CompileStatic` annotations to incoming scripts thus removing dynamic dispatch. More information about static compilation can be found in the link:http://docs.groovy-lang.org/latest/html/documentation/#_static_compilation[Groovy Documentation]. It is possible to configure this `CustomizerProvider` by specifying a comma separated list of link:http://docs.groovy-lang.org/latest/html/documentation/#Typecheckingextensions-Workingwithextensions[type checking extensions] that can have the effect of securing calls to various methods.
+|`ThreadInterruptCustomizerProvider` |Injects checks for thread interruption, thus allowing the thread to potentially respect calls to `Thread.interrupt()`
+|`TimedInterruptCustomizerProvider` |Injects checks into loops to interrupt them if they exceed the configured timeout in milliseconds.
+|`TypeCheckedCustomizerProvider` |Similar to the above mentioned, `CompileStaticCustomizerProvider`, the `TypeCheckedCustomizerProvider` injects `TypeChecked` annotations to incoming scripts. More information on the nature of this annotation can be found in the link:http://docs.groovy-lang.org/latest/html/documentation/#_the_code_typechecked_code_annotation[Groovy Documentation]. It too takes a comma separated list of link:http://docs.groovy-lang.org/latest/html/documentation/#Typecheckingextensions-Workingwithextensions[type checking extensions].
+|=========================================================
+
+To provide some basic out-of-the-box protections against troublesome scripts, the following configuration can be used:
+
+[source,yaml]
+scriptEngines: {
+ gremlin-groovy: {
+ imports: [java.lang.Math],
+ staticImports: [java.lang.Math.PI],
+ scripts: [scripts/empty-sample.groovy],
+ config: {
+ compilerCustomizerProviders: {
+ "org.apache.tinkerpop.gremlin.groovy.jsr223.customizer.ThreadInterruptCustomizerProvider":[],
+ "org.apache.tinkerpop.gremlin.groovy.jsr223.customizer.TimedInterruptCustomizerProvider":[10000],
+ "org.apache.tinkerpop.gremlin.groovy.jsr223.customizer.CompileStaticCustomizerProvider":["org.apache.tinkerpop.gremlin.groovy.jsr223.customizer.SimpleSandboxExtension"]}}}}
+
+NOTE: The above configuration could also use the `TypeCheckedCustomizerProvider` in place of the
+`CompileStaticCustomizerProvider`. The differences between `TypeChecked` and `CompileStatic` are beyond the scope of
+this documentation. Consult the latest link:http://docs.groovy-lang.org/latest/html/documentation/#_typing[Groovy Documentation]
+for information on the differences. It is important to understand the impact that these configuration will have on
+submitted scripts before enabling this feature.
+
+This configuration uses the `SimpleSandboxExtension`, which blacklists calls to methods on the `System` class,
+thereby preventing someone from remotely killing the server:
+
+[source,groovy]
+----
+gremlin> :> System.exit(0)
+Script8.groovy: 1: [Static type checking] - Not authorized to call this method: java.lang.System#exit(int)
+ @ line 1, column 1.
+ System.exit(0)
+ ^
+
+1 error
+----
+
+The `SimpleSandboxExtension` is by no means a "complete" implementation protecting against all manner of nefarious
+scripts, but it does provide an example for how such a capability might be implemented. A full implementation would
+likely represent domain specific white-listing of methods (and possibly variables) available for execution in the
+script engine.
+
+A final thought on the topic of `CompilerCustomizerProvider` implementations is that they are not just for
+"security" (though they are demonstrated in that capacity here). They can be used for a variety of features that
+can fine tune the Groovy compilation process. Read more about compilation customization in the
+link:http://docs.groovy-lang.org/latest/html/documentation/#compilation-customizers[Groovy Documentation].
+
+NOTE: The import of classes to the script engine is handled by the `ImportCustomizerProvider`. As the concept of
+"imports" is a first-class citizen (i.e. has its own configuration options), it is not recommended that the
+`ImportCustomizerProvider` be used as a configuration option to `compilerCustomizerProviders`.
+
+Serialization
+^^^^^^^^^^^^^
+
+Gremlin Server can accept requests and return results using different serialization formats. The format of the
+serialization is configured by the `serializers` setting described in the table above. Note that some serializers
+have additional configuration options as defined by the `serializers[X].config` setting. The `config` setting is a
+`Map` where the keys and values get passed to the serializer at its initialization. The available and/or expected
+ keys are dependent on the serializer being used. Gremlin Server comes packaged with two different serializers:
+ GraphSON and Gryo.
+
+GraphSON
+++++++++
+
+The GraphSON serializer produces human readable output in JSON format and is a good configuration choice for those
+trying to use TinkerPop from non-JVM languages. JSON obviously has wide support across virtually all major
+programming languages and can be consumed by a wide variety of tools.
+
+[source,yaml]
+ - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0 }
+
+The above configuration represents the default serialization under the `application/json` MIME type and produces JSON
+consistent with standard JSON data types. It has the following configuration option:
+
+[width="100%",cols="3,10,^2",options="header"]
+|=========================================================
+|Key |Description |Default
+|useMapperFromGraph |Specifies the name of the `Graph` (from the `graphs` `Map` in the configuration file) from which to plugin any custom serializers that are tied to it. |_none_
+|=========================================================
+
+[source,yaml]
+ - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0 }
+
+When the standard JSON data types are not enough (e.g. need to identify the difference between `double` and `float`
+data types), the above configuration will embed types into the JSON itself. The type embedding uses standard Java
+type names, so interpretation from non-JVM languages will be required. It has the MIME type of
+`application/vnd.gremlin-v1.0+json` and the following configuration options:
+
+[width="100%",cols="3,10,^2",options="header"]
+|=========================================================
+|Key |Description |Default
+|useMapperFromGraph |Specifies the name of the `Graph` (from the `graphs` `Map` in the configuration file) from which to plugin any custom serializers that are tied to it. |_none_
+|=========================================================
+
+Gryo
+++++
+
+The Gryo serializer utilizes Kryo-based serialization which produces a binary output. This format is best consumed
+by JVM-based languages.
+
+[source,yaml]
+ - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerGremlinV1d0 }
+
+It has the MIME type of `application/vnd.gremlin-v1.0+gryo` and the following configuration options:
+
+[width="100%",cols="3,10,^2",options="header"]
+|=========================================================
+|Key |Description |Default
+|bufferSize |The maximum size of the Kryo buffer for use on a single object being serialized. Increasing this value will correct `KryoException` errors that complain of "Buffer too small". |_4096_
+|serializeResultToString |When set to `true`, results are serialized by first calling `toString()` on each object in the result list resulting in an extended MIME Type of `application/vnd.gremlin-v1.0+gryo-stringd`. When set to `false` Kryo-based serialization is applied. |_false_
+|useMapperFromGraph |Specifies the name of the `Graph` (from the `graphs` `Map` in the configuration file) from which to plugin any custom serializers that are tied to it. |_none_
+|ioRegistries |A list of `IoRegistry` implementations to be applied to the serializer.
+|custom |A list of classes with custom kryo `Serializer` implementations related to them in the form of `<class>;<serializer-class>`. |_none_
+|=========================================================
+
+As described above, there are multiple ways in which to register serializers for Kryo-based serialization. These
+configurations can be used in conjunction with one another where there is a specific ordering to how the configurations
+are applied. The `userMapperFromGraph` setting is applied first, followed by any `ioRegistries` and finalized by the
+`custom` setting.
+
+Best Practices
+~~~~~~~~~~~~~~
+
+The following sections define best practices for working with Gremlin Server.
+
+Tuning
+^^^^^^
+
+image:gremlin-handdrawn.png[width=120,float=right] Tuning Gremlin Server for a particular environment may require some simple trial-and-error, but the following represent some basic guidelines that might be useful:
+
+* When configuring the size of `threadPoolWorker` start with the default of `1` and increment by one as needed to a maximum of `2*number of cores`.
+* The "right" size of the `gremlinPool` setting is somewhat dependent on the type of scripts that will be processed
+by Gremlin Server. As requests arrive to Gremlin Server they are decoded and queued to be processed by threads in
+this pool. When this pool is exhausted of threads, Gremlin Server will continue to accept incoming requests, but
+the queue will continue to grow. If left to grow too large, the server will begin to slow. When tuning around
+this setting, consider whether the bulk of the scripts being processed will be "fast" or "slow", where "fast"
+generally means being measured in the low hundreds of milliseconds and "slow" means anything longer than that.
+** If the bulk of the scripts being processed are expected to be "fast", then a good starting point for this setting is `2*threadPoolWorker`.
+** If the bulk of the scripts being processed are expected to be "slow", then a good starting point for this setting is `4*threadPoolWorker`.
+* Scripts that are "slow" can really hurt Gremlin Server if they are not properly accounted for. `ScriptEngine`
+evaluations are blocking operations that aren't easily interrupted, so once a "slow" script is being evaluated in
+the context of a `ScriptEngine` it must finish its work. Lots of "slow" scripts will eventually consume the
+`gremlinPool` preventing other scripts from getting processed from the queue.
+** To limit the impact of this problem consider properly setting the `scriptEvaluationTimeout` and the `serializedResponseTimeout` to something "sane".
+** Test the traversals being sent to Gremlin Server and determine the maximum time they take to evaluate and iterate
+over results, then set these configurations accordingly.
+** Note that `scriptEvaluationTimeout` does not interrupt the evaluation on timeout. It merely allows Gremlin Server
+to "ignore" the result of that evaluation, which means the thread in the `gremlinPool` will still be consumed after
+the timeout.
+** The `serializedResponseTimeout` will kill the result iteration process and prevent additional processing. In most
+situations, the iteration and serialization process is the more costly step in this process as an errant script that
+returns a million or more results could send Gremlin Server into a long streaming cycle. Script evaluation on the
+other hand is usually very fast, occurring on the order of milliseconds, but that is entirely dependent on the
+contents of the script itself.
+
+[[parameterized-scripts]]
+Parameterized Scripts
+^^^^^^^^^^^^^^^^^^^^^
+
+image:gremlin-parameterized.png[width=150,float=left] Use script parameterization. Period. Gremlin Server caches
+all scripts that are passed to it. The cache is keyed based on the a hash of the script. Therefore `g.V(1)` and
+`g.V(2)` will be recognized as two separate scripts in the cache. If that script is parameterized to `g.V(x)`
+where `x` is passed as a parameter from the client, there will be no additional compilation cost for future requests
+on that script. Compilation of a script should be considered "expensive" and avoided when possible.
+
+Cache Management
+^^^^^^^^^^^^^^^^
+
+If Gremlin Server processes a large number of unique scripts, the cache will grow beyond the memory available to
+Gremlin Server and an `OutOfMemoryError` will loom. Script parameterization goes a long way to solving this problem
+and running out of memory should not be an issue for those cases. If it is a problem or if there is no script
+parameterization due to a given use case (perhaps using with use of <<sessions,sessions>>), it is possible to better
+control the nature of the script cache from the client side, by issuing scripts with a parameter to help define how
+the garbage collector should treat the references.
+
+The parameter is called `#jsr223.groovy.engine.keep.globals` and has four options:
+
+* `hard` - available in the cache for the life of the JVM (default when not specified).
+* `soft` - retained until memory is "low" and should be reclaimed before an `OutOfMemoryError` is thrown.
+* `weak` - garbage collected even when memory is abundant.
+* `phantom` - removed immediately after being evaluated by the `ScriptEngine`.
+
+By specifying an option other than `hard`, an `OutOfMemoryError` in Gremlin Server should be avoided. Of course,
+this approach will come with the downside that compiled scripts could be garbage collected and thus removed from the
+cache, forcing Gremlin Server to recompile later.
+
+[[sessions]]
+Considering Sessions
+^^^^^^^^^^^^^^^^^^^^
+
+The preferred approach for issuing requests to Gremlin Server is to do so in a sessionless manner. The concept of
+"sessionless" refers to a request that is completely encapsulated within a single transaction, such that the script
+in the request starts with a new transaction and ends with closed transaction. Sessionless requests have automatic
+transaction management handled by Gremlin Server, thus automatically opening and closing transactions as previously
+described. The downside to the sessionless approach is that the entire script to be executed must be known at the
+time of submission so that it can all be executed at once. This requirement makes it difficult for some use cases
+where more control over the transaction is desired.
+
+For such use cases, Gremlin Server supports sessions. With sessions, the user is in complete control of the start
+and end of the transaction. This feature comes with some additional expense to consider:
+
+* Initialization scripts will be executed for each session created so any expense related to them will be established
+each time a session is constructed.
+* There will be one script cache per session, which obviously increases memory requirements. The cache is not shared,
+so as to ensure that a session has isolation from other session environments. As a result, if the same script is
+executed in each session the same compilation cost will be paid for each session it is executed in.
+* Each session will require its own thread pool with a single thread in it - this ensures that transactional
+boundaries are managed properly from one request to the next.
+* If there are multiple Gremlin Server instances, communication from the client to the server must be bound to the
+server that the session was initialized in. Gremlin Server does not share session state as the transactional context
+of a `Graph` is bound to the thread it was initialized in.
+
+A session is a "heavier" approach to the simple "request/response" approach of sessionless requests, but is sometimes
+necessary for a given use case.
+
+Developing a Driver
+~~~~~~~~~~~~~~~~~~~
+
+image::gremlin-server-protocol.png[width=325]
+
+One of the roles for Gremlin Server is to provide a bridge from TinkerPop to non-JVM languages (e.g. Go, Python,
+etc.). Developers can build language bindings (or driver) that provide a way to submit Gremlin scripts to Gremlin
+Server and get back results. Given the exstensible nature of Gremlin Server, it is difficult to provide an
+authoritative guide to developing a driver. It is however possible to describe the core communication protocal
+using the standard out-of-the-box configuration which should provide enough information to develop a driver for a
+specific language.
+
+image::gremlin-server-flow.png[width=300,float=right]
+
+Gremlin Server is distributed with a configuration that utilizes link:http://en.wikipedia.org/wiki/WebSocket[WebSockets]
+with a custom sub-protocol. Under this configuration, Gremlin Server accepts requests containing a Gremlin script,
+evaluates that script and then streams back the results. The notion of "streaming" is depicted in the diagram to the right.
+
+The diagram shows an incoming request to process the Gremlin script of `g.V`. Gremlin Server evaluates that script,
+getting an `Iterator` of vertices as a result, and steps through each `Vertex` within it. The vertices are batched
+together given the `resultIterationBatchSize` configuration. In this case, that value must be `2` given that each
+"response" contains two vertices. Each response is serialized given the requested serializer type (JSON is likely
+best for non-JVM languages) and written back to the requesting client immediately. Gremlin Server does not wait for
+the entire result to be iterated, before sending back a response. It will send the responses as they are realized.
+
+This approach allows for the processing of large result sets without having to serialize the entire result into memory
+for the response. It places a bit of a burden on the developer of the driver however, because it becomes necessary to
+provide a way to reconstruct the entire result on the client side from all of the individual responses that Gremlin
+Server returns for a single request. Again, this description of Gremlin Server's "flow" is related to the
+out-of-the-box configuration. It is quite possible to construct other flows, that might be more amenable to a
+particular language or style of processing.
+
+To formulate a request to Gremlin Server, a `RequestMessage` needs to be constructed. The `RequestMessage` is a
+generalized representation of a request that carries a set of "standard" values in addition to optional ones that are
+dependent on the operation being performed. A `RequestMessage` has these fields:
+
+[width="100%",cols="3,10",options="header"]
+|=========================================================
+|Key |Description
+|requestId |A link:http://en.wikipedia.org/wiki/Globally_unique_identifier[UUID] representing the unique identification for the request.
+|op |The name of the "operation" to execute based on the available `OpProcessor` configured in the Gremlin Server. To evaluate a script, use `eval`.
+|processor |The name of the `OpProcessor` to utilize. The default `OpProcessor` for evaluating scripts is unamed and therefore script evaluation purposes, this value can be an empty string.
+|args |A `Map` of arbitrary parameters to pass to Gremlin Server. The requirements for the contents of this `Map` are dependent on the `op` selected.
+|=========================================================
+
+This message can be serialized in any fashion that is supported by Gremlin Server. New serialization methods can
+be plugged in by implementing a `ServiceLoader` enabled `MessageSerializer`, however Gremlin Server provides for
+JSON serialization by default which will be good enough for purposes of most developers building drivers.
+A `RequestMessage` to evaluate a script with variable bindings looks like this in JSON:
+
+[source,js]
+----
+{ "requestId":"1d6d02bd-8e56-421d-9438-3bd6d0079ff1",
+ "op":"eval",
+ "processor":"",
+ "args":{"gremlin":"g.traversal().V(x).out()",
+ "bindings":{"x":1},
+ "language":"gremlin-groovy"}}
+----
+
+The above JSON represents the "body" of the request to send to Gremlin Server. When sending this "body" over
+websockets Gremlin Server can accept a packet frame using a "text" (1) or a "binary" (2) opcode. Using "text"
+is a bit more limited in that Gremlin Server will always process the body of that request as JSON. Generally speaking
+"text" is just for testing purposes.
+
+The preferred method for sending requests to Gremlin Server is to use the "binary" opcode. In this case, a "header"
+will need be sent in addition to to the "body". The "header" basically consists of a "mime type" so that Gremlin
+Server knows how to deserialize the `RequestMessage`. So, the actual byte array sent to Gremlin Server would be
+formatted as follows:
+
+image::gremlin-server-request.png[]
+
+The first byte represents the length of the "mime type" string value that follows. Given the default configuration of
+Gremlin Server, this value should be set to `application/json`. The "payload" represents the JSON message above
+encoded as bytes.
+
+NOTE: Gremlin Server will only accept masked packets as it pertains to websocket packet header construction.
+
+When Gremlin Server receives that request, it will decode it given the "mime type", pass it to the requested
+`OpProcessor` which will execute the `op` defined in the message. In this case, it will evaluate the script
+`g.traversal().V(x).out()` using the `bindings` supplied in the `args` and stream back the results in a series of
+`ResponseMessages`. A `ResponseMessage` looks like this:
+
+[width="100%",cols="3,10",options="header"]
+|=========================================================
+|Key |Description
+|requestId |The identifier of the `RequestMessage` that generated this `ResponseMessage`.
+|status | The `status` contains a `Map` of three keys: `code` which refers to a `ResultCode` that is somewhat analogous to an link:http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html[HTTP status code], `attributes` that represent a `Map` of protocol-level information, and `message` which is just a human-readable `String` usually associated with errors.
+|result | The `result` contains a `Map` of two keys: `data` which refers to the actual data returned from the server (the type of data is determined by the operation requested) and `meta` which is a `Map` of meta-data related to the response.
+|=========================================================
+
+In this case the `ResponseMessage` returned to the client would look something like this:
+
+[source,js]
+----
+{"result":{"data":[{"id": 2,"label": "person","type": "vertex","properties": [
+ {"id": 2, "value": "vadas", "label": "name"},
+ {"id": 3, "value": 27, "label": "age"}]},
+ ], "meta":{}},
+ "requestId":"1d6d02bd-8e56-421d-9438-3bd6d0079ff1",
+ "status":{"code":206,"attributes":{},"message":""}}
+----
+
+Gremlin Server is capable of streaming results such that additional responses will arrive over the websocket until
+the iteration of the result on the server is complete. Each successful incremental message will have a `ResultCode`
+of `206`. Termination of the stream will be marked by a final `200` status code. Note that all messages without a
+`206` represent terminating conditions for a request. The following table details the various status codes that
+Gremlin Server will send:
+
+[width="100%",cols="2,2,9",options="header"]
+|=========================================================
+|Code |Name |Description
+|200 |SUCCESS |The server successfully processed a request to completion - there are no messages remaining in this stream.
+|204 |NO CONTENT |The server processed the request but there is no result to return (e.g. an `Iterator` with no elements).
+|206 |PARTIAL CONTENT |The server successfully returned some content, but there is more in the stream to arrive - wait for a `SUCCESS` to signify the end of the stream.
+|401 |UNAUTHORIZED |The request attempted to access resources that the requesting user did not have access to.
+|407 |AUTHENTICATE |A challenge from the server for the client to authenticate its request.
+|498 |MALFORMED REQUEST | The request message was not properly formatted which means it could not be parsed at all or the "op" code was not recognized such that Gremlin Server could properly route it for processing. Check the message format and retry the request.
+|499 |INVALID REQUEST ARGUMENTS |The request message was parseable, but the arguments supplied in the message were in conflict or incomplete. Check the message format and retry the request.
+|500 |SERVER ERROR |A general server error occurred that prevented the request from being processed.
+|597 |SCRIPT EVALUATION ERROR |The script submitted for processing evaluated in the `ScriptEngine` with errors and could not be processed. Check the script submitted for syntax errors or other problems and then resubmit.
+|598 |SERVER TIMEOUT |The server exceeded one of the timeout settings for the request and could therefore only partially responded or did not respond at all.
+|599 |SERVER SERIALIZATION ERROR |The server was not capable of serializing an object that was returned from the script supplied on the request. Either transform the object into something Gremlin Server can process within the script or install mapper serialization classes to Gremlin Server.
+|=========================================================
+
+`SUCCESS` and `NO CONTENT` messages are terminating messages that indicate that a request was properly handled on the
+server and that there are no additional messages streaming in for that request. When developing a driver, it is
+important to note the slight differences in semantics for these result codes when it comes to sessionless versus
+in-session requests. For a sessionless request, which operates under automatic transaction management, Gremlin Server
+will only send one of these message types after result iteration and transaction `commit()`. In other words, the
+driver could potentially expect to receive a number of "successful" `PARTIAL CONTENT` messages before ultimately
+ending in failure on `commit()`. For in-session requests, the client is responsible for managing the transaction
+and therefore, a first request could receive multiple "success" related messages, only to fail on a future request
+that finally issues the `commit()`.
+
+OpProcessors Arguments
+^^^^^^^^^^^^^^^^^^^^^^
+
+The following sections define a non-exhaustive list of available operations and arguments for embedded `OpProcessors`
+(i.e. ones packaged with Gremlin Server).
+
+Common
+++++++
+
+All `OpProcessor` instances support these arguments.
+
+[width="100%",cols="2,2,9",options="header"]
+|=========================================================
+|Key |Type |Description
+|batchSize |Int |When the result is an iterator this value defines the number of iterations each `ResponseMessage` should contain - overrides the `resultIterationBatchSize` server setting.
+|=========================================================
+
+Standard OpProcessor
+++++++++++++++++++++
+
+The "standard" `OpProcessor` handles requests for the primary function of Gremlin Server - executing Gremlin.
+Requests made to this `OpProcessor` are "sessionless" in the sense that a request must encapsulate the entirety
+of a transaction. There is no state maintained between requests. A transaction is started when the script is first
+evaluated and is committed when the script completes (or rolled back if an error occurred).
+
+[width="100%",cols="3,10a",options="header"]
+|=========================================================
+|Key |Description
+|processor |As this is the default `OpProcessor` this value can be set to an empty string
+|op |[width="100%",cols="3,10",options="header"]
+!=========================================================
+!Key !Description
+!`authentication` !A request that contains the response to a server challenge for authentication.
+!`eval` !Evaluate a Gremlin script provided as a `String`
+!=========================================================
+|=========================================================
+
+'`authentication` operation arguments'
+[width="100%",cols="2,2,9",options="header"]
+|=========================================================
+|Key |Type |Description
+|sasl |byte[] | *Required* The response to the server authentication challenge. This value is dependent on the SASL authentication mechanism required by the server.
+|=========================================================
+
+'`eval` operation arguments'
+[width="100%",cols="2,2,9",options="header"]
+|=========================================================
+|Key |Type |Description
+|gremlin |String | *Required* The Gremlin script to evaluate
+|bindings |Map |A map of key/value pairs to apply as variables in the context of the Gremlin script
+|language |String |The flavor used (e.g. `gremlin-groovy`)
+|aliases |Map |A map of key/value pairs that allow globally bound `Graph` and `TraversalSource` objects to
+be aliased to different variable names for purposes of the current request. The value represents the name the
+global variable and its key represents the new binding name as it will be referenced in the Gremlin query. For
+example, if the Gremlin Server defines two `TraversalSource` instances named `g1` and `g2`, it would be possible
+to send an alias pair with key of "g" and value of "g2" and thus allow the script to refer to "g2" simply as "g".
+|=========================================================
+
+Session OpProcessor
++++++++++++++++++++
+
+The "session" `OpProcessor` handles requests for the primary function of Gremlin Server - executing Gremlin. It is
+like the "standard" `OpProcessor`, but instead maintains state between sessions and leaves all transaction management
+up to the calling client. It is important that clients that open sessions, commit or roll them back, however Gremlin
+Server will try to clean up such things when a session is killed that has been abandoned. It is important to consider
+that a session can only be maintained with a single machine. In the event that multiple Gremlin Server are deployed,
+session state is not shared among them.
+
+[width="100%",cols="3,10a",options="header"]
+|=========================================================
+|Key |Description
+|processor |This value should be set to `session`
+|op |
+[cols="3,10",options="header"]
+!=========================================================
+!Key !Description
+!`authentication` !A request that contains the response to a server challenge for authentication
+!`eval` !Evaluate a Gremlin script provided as a `String`
+!`close` !Close the specified session and rollback any open transactions.
+|=========================================================
+
+'`authentication` operation arguments'
+[width="100%",cols="2,2,9",options="header"]
+|=========================================================
+|Key |Type |Description
+|sasl |byte[] | *Required* The response to the server authentication challenge. This value is dependent on the SASL authentication mechanism required by the server.
+|=========================================================
+
+'`eval` operation arguments'
+[width="100%",options="header"]
+|=========================================================
+|Key |Type |Description
+|gremlin |String | *Required* The Gremlin script to evaluate
+|session |String | *Required* The session identifier for the current session - typically this value should be a UUID (the session will be created if it doesn't exist)
+|bindings |Map |A map of key/value pairs to apply as variables in the context of the Gremlin script
+|language |String |The flavor used (e.g. `gremlin-groovy`)
+|=========================================================
+
+'`close` operation arguments'
+[width="100%",cols="2,2,9",options="header"]
+|=========================================================
+|Key |Type |Description
+|session |String | *Required* The session identifier for the session to close.
+|=========================================================
+
+Authentication
+^^^^^^^^^^^^^^
+
+Gremlin Server supports link:https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer[SASL-based]
+authentication. A SASL implementation provides a series of challenges and responses that a driver must comply with
+in order to authenticate. By default, Gremlin Server only supports the "PLAIN" SASL mechanism, which is a cleartext
+password system. When authentication is enabled, an incoming request is intercepted before it is evaluated by the
+`ScriptEngine`. The request is saved on the server and a `AUTHENTICATE` challenge response (status code `407`) is
+returned to the client.
+
+The client will detect the `AUTHENTICATE` and respond with an `authentication` for the `op` and an `arg` named `sasl`
+that contains the password. The password should be either, an encoded sequence of UTF-8 bytes, delimited by 0
+(US-ASCII NUL), where the form is : `<NUL>username<NUL>password`, or a Base64 encoded string of the former (which
+in this instance would be `AHVzZXJuYW1lAHBhc3N3b3Jk`). Should Gremlin Server be able to authenticate with the
+provided credentials, the server will return the results of the original request as it normally does without
+authentication. If it cannot authenticate given the challenge response from the client, it will return `UNAUTHORIZED`
+(status code `401`).
+
+NOTE: Gremlin Server does not support the "authorization identity" as described in link:https://tools.ietf.org/html/rfc4616[RFC4616].
+
+[[gremlin-plugins]]
+Gremlin Plugins
+---------------
+
+image:gremlin-plugin.png[width=125]
+
+Plugins provide a way to expand the features of Gremlin Console and Gremlin Server. The first step to developing a
+plugin is to implement the `GremlinPlugin` interface:
+
+[source,java]
+----
+include::{basedir}/gremlin-groovy/src/main/java/org/apache/tinkerpop/gremlin/groovy/plugin/GremlinPlugin.java[]
+----
+
+The most simple plugin and the one most commonly implemented will likely be one that just provides a list of classes
+to import to the Gremlin Console. This type of plugin is the easiest way for implementers of the TinkerPop Structure
+and Process APIs to make their implementations available to users. The TinkerGraph implementation has just such a plugin:
+
+[source,java]
+----
+include::{basedir}/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/groovy/plugin/TinkerGraphGremlinPlugin.java[]
+----
+
+Note that the plugin provides a unique name for the plugin which follows a namespaced pattern as _namespace_._plugin-name_
+(e.g. "tinkerpop.hadoop" - "tinkerpop" is the reserved namespace for TinkerPop maintained plugins). To make TinkerGraph
+classes available to the Console, the `PluginAcceptor` is given a `Set` of imports to provide to the plugin host. The
+`PluginAcceptor` essentially behaves as an abstraction to the "host" that is handling the `GremlinPlugin`. `GremlinPlugin`
+implementations maybe hosted by the Console as well as the `ScriptEngine` in Gremlin Server. Obviously, registering
+new commands and other operations that are specific to the Groovy Shell don't make sense there. Write the code for
+the plugin defensively by checking the `GremlinPlugin.env` key in the `PluginAcceptor.environment()` to understand
+which environment the plugin is being used in.
+
+There is one other step to follow to ensure that the `GremlinPlugin` is visible to its hosts. `GremlinPlugin`
+implementations are loaded via link:http://docs.oracle.com/javase/8/docs/api/java/util/ServiceLoader.html[ServiceLoader]
+and therefore need a resource file added to the jar file where the plugin exists. Add a file called
+`org.apache.tinkerpop.gremlin.groovy.plugin.GremlinPlugin` to `META-INF.services`. In the case of the TinkerGraph
+plugin above, that file will have this line in i
<TRUNCATED>
[17/22] incubator-tinkerpop git commit: Fixed up headings that were
out of sync
Posted by sp...@apache.org.
Fixed up headings that were out of sync
This cleans up the warnings asciidoctor was throwing during doc generation.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/7d20789b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/7d20789b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/7d20789b
Branch: refs/heads/master
Commit: 7d20789b87a6d0c635c82c7981c58f39bf75c29a
Parents: 28e0b28
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Fri Nov 20 13:44:40 2015 -0500
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Fri Nov 20 13:44:40 2015 -0500
----------------------------------------------------------------------
docs/src/tutorials/getting-started/index.asciidoc | 14 +++++++-------
1 file changed, 7 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/7d20789b/docs/src/tutorials/getting-started/index.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/tutorials/getting-started/index.asciidoc b/docs/src/tutorials/getting-started/index.asciidoc
index 8787e57..e800c41 100644
--- a/docs/src/tutorials/getting-started/index.asciidoc
+++ b/docs/src/tutorials/getting-started/index.asciidoc
@@ -179,7 +179,7 @@ This model is referred to as a _property graph_ and it provides a flexible and i
data.
Creating a Graph
-^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~
As intuitive as it is to you, it is perhaps more intuitive to Gremlin himself, as vertices, edges and properties make
up the very elements of his existence. It is indeed helpful to think of our friend, Gremlin, moving about a graph when
@@ -209,7 +209,7 @@ parameter supplied, followed by the `Vertex` to which `v1` should be connected.
creating an edge that goes _out_ of `v1` and into `v2` with a label of "created".
Graph Traversal - Staying Simple
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Now that Gremlin knows where the graph data is, you can ask him to get you some data from it by doing a traversal,
which you can think of as executing some link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#_the_graph_process[process]
@@ -298,7 +298,7 @@ vertices to edges and so on. Your ability to string together steps to ask Gremli
on your understanding of these basic concepts.
Graph Traversal - Increasing Complexity
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Armed with the knowledge from the previous section, let's ask Gremlin to perform some more difficult traversal tasks.
There's not much more that can be done with the "baby" graph we had, so let's return to the "modern" toy graph from
@@ -416,7 +416,7 @@ introduce different features of the stack in order to orient you with what it of
identify areas of interest and dig into the details from there.
Why TinkerPop?
-^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~
image:provider-integration.png[float=right,width=350] The goal of TinkerPop, as a Graph Computing Framework, is to make it
easy for developers to create graph applications by providing APIs and tools that simplify their endeavors. One of
@@ -445,7 +445,7 @@ TIP: To maintain an abstraction over `Graph` creation use `GraphFactory.open()`
the documentation for individual `Graph` implementations to learn about the configuration options to provide.
Loading Data
-^^^^^^^^^^^^
+~~~~~~~~~~~~
image:gremlin-to-the-7.png[width=100,float=left] There are many strategies for getting data into your graph. As you are just
getting started, let's look the more simple methods aimed at "smaller" graphs. A "small" graph, in this context, is
@@ -503,7 +503,7 @@ link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#bulkloadervertexprogram[B
provides a generalized method for loading graphs of virtually any size.
Gremlin Server
-^^^^^^^^^^^^^^
+~~~~~~~~~~~~~~
image:gremlin-server-protocol.png[width=325,float=right] link:http://tinkerpop.incubator.apache.org/docs/x.y.z/#gremlin-server[Gremlin Server]
provides a way to remotely execute Gremlin scripts against one or more `Graph` instances hosted within it. It does
@@ -564,7 +564,7 @@ such as Python, Javascript, etc. Gremlin Server therefore represents the method
interact with TinkerPop.
Conclusion
-^^^^^^^^^^
+~~~~~~~~~~
...and that is the end of _The TinkerPop Workout - by Gremlin_. You are hopefully feeling more confident in your
TinkerPop skills and have a good overview of what the stack has to offer, as well as some entry points to further
[21/22] incubator-tinkerpop git commit: Merge remote-tracking branch
'origin/TINKERPOP3-928'
Posted by sp...@apache.org.
Merge remote-tracking branch 'origin/TINKERPOP3-928'
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/bbae082f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/bbae082f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/bbae082f
Branch: refs/heads/master
Commit: bbae082fdcb4b7cdd0e35b6b0385148312e021ce
Parents: 4a995c2 ead508d
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Sun Nov 22 00:57:40 2015 -0500
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Sun Nov 22 00:57:40 2015 -0500
----------------------------------------------------------------------
bin/process-docs.sh | 2 +-
docs/postprocessor/postprocess.sh | 13 +-
docs/preprocessor/preprocess-file.sh | 5 +-
docs/preprocessor/preprocess.sh | 36 +-
docs/src/acknowledgements.asciidoc | 40 -
docs/src/conclusion.asciidoc | 26 -
docs/src/dev/developer/administration.asciidoc | 91 +
docs/src/dev/developer/contributing.asciidoc | 383 +++
docs/src/dev/developer/index.asciidoc | 33 +
docs/src/dev/developer/meetings.asciidoc | 64 +
docs/src/dev/developer/release.asciidoc | 184 ++
docs/src/developer-administration.asciidoc | 91 -
docs/src/developer-contributing.asciidoc | 362 ---
docs/src/developer-meetings.asciidoc | 64 -
docs/src/developer-release.asciidoc | 184 --
docs/src/developer.asciidoc | 33 -
docs/src/gremlin-applications.asciidoc | 1760 -------------
docs/src/implementations.asciidoc | 1764 -------------
docs/src/index.asciidoc | 39 -
docs/src/intro.asciidoc | 397 ---
docs/src/preface.asciidoc | 97 -
docs/src/reference/acknowledgements.asciidoc | 40 +
docs/src/reference/conclusion.asciidoc | 26 +
.../src/reference/gremlin-applications.asciidoc | 1760 +++++++++++++
docs/src/reference/implementations.asciidoc | 1764 +++++++++++++
docs/src/reference/index.asciidoc | 39 +
docs/src/reference/intro.asciidoc | 397 +++
docs/src/reference/preface.asciidoc | 97 +
docs/src/reference/the-graph.asciidoc | 771 ++++++
docs/src/reference/the-graphcomputer.asciidoc | 475 ++++
docs/src/reference/the-traversal.asciidoc | 2378 ++++++++++++++++++
docs/src/the-graph.asciidoc | 771 ------
docs/src/the-graphcomputer.asciidoc | 475 ----
docs/src/the-traversal.asciidoc | 2378 ------------------
docs/src/tutorials-getting-started.asciidoc | 571 -----
.../tutorials/getting-started/index.asciidoc | 571 +++++
.../upgrade-release-3.0.x-incubating.asciidoc | 180 --
.../upgrade-release-3.1.x-incubating.asciidoc | 322 ---
docs/src/upgrade.asciidoc | 39 -
docs/src/upgrade/index.asciidoc | 39 +
.../upgrade/release-3.0.x-incubating.asciidoc | 180 ++
.../upgrade/release-3.1.x-incubating.asciidoc | 322 +++
pom.xml | 87 +-
43 files changed, 9679 insertions(+), 9671 deletions(-)
----------------------------------------------------------------------
[11/22] incubator-tinkerpop git commit: Made subdirectories for
various "books" in the docs.
Posted by sp...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/index.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/index.asciidoc b/docs/src/index.asciidoc
deleted file mode 100644
index 5a01291..0000000
--- a/docs/src/index.asciidoc
+++ /dev/null
@@ -1,39 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-image::apache-tinkerpop-logo.png[width=500]
-
-*x.y.z*
-
-:toc-position: left
-
-include::preface.asciidoc[]
-
-include::intro.asciidoc[]
-
-include::the-graph.asciidoc[]
-
-include::the-traversal.asciidoc[]
-
-include::the-graphcomputer.asciidoc[]
-
-include::gremlin-applications.asciidoc[]
-
-include::implementations.asciidoc[]
-
-include::conclusion.asciidoc[]
-
-include::acknowledgements.asciidoc[]
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/intro.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/intro.asciidoc b/docs/src/intro.asciidoc
deleted file mode 100644
index 2affc01..0000000
--- a/docs/src/intro.asciidoc
+++ /dev/null
@@ -1,397 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-[[intro]]
-Introduction to Graph Computing
-===============================
-
-image::graph-computing.png[width=350]
-
-[source,xml]
-<dependency>
- <groupId>org.apache.tinkerpop</groupId>
- <artifactId>gremlin-core</artifactId>
- <version>x.y.z</version>
-</dependency>
-
-A link:http://en.wikipedia.org/wiki/Graph_(data_structure)[graph] is a data structure composed of vertices (nodes,
-dots) and edges (arcs, lines). When modeling a graph in a computer and applying it to modern data sets and practices,
-the generic mathematically-oriented, binary graph is extended to support both labels and key/value properties. This
-structure is known as a property graph. More formally, it is a directed, binary, attributed multi-graph. An example
-property graph is diagrammed below. This graph example will be used extensively throughout the documentation and is
-called "TinkerPop Classic" as it is the original demo graph distributed with TinkerPop0 back in 2009 (i.e. the good
-ol' days -- it was the best of times and it was the worst of times).
-
-TIP: The TinkerPop graph is available with <<tinkergraph-gremlin,TinkerGraph>> via `TinkerFactory.createModern()`.
-TinkerGraph is the reference implementation of TinkerPop3 and is used in nearly all the examples in this documentation.
-Note that there also exists the classic `TinkerFactory.createClassic()` which is the graph used in TinkerPop2 and does
-not include vertex labels.
-
-[[tinkerpop-modern]]
-.TinkerPop Modern
-image::tinkerpop-modern.png[width=500]
-
-TinkerPop3 is the third incarnation of the TinkerPop graph computing framework. Similar to computing in general, graph
-computing makes a distinction between *structure* (graph) and *process* (traversal). The structure of the graph is the
-data model defined by a vertex/edge/property link:http://en.wikipedia.org/wiki/Network_topology[topology]. The process
-of the graph is the means by which the structure is analyzed. The typical form of graph processing is called a
-link:http://en.wikipedia.org/wiki/Graph_traversal[traversal].
-
-.Primary components of the TinkerPop3 *structure* API
- * `Graph`: maintains a set of vertices and edges, and access to database functions such as transactions.
- * `Element`: maintains a collection of properties and a string label denoting the element type.
- ** `Vertex`: extends Element and maintains a set of incoming and outgoing edges.
- ** `Edge`: extends Element and maintains an incoming and outgoing vertex.
- * `Property<V>`: a string key associated with a `V` value.
- ** `VertexProperty<V>`: a string key associated with a `V` value as well as a collection of `Property<U>` properties (*vertices only*)
-
-.Primary components of the TinkerPop3 *process* API
- * `TraversalSource`: a generator of traversals for a particular graph, link:http://en.wikipedia.org/wiki/Domain-specific_language[domain specific language] (DSL), and execution engine.
- ** `Traversal<S,E>`: a functional data flow process transforming objects of type `S` into object of type `E`.
- *** `GraphTraversal`: a traversal DSL that is oriented towards the semantics of the raw graph (i.e. vertices, edges, etc.).
- * `GraphComputer`: a system that processes the graph in parallel and potentially, distributed over a multi-machine cluster.
- ** `VertexProgram`: code executed at all vertices in a logically parallel manner with intercommunication via message passing.
- ** `MapReduce`: a computations that analyzes all vertices in the graph in parallel and yields a single reduced result.
-
-IMPORTANT: TinkerPop3 is licensed under the popular link:http://www.apache.org/licenses/LICENSE-2.0.html[Apache2]
-free software license. However, note that the underlying graph engine used with TinkerPop3 may have a difference
-license. Thus, be sure to respect the license caveats of the graph system product.
-
-image:tinkerpop-enabled.png[width=135,float=left] When a graph system implements the TinkerPop3 structure and process
-link:http://en.wikipedia.org/wiki/Application_programming_interface[APIs], their technology is considered
-_TinkerPop3-enabled_ and becomes nearly indistinguishable from any other TinkerPop-enabled graph system save for
-their respective time and space complexity. The purpose of this documentation is to describe the structure/process
-dichotomy at length and in doing so, explain how to leverage TinkerPop3 for the sole purpose of graph system-agnostic
-graph computing. Before deep-diving into the various structure/process APIs, a short introductory review of both APIs
-is provided.
-
-NOTE: The TinkerPop3 API rides a fine line between providing concise "query language" method names and respecting
-Java method naming standards. The general convention used throughout TinkerPop3 is that if a method is "user exposed,"
-then a concise name is provided (e.g. `out()`, `path()`, `repeat()`). If the method is primarily for graph systems
-providers, then the standard Java naming convention is followed (e.g. `getNextStep()`, `getSteps()`,
-`getElementComputeKeys()`).
-
-The Graph Structure
--------------------
-
-image:gremlin-standing.png[width=125,float=left] A graph's structure is the topology formed by the explicit references
-between its vertices, edges, and properties. A vertex has incident edges. A vertex is adjacent to another vertex if
-they share an incident edge. A property is attached to an element and an element has a set of properties. A property
-is a key/value pair, where the key is always a character `String`. The graph structure API of TinkerPop3 provides the
-methods necessary to create such a structure. The TinkerPop graph previously diagrammed can be created with the
-following Java8 code. Note that this graph is available as an in-memory TinkerGraph using
-`TinkerFactory.createClassic()`.
-
-[source,java]
-Graph graph = TinkerGraph.open(); <1>
-Vertex marko = graph.addVertex(T.label, "person", T.id, 1, "name", "marko", "age", 29); <2>
-Vertex vadas = graph.addVertex(T.label, "person", T.id, 2, "name", "vadas", "age", 27);
-Vertex lop = graph.addVertex(T.label, "software", T.id, 3, "name", "lop", "lang", "java");
-Vertex josh = graph.addVertex(T.label, "person", T.id, 4, "name", "josh", "age", 32);
-Vertex ripple = graph.addVertex(T.label, "software", T.id, 5, "name", "ripple", "lang", "java");
-Vertex peter = graph.addVertex(T.label, "person", T.id, 6, "name", "peter", "age", 35);
-marko.addEdge("knows", vadas, T.id, 7, "weight", 0.5f); <3>
-marko.addEdge("knows", josh, T.id, 8, "weight", 1.0f);
-marko.addEdge("created", lop, T.id, 9, "weight", 0.4f);
-josh.addEdge("created", ripple, T.id, 10, "weight", 1.0f);
-josh.addEdge("created", lop, T.id, 11, "weight", 0.4f);
-peter.addEdge("created", lop, T.id, 12, "weight", 0.2f);
-
-<1> Create a new in-memory `TinkerGraph` and assign it to the variable `graph`.
-<2> Create a vertex along with a set of key/value pairs with `T.label` being the vertex label and `T.id` being the vertex id.
-<3> Create an edge along with a set of key/value pairs with the edge label being specified as the first argument.
-
-In the above code all the vertices are created first and then their respective edges. There are two "accessor tokens":
-`T.id` and `T.label`. When any of these, along with a set of other key value pairs is provided to
-`Graph.addVertex(Object...)` or `Vertex.addEdge(String,Vertex,Object...)`, the respective element is created along
-with the provided key/value pair properties appended to it.
-
-CAUTION: Many graph systems do not allow the user to specify an element ID and in such cases, an exception is thrown.
-
-NOTE: In TinkerPop3, vertices are allowed a single immutable string label (similar to an edge label). This
-functionality did not exist in TinkerPop2. Likewise, element id's are immutable as they were in TinkerPop2.
-
-Mutating the Graph
-~~~~~~~~~~~~~~~~~~
-
-Below is a sequence of basic graph mutation operations represented in Java8. One of the major differences between
-TinkerPop2 and TinkerPop3 is that in TinkerPop3, the Java convention of using setters and getters has been abandoned
-in favor of a syntax that is more aligned with the syntax of Gremlin-Groovy in TinkerPop2. Given that Gremlin-Java8
-and Gremlin-Groovy are nearly identical due to the inclusion of Java8 lambdas, a big efforts was made to ensure that
-both languages are as similar as possible.
-
-CAUTION: In the code examples presented throughout this documentation, either Gremlin-Java8 or Gremlin-Groovy is used.
-It is possible to determine which derivative of Gremlin is being used by "mousing over" on the code block and see
-either "JAVA" or "GROOVY" pop up in the top right corner of the code block.
-
-image:basic-mutation.png[width=240,float=right]
-[source,java]
-// create a new graph
-Graph graph = TinkerGraph.open();
-// add a software vertex with a name property
-Vertex gremlin = graph.addVertex(T.label, "software",
- "name", "gremlin"); <1>
-// only one vertex should exist
-assert(IteratorUtils.count(graph.vertices()) == 1)
-// no edges should exist as none have been created
-assert(IteratorUtils.count(graph.edges()) == 0)
-// add a new property
-gremlin.property("created",2009) <2>
-// add a new software vertex to the graph
-Vertex blueprints = graph.addVertex(T.label, "software",
- "name", "blueprints"); <3>
-// connect gremlin to blueprints via a dependsOn-edge
-gremlin.addEdge("dependsOn",blueprints); <4>
-// now there are two vertices and one edge
-assert(IteratorUtils.count(graph.vertices()) == 2)
-assert(IteratorUtils.count(graph.edges()) == 1)
-// add a property to blueprints
-blueprints.property("created",2010) <5>
-// remove that property
-blueprints.property("created").remove() <6>
-// connect gremlin to blueprints via encapsulates
-gremlin.addEdge("encapsulates",blueprints) <7>
-assert(IteratorUtils.count(graph.vertices()) == 2)
-assert(IteratorUtils.count(graph.edges()) == 2)
-// removing a vertex removes all its incident edges as well
-blueprints.remove() <8>
-gremlin.remove() <9>
-// the graph is now empty
-assert(IteratorUtils.count(graph.vertices()) == 0)
-assert(IteratorUtils.count(graph.edges()) == 0)
-// tada!
-
-IMPORTANT: image:groovy-logo.png[width=175,float=left] Gremlin-Groovy leverages the
-link:http://groovy.codehaus.org/[Groovy 2.x language] to express Gremlin traversals. One of the major benefits of
-Groovy is the inclusion of a runtime console that makes it easy for developers to practice with the Gremlin language
-and for production users to connect to their graph and execute traversals in an interactive manner. Moreover,
-Gremlin-Groovy provides various syntax simplifications.
-
-TIP: image:gremlin-sugar.png[width=100,float=left] For those wishing to use the Gremlin2 syntax, please see
-<<sugar-plugin,SugarPlugin>>. This plugin provides syntactic sugar at, typically, a runtime cost. It can be loaded
-programmatically via `SugarLoader.load()`. Once loaded, it is possible to do `g.V.out.name` instead of
-`g.V().out().values('name')` as well as a host of other conveniences.
-
-Here is the same code, but using Gremlin-Groovy in the <<gremlin-console,Gremlin Console>>.
-
-[source,groovy]
-----
-$ bin/gremlin.sh
-
- \,,,/
- (o o)
------oOOo-(3)-oOOo-----
-gremlin> graph = TinkerGraph.open()
-==>tinkergraph[vertices:0 edges:0]
-gremlin> gremlin = graph.addVertex(label,'software','name','gremlin')
-==>v[0]
-gremlin> gremlin.property('created',2009)
-==>vp[created->2009]
-gremlin> blueprints = graph.addVertex(label,'software','name','blueprints')
-==>v[3]
-gremlin> gremlin.addEdge('dependsOn',blueprints)
-==>e[5][0-dependsOn->3]
-gremlin> blueprints.property('created',2010)
-==>vp[created->2010]
-gremlin> blueprints.property('created').remove()
-==>null
-gremlin> gremlin.addEdge('encapsulates',blueprints)
-==>e[7][0-encapsulates->3]
-gremlin> blueprints.remove()
-==>null
-gremlin> gremlin.remove()
-==>null
-----
-
-IMPORTANT: TinkerGraph is not a transactional graph. For more information on transaction handling (for those graph
-systems that support them) see the section dedicated to <<transactions,transactions>>.
-
-The Graph Process
------------------
-
-image:gremlin-running.png[width=125,float=left] The primary way in which graphs are processed are via graph
-traversals. The TinkerPop3 process API is focused on allowing users to create graph traversals in a
-syntactically-friendly way over the structures defined in the previous section. A traversal is an algorithmic walk
-across the elements of a graph according to the referential structure explicit within the graph data structure.
-For example: _"What software does vertex 1's friends work on?"_ This English-statement can be represented in the
-following algorithmic/traversal fashion:
-
- . Start at vertex 1.
- . Walk the incident knows-edges to the respective adjacent friend vertices of 1.
- . Move from those friend-vertices to software-vertices via created-edges.
- . Finally, select the name-property value of the current software-vertices.
-
-Traversals in Gremlin are spawned from a `TraversalSource`. The `GraphTraversalSource` is the typical "graph-oriented"
-DSL used throughout the documentation and will most likely be the most used DSL in a TinkerPop application.
-`GraphTraversalSource` provides two traversal methods.
-
- . `GraphTraversalSource.V(Object... ids)`: generates a traversal starting at vertices in the graph (if no ids are provided, all vertices).
- . `GraphTraversalSource.E(Object... ids)`: generates a traversal starting at edges in the graph (if no ids are provided, all edges).
-
-The return type of `V()` and `E()` is a `GraphTraversal`. A GraphTraversal maintains numerous methods that return
-`GraphTraversal`. In this way, a `GraphTraversal` supports function composition. Each method of `GraphTraversal` is
-called a step and each step modulates the results of the previous step in one of five general ways.
-
- . `map`: transform the incoming traverser's object to another object (S → E).
- . `flatMap`: transform the incoming traverser's object to an iterator of other objects (S → E*).
- . `filter`: allow or disallow the traverser from proceeding to the next step (S → S ∪ ∅).
- . `sideEffect`: allow the traverser to proceed unchanged, but yield some computational sideEffect in the process (S ↬ S).
- . `branch`: split the traverser and send each to an arbitrary location in the traversal (S → { S~1~ → E*, ..., S~n~ → E* } → E*).
-
-Nearly every step in GraphTraversal either extends `MapStep`, `FlatMapStep`, `FilterStep`, `SideEffectStep`, or `BranchStep`.
-
-TIP: `GraphTraversal` is a link:http://en.wikipedia.org/wiki/Monoid[monoid] in that it is an algebraic structure
-that has a single binary operation that is associative. The binary operation is function composition (i.e. method
-chaining) and its identity is the step `identity()`. This is related to a
-link:http://en.wikipedia.org/wiki/Monad_(functional_programming)[monad] as popularized by the functional programming
-community.
-
-Given the TinkerPop graph, the following query will return the names of all the people that the marko-vertex knows.
-The following query is demonstrated using Gremlin-Groovy.
-
-[source,groovy]
-----
-$ bin/gremlin.sh
-
- \,,,/
- (o o)
------oOOo-(3)-oOOo-----
-gremlin> graph = TinkerFactory.createModern() // <1>
-==>tinkergraph[vertices:6 edges:6]
-gremlin> g = graph.traversal(standard()) // <2>
-==>graphtraversalsource[tinkergraph[vertices:6 edges:6], standard]
-gremlin> g.V().has('name','marko').out('knows').values('name') // <3>
-==>vadas
-==>josh
-----
-
-<1> Open the toy graph and reference it by the variable `graph`.
-<2> Create a graph traversal source from the graph using the standard, OLTP traversal engine.
-<3> Spawn a traversal off the traversal source that determines the names of the people that the marko-vertex knows.
-
-.The Name of The People That Marko Knows
-image::tinkerpop-classic-ex1.png[width=500]
-
-Or, if the marko-vertex is already realized with a direct reference pointer (i.e. a variable), then the traversal can
-be spawned off that vertex.
-
-[gremlin-groovy,modern]
-----
-marko = g.V().has('name','marko').next() <1>
-g.V(marko).out('knows') <2>
-g.V(marko).out('knows').values('name') <3>
-----
-
-<1> Set the variable `marko` to the the vertex in the graph `g` named "marko".
-<2> Get the vertices that are outgoing adjacent to the marko-vertex via knows-edges.
-<3> Get the names of the marko-vertex's friends.
-
-The Traverser
-~~~~~~~~~~~~~
-
-When a traversal is executed, the source of the traversal is on the left of the expression (e.g. vertex 1), the steps
-are the middle of the traversal (e.g. `out('knows')` and `values('name')`), and the results are "traversal.next()'d"
-out of the right of the traversal (e.g. "vadas" and "josh").
-
-image::traversal-mechanics.png[width=500]
-
-In TinkerPop3, the objects propagating through the traversal are wrapped in a `Traverser<T>`. The traverser concept
-is new to TinkerPop3 and provides the means by which steps remain stateless. A traverser maintains all the metadata
-about the traversal -- e.g., how many times the traverser has gone through a loop, the path history of the traverser,
-the current object being traversed, etc. Traverser metadata may be accessed by a step. A classic example is the
-<<path-step,`path()`>>-step.
-
-[gremlin-groovy,modern]
-----
-g.V(marko).out('knows').values('name').path()
-----
-
-CAUTION: Path calculation is costly in terms of space as an array of previously seen objects is stored in each path
-of the respective traverser. Thus, a traversal strategy analyzes the traversal to determine if path metadata is
-required. If not, then path calculations are turned off.
-
-Another example is the <<repeat-step,`repeat()`>>-step which takes into account the number of times the traverser
-has gone through a particular section of the traversal expression (i.e. a loop).
-
-[gremlin-groovy,modern]
-----
-g.V(marko).repeat(out()).times(2).values('name')
-----
-
-CAUTION: A Traversal's result are never ordered unless explicitly by means of <<order-step,`order()`>>-step. Thus,
-never rely on the iteration order between TinkerPop3 releases and even within a release (as traversal optimizations
-may alter the flow).
-
-On Gremlin Language Variants
-----------------------------
-
-Gremlin is written in Java8. There are various language variants of Gremlin such as Gremlin-Groovy (packaged with
-TinkerPop3), link:https://github.com/mpollmeier/gremlin-scala[Gremlin-Scala], Gremlin-JavaScript, Gremlin-Clojure
-(known as link:https://github.com/clojurewerkz/ogre[Ogre]), etc. It is best to think of Gremlin as a style of graph
-traversing that is not bound to a particular programming language per se. Within a programming language familiar to
-the developer, there is a Gremlin variant that they can use that leverages the idioms of that language. At minimum,
-a programming language providing a Gremlin implementation must support
-link:http://en.wikipedia.org/wiki/Method_chaining[function chaining] (with
-link:http://en.wikipedia.org/wiki/Anonymous_function[lambdas/anonymous functions] being a "nice to have" if the
-variants wishes to offer arbitrary computations beyond the provided Gremlin steps).
-
-Throughout the documentation, the examples provided are primarily written in Gremlin-Groovy. The reason for this is
-the <<gremlin-console,Gremlin Console>> whereby an interactive programming environment exists that does not require
-code compilation. For learning TinkerPop3 and interacting with a live graph system in an ad hoc manner, the Gremlin
-Console is invaluable. However, for developers interested in working with Gremlin-Java, a few Groovy-to-Java patterns
-are presented below.
-
-[source,groovy]
-// Gremlin-Groovy
-g.V().out('knows').values('name') <1>
-g.V().out('knows').map{it.get().value('name') + ' is the friend name'} <2>
-g.V().out('knows').sideEffect(System.out.&println) <3>
-g.V().as('person').out('knows').as('friend').select().by{it.value('name').length()} <4>
-
-[source,java]
-// Gremlin-Java
-g.V().out("knows").values("name") <1>
-g.V().out("knows").map(t -> t.get().value("name") + " is the friend name") <2>
-g.V().out("knows").sideEffect(System.out::println) <3>
-g.V().as("person").out("knows").as("friend").select().by((Function<Vertex, Integer>) v -> v.<String>value("name").length()) <4>
-
-<1> All the non-lambda step chaining is identical in Gremlin-Groovy and Gremlin-Java. However, note that Groovy
-supports `'` strings as well as `"` strings.
-<2> In Groovy, lambdas are called closures and have a different syntax, where Groovy supports the `it` keyword and
-Java doesn't with all parameters requiring naming.
-<3> The syntax for method references differs slightly between link:https://docs.oracle.com/javase/tutorial/java/javaOO/methodreferences.html[Java]
-and link:http://mrhaki.blogspot.de/2009/08/groovy-goodness-turn-methods-into.html[Gremlin-Groovy].
-<4> Groovy is lenient on object typing and Java is not. When the parameter type of the lambda is not known,
-typecasting is required.
-
-Graph System Integration
-------------------------
-
-image:provider-integration.png[width=395,float=right] TinkerPop is a framework composed of various interoperable
-components. At the foundation there is the <<graph,core TinkerPop3 API>> which defines what a `Graph`, `Vertex`,
-`Edge`, etc. are. At minimum a graph system provider must implement the core API. Once implemented, the Gremlin
-<<traversal,traversal language>> is available to the graph system's users. However, the provider can go further and
-develop specific <<traversalstrategy,`TraversalStrategy`>> optimizations that allow the graph system to inspect a
-Gremlin query at runtime and optimize it for its particular implementation (e.g. index lookups, step reordering). If
-the graph system is a graph processor (i.e. provides OLAP capabilities), the system should implement the
-<<graphcomputer,`GraphComputer`>> API. This API defines how messages/traversers are passed between communicating
-workers (i.e. threads and/or machines). Once implemented, the same Gremlin traversals execute against both the graph
-database (OLTP) and the graph processor (OLAP). Note that the Gremlin language interprets the graph in terms of
-vertices and edges -- i.e. Gremlin is a graph-based domain specific language. Users can create their own domain
-specific languages to process the graph in terms of higher-order constructs such as people, companies, and their
-various relationships. Finally, <<gremlin-server,Gremlin Server>> can be leveraged to allow over the wire
-communication with the TinkerPop-enabled graph system. Gremlin Server provides a configurable communication interface
-along with metrics and monitoring capabilities. In total, this is The TinkerPop.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/preface.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/preface.asciidoc b/docs/src/preface.asciidoc
deleted file mode 100644
index 142f935..0000000
--- a/docs/src/preface.asciidoc
+++ /dev/null
@@ -1,97 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-[[preface]]
-TinkerPop3 Documentation
-========================
-
-In the beginning...
-
-TinkerPop0
-----------
-
-Gremlin realized. The more he did so, the more ideas he created. The more ideas he created, the more they related.
-Into a concatenation of that which he accepted wholeheartedly and that which perhaps may ultimately come to be through
-concerted will, a world took form which was seemingly separate from his own realization of it. However, the world
-birthed could not bear its own weight without the logic Gremlin had come to accept -- the logic of left is not right,
-up not down, and west far from east unless one goes the other way. Gremlin's realization required Gremlin's
-realization. Perhaps, the world is simply an idea that he once had -- The TinkerPop.
-
-image::gremlin-logo.png[width=300]
-
-TinkerPop1
-----------
-
-What is The TinkerPop? Where is The TinkerPop? Who is The TinkerPop? When is The TinkerPop?. The more he wonder, the
-more these thoughts blurred into a seeming identity -- distinctions unclear. Unwilling to accept the morass of the
-maze he wandered, Gremlin crafted a collection of machines to help hold the fabric together: Blueprints, Pipes,
-Frames, Furnace, and Rexster. With their help, could he stave off the thought he was not ready to have? Could he hold
-back The TinkerPop by searching for The TinkerPop?
-
- "If I haven't found it, it is not here and now."
-
-image::gremlin-and-friends.png[width=500]
-
-Upon their realization of existence, the machines turned to their link:http://non-aliencreatures.wikia.com/wiki/Machine_Elf[machine elf] creator and asked:
-
- "Why am I what I am?"
-
-Gremlin responded:
-
- "You will help me realize the ultimate realization -- The TinkerPop. The world you find yourself in and the logic
- that allows you to move about it is because of the TinkerPop."
-
-The machines wondered:
-
- "If what is is the TinkerPop, then perhaps we are The TinkerPop and our realization is simply the realization of
- the TinkerPop?"
-
-Would the machines, by their very nature of realizing The TinkerPop, be The TinkerPop? Or, on the same side of the
-coin, do the machines simply provide the scaffolding by which Gremlin's world sustains itself and yielding its
-justification by means of the word "The TinkerPop?" Regardless, it all turns out the same -- The TinkerPop.
-
-TinkerPop2
-----------
-
-Gremlin spoke:
-
- "Please listen to what I have to say. I am no closer to The TinkerPop. However, all along The TinkerPop has
- espoused the form I willed upon it... this is the same form I have willed upon you, my machine friends. Let me
- train you in the ways of my thought such that it can continue indefinitely."
-
-image::tinkerpop-reading.png[width=450]
-
-The machines, simply moving algorithmically through Gremlin's world, endorsed his logic. Gremlin labored to make them
-more efficient, more expressive, better capable of reasoning upon his thoughts. Faster, quickly, now towards the
-world's end, where there would be forever currently, emanatingly engulfing that which is -- The TinkerPop.
-
-TinkerPop3
-----------
-
-image::tinkerpop3-splash.png[width=450]
-
-Gremlin approached The TinkerPop. The closer he got, the more his world dissolved -- west is right, around is
-straight, and form nothing more than nothing. With each step towards The TinkerPop, more and more of all the other
-worlds made possible were laid upon his paradoxed mind. Everything is everything in The TinkerPop, and when the dust
-settled, Gremlin emerged Gremlitron. He realized that all that he realized was just a realization and that all
-realized realizations are just as real. For that is -- The TinkerPop.
-
-image::gremlintron.png[width=400]
-
-NOTE: TinkerPop2 and below made a sharp distinction between the various TinkerPop projects: Blueprints, Pipes,
-Gremlin, Frames, Furnace, and Rexster. With TinkerPop3, all of these projects have been merged and are generally
-known as Gremlin. *Blueprints* -> Gremlin Structure API : *Pipes* -> `GraphTraversal` : *Frames* -> `Traversal` :
-*Furnace* -> `GraphComputer` and `VertexProgram` : *Rexster* -> GremlinServer.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/reference/acknowledgements.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/reference/acknowledgements.asciidoc b/docs/src/reference/acknowledgements.asciidoc
new file mode 100644
index 0000000..76cddec
--- /dev/null
+++ b/docs/src/reference/acknowledgements.asciidoc
@@ -0,0 +1,40 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+[[acknowledgements]]
+Acknowledgements
+================
+
+image:yourkit-logo.png[width=200,float=left] YourKit supports the TinkerPop open source project with its full-featured
+Java Profiler. YourKit, LLC is the creator of innovative and intelligent tools for profiling Java and .NET
+applications. YourKit's leading software products: link:http://www.yourkit.com/java/profiler/index.jsp[YourKit Java Profiler]
+and link:http://www.yourkit.com/.net/profiler/index.jsp[YourKit .NET Profiler]
+
+image:egg-logo.png[width=200,float=left] link:http://incubator.apache.org/projects/tinkerpop.html[Apache TinkerPop] is
+an effort undergoing incubation at link:http://apache.org[The Apache Software Foundation] (ASF), sponsored by the
+Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the
+infrastructure, communications, and decision making process have stabilized in a manner consistent with other
+successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of
+the code, it does indicate that the project has yet to be fully endorsed by the ASF. Apache TinkerPop is distributed
+under the Apache License v2.0.
+
+image:ketrina-tinkerpop3.png[width=150,float=right] link:http://ketrinayim.tumblr.com[Ketrina Yim] -- Designing
+Gremlin and his friends for TinkerPop was one of my first major projects as a freelancer, and it's delightful to
+see them on the Web and all over the documentation! Drawing and tweaking the characters over time is like watching
+them grow up. They've gone from sketches on paper to full-color logos, and from logos to living characters that
+cheerfully greet visitors to the TinkerPop website. And it's been a great time all throughout!
+
+...in the beginning.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/reference/conclusion.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/reference/conclusion.asciidoc b/docs/src/reference/conclusion.asciidoc
new file mode 100644
index 0000000..c5eecf3
--- /dev/null
+++ b/docs/src/reference/conclusion.asciidoc
@@ -0,0 +1,26 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+[[conclusion]]
+Conclusion
+==========
+
+image:tinkerpop-character.png[width=100,float=left] The world that we know, you and me, is but a subset of the world
+that Gremlin has weaved within The TinkerPop. Gremlin has constructed a fully connected graph and only the subset that
+makes logical sense to our traversing thoughts is the fragment we have come to know and have come to see one another
+within. But there are many more out there, within other webs of logics unfathomed. From any thought, every other
+thought, we come to realize that which is -- The TinkerPop.
+
[16/22] incubator-tinkerpop git commit: adjustments in the AsciiDoc
preprocessor in order to make it work using the new directory structure
Posted by sp...@apache.org.
adjustments in the AsciiDoc preprocessor in order to make it work using the new directory structure
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/28e0b280
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/28e0b280
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/28e0b280
Branch: refs/heads/master
Commit: 28e0b280b598264483207693162dc42b23cf8b04
Parents: bc46d64
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Fri Nov 20 19:31:43 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Fri Nov 20 19:31:43 2015 +0100
----------------------------------------------------------------------
docs/preprocessor/preprocess-file.sh | 5 ++--
docs/preprocessor/preprocess.sh | 36 +++++++++++++----------
docs/src/dev/developer/contributing.asciidoc | 9 ++++++
3 files changed, 33 insertions(+), 17 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/28e0b280/docs/preprocessor/preprocess-file.sh
----------------------------------------------------------------------
diff --git a/docs/preprocessor/preprocess-file.sh b/docs/preprocessor/preprocess-file.sh
index b9d7f4e..a8bd3e3 100755
--- a/docs/preprocessor/preprocess-file.sh
+++ b/docs/preprocessor/preprocess-file.sh
@@ -24,8 +24,9 @@ CONSOLE_HOME=$1
AWK_SCRIPTS="${TP_HOME}/docs/preprocessor/awk"
input=$2
-name=`basename ${input}`
-output="${TP_HOME}/target/postprocess-asciidoc/${name}"
+output=`sed 's@/docs/src/@/target/postprocess-asciidoc/@' <<< "${input}"`
+
+mkdir -p `dirname ${output}`
if hash stdbuf 2> /dev/null; then
lb="stdbuf -oL"
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/28e0b280/docs/preprocessor/preprocess.sh
----------------------------------------------------------------------
diff --git a/docs/preprocessor/preprocess.sh b/docs/preprocessor/preprocess.sh
index 5d66c17..62a8b23 100755
--- a/docs/preprocessor/preprocess.sh
+++ b/docs/preprocessor/preprocess.sh
@@ -36,11 +36,6 @@ do
fi
done
-nc -z localhost 2181 || (
- echo "ZooKeeper is not running, be sure to start it before processing the docs."
- exit 1
-)
-
if [ -e /tmp/neo4j ]; then
echo "The directory '/tmp/neo4j' is required by the pre-processor, be sure to delete it before processing the docs."
exit 1
@@ -121,16 +116,27 @@ echo
echo "============================"
echo "+ Processing AsciiDocs +"
echo "============================"
-find "${TP_HOME}/docs/src/" -name "*.asciidoc" |
- xargs -n1 basename |
- xargs -n1 -I {} echo "echo -ne {}' '; (grep -n {} ${TP_HOME}/docs/src/index.asciidoc || echo 0) | cut -d ':' -f1" | /bin/bash | sort -nk2 | cut -d ' ' -f1 |
- xargs -n1 -I {} echo "${TP_HOME}/docs/src/{}" |
- xargs -n1 ${TP_HOME}/docs/preprocessor/preprocess-file.sh "${CONSOLE_HOME}"
-
-ps=(${PIPESTATUS[@]})
-for i in {0..7}; do
- ec=${ps[i]}
- [ ${ec} -eq 0 ] || break
+
+ec=0
+process_subdirs=1
+find "${TP_HOME}/docs/src/" -name index.asciidoc | xargs -n1 dirname | while read subdir ; do
+ if [ ${process_subdirs} -eq 1 ]; then
+ find "${subdir}" -name "*.asciidoc" |
+ xargs -n1 basename |
+ xargs -n1 -I {} echo "echo -ne {}' '; (grep -n {} ${subdir}/index.asciidoc || echo 0) | cut -d ':' -f1" | /bin/bash | sort -nk2 | cut -d ' ' -f1 |
+ xargs -n1 -I {} echo "${subdir}/{}" |
+ xargs -n1 ${TP_HOME}/docs/preprocessor/preprocess-file.sh "${CONSOLE_HOME}"
+
+ ps=(${PIPESTATUS[@]})
+ for i in {0..7}; do
+ ec=${ps[i]}
+ [ ${ec} -eq 0 ] || break
+ done
+
+ if [ ${ec} -ne 0 ]; then
+ process_subdirs=0
+ fi
+ fi
done
tput smam
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/28e0b280/docs/src/dev/developer/contributing.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/dev/developer/contributing.asciidoc b/docs/src/dev/developer/contributing.asciidoc
index 38da30d..deaccfa 100644
--- a/docs/src/dev/developer/contributing.asciidoc
+++ b/docs/src/dev/developer/contributing.asciidoc
@@ -360,3 +360,12 @@ Everything else should be listed and present under the appropriate license secti
There is also a good chance that the TinkerPop binary NOTICE should be updated. Check if the newly added dependency
contains a NOTICE of its own. If so, include that NOTICE in the TinkerPop NOTICE.
+
+[[documentation]]
+Documentation
+~~~~~~~~~~~~~
+
+The TinkerPop docs are splitted into several subdirectories, each having its own index.asciidoc file. If a new AsciiDoc
+file is added, then it should also be included in the index.asciidoc file, otherwise the preprocessor will ignore it.
+Likewise, if a whole new section (subdirectory) is added, it must include an index.asciidoc file in order to be
+recognized by the AsciiDoc preprocessor.
[02/22] incubator-tinkerpop git commit: Made subdirectories for
various "books" in the docs.
Posted by sp...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/upgrade/index.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/upgrade/index.asciidoc b/docs/src/upgrade/index.asciidoc
new file mode 100644
index 0000000..e877e07
--- /dev/null
+++ b/docs/src/upgrade/index.asciidoc
@@ -0,0 +1,39 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+image::apache-tinkerpop-logo.png[width=500]
+
+:toc-position: left
+
+TinkerPop Upgrade Information
+=============================
+
+This document helps users of TinkerPop to understand the changes that come with each software release. It outlines
+new features, how to resolve breaking changes and other information specific to a release. This document is useful
+to end-users who are building applications on TinkerPop, but it is equally useful to TinkerPop providers, who
+build libraries and other systems on the the core APIs and protocols that TinkerPop exposes.
+
+These providers include:
+
+* Graph System Provider
+** Graph Database Provider
+** Graph Processor Provider
+* Graph Driver Provider
+* Graph Language Provider
+
+include::release-3.1.x-incubating.asciidoc[]
+
+include::release-3.0.x-incubating.asciidoc[]
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/upgrade/release-3.0.x-incubating.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/upgrade/release-3.0.x-incubating.asciidoc b/docs/src/upgrade/release-3.0.x-incubating.asciidoc
new file mode 100644
index 0000000..4bb1473
--- /dev/null
+++ b/docs/src/upgrade/release-3.0.x-incubating.asciidoc
@@ -0,0 +1,180 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+
+TinkerPop 3.0.0
+===============
+
+image::https://raw.githubusercontent.com/apache/incubator-tinkerpop/master/docs/static/images/gremlin-hindu.png[width=225]
+
+*A Gremlin Rāga in 7/16 Time*
+
+TinkerPop 3.0.2
+---------------
+
+*Release Date: October 19, 2015*
+
+Please see the link:https://github.com/apache/incubator-tinkerpop/blob/3.0.2-incubating/CHANGELOG.asciidoc#tinkerpop-302-release-date-october-19-2015[changelog] for a complete list of all the modifications that are part of this release.
+
+Upgrading for Users
+~~~~~~~~~~~~~~~~~~~
+
+BulkLoaderVertexProgram (BLVP)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+`BulkLoaderVertexProgram` now supports arbitrary inputs (i addition to `HadoopGraph`, which was already supported in
+version 3.0.1-incubating). It can now also read from any TP3 enabled graph, like `TinkerGraph`
+or `Neo4jGraph`.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-814[TINKERPOP3-319],
+link:http://tinkerpop.incubator.apache.org/docs/3.0.2-incubating/#bulkloadervertexprogram[Reference Documentation - BLVP]
+
+TinkerGraph
+^^^^^^^^^^^
+
+TinkerGraph can now be configured to support persistence, where TinkerGraph will try to load a graph from a specified
+location and calls to `close()` will save the graph data to that location.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-828[TINKERPOP3-828],
+link:http://tinkerpop.incubator.apache.org/docs/3.0.2-incubating/#_configuration[Reference Documentation - TinkerGraph]
+
+Gremlin Driver and Server
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There were a number of fixes to `gremlin-driver` that prevent protocol desynchronization when talking to Gremlin
+Server.
+
+On the Gremlin Server side, Websocket sub-protocol introduces a new "close" operation to explicitly close sessions.
+Prior to this change, sessions were closed in a more passive fashion (i.e. session timeout). There were also so
+bug fixes around the protocol as it pertained to third-party drivers (e.g. python) using JSON for authentication.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-814[TINKERPOP3-814],
+link:https://issues.apache.org/jira/browse/TINKERPOP3-816[TINKERPOP3-816],
+link:https://issues.apache.org/jira/browse/TINKERPOP3-817[TINKERPOP3-817],
+link:https://issues.apache.org/jira/browse/TINKERPOP3-855[TINKERPOP3-855],
+link:https://issues.apache.org/jira/browse/TINKERPOP3-870[TINKERPOP3-870],
+link:https://issues.apache.org/jira/browse/TINKERPOP3-877[TINKERPOP3-877]
+
+Upgrading for Providers
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Graph Driver Providers
+^^^^^^^^^^^^^^^^^^^^^^
+
+Gremlin Server close Operation
+++++++++++++++++++++++++++++++
+
+It is important to note that this feature of the sub-protocol applies to the `SessionOpProcessor` (i.e. for
+session-based requests). Prior to this change, there was no way to explicitly close a session. Sessions would get
+closed by the server after timeout of activity. This new "op" gives drivers the ability to close the session
+explicitly and as needed.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-849[TINKERPOP3-849],
+link:http://tinkerpop.incubator.apache.org/docs/3.0.2-incubating/#_opprocessors_arguments[Reference Documentation - OpProcessor]
+
+TinkerPop 3.0.1
+---------------
+
+*Release Date: September 2, 2015*
+
+Please see the link:https://github.com/apache/incubator-tinkerpop/blob/3.0.1-incubating/CHANGELOG.asciidoc#tinkerpop-301-release-date-september-2-2015[changelog] for a complete list of all the modifications that are part of this release.
+
+Upgrading for Users
+~~~~~~~~~~~~~~~~~~~
+
+Gremlin Server
+^^^^^^^^^^^^^^
+
+Gremlin Server now supports a link:https://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer[SASL-based]
+(Simple Authentication and Security Layer) authentication model and a default `SimpleAuthenticator` which implements
+the `PLAIN` SASL mechanism (i.e. plain text) to authenticate requests. This gives Gremlin Server some basic security
+capabilities, especially when combined with its built-in SSL feature.
+
+There have also been changes in how global variable bindings in Gremlin Server are established via initialization
+scripts. The initialization scripts now allow for a `Map` of values that can be returned from those scripts.
+That `Map` will be used to set global bindings for the server. See this
+link:https://github.com/apache/incubator-tinkerpop/blob/3.0.1-incubating/gremlin-server/scripts/generate-modern.groovy[sample script]
+for an example.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-576[TINKERPOP3-576]
+
+Neo4j
+^^^^^
+
+Problems related to using `:install` to get the Neo4j plugin operating in Gremlin Console on Windows have been
+resolved.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-804[TINKERPOP3-804]
+
+Upgrading for Providers
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Graph System Providers
+^^^^^^^^^^^^^^^^^^^^^^
+
+GraphFactoryClass Annotation
+++++++++++++++++++++++++++++
+
+Providers can consider the use of the new `GraphFactoryClass` annotation to specify the factory class that `GraphFactory` will use to open a new `Graph` instance. This is an optional feature and will generally help implementations that have an interface extending `Graph`. If that is the case, then this annotation can be used in the following fashion:
+
+[source,java]
+----
+@GraphFactory(MyGraphFactory.class)
+public interface MyGraph extends Graph{
+}
+----
+
+`MyGraphFactory` must contain the static `open` method that is normally expected by `GraphFactory`.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-778[TINKERPOP3-778]
+
+GraphProvider.Descriptor Annotation
++++++++++++++++++++++++++++++++++++
+
+There was a change that affected providers who implemented `GraphComputer` related tests such as the `ProcessComputerSuite`. If the provider runs those tests, then edit the `GraphProvider` implementation for those suites to include the `GraphProvider.Descriptor` annotation as follows:
+
+[source,java]
+----
+@GraphProvider.Descriptor(computer = GiraphGraphComputer.class)
+public final class HadoopGiraphGraphProvider extends HadoopGraphProvider {
+
+ public GraphTraversalSource traversal(final Graph graph) {
+ return GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(GiraphGraphComputer.class)).create(graph);
+ }
+}
+----
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-690[TINKERPOP3-690] for more information.
+
+Semantics of Transaction.close()
+++++++++++++++++++++++++++++++++
+
+There were some adjustments to the test suite with respect to how `Transaction.close()` was being validated. For most providers, this will generally mean checking `OptOut` annotations for test renaming problems. The error that occurs when running the test suite should make it apparent that a test name is incorrect in an `OptOut` if there are issues there.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-764[TINKERPOP3-764] for more information.
+
+Graph Driver Providers
+^^^^^^^^^^^^^^^^^^^^^^
+
+Authentication
+++++++++++++++
+
+Gremlin Server now supports SASL-based authentication. By default, Gremlin Server is not configured with
+authentication turned on and authentication is not required, so existing drivers should still work without any
+additional change. Drivers should however consider implementing this feature as it is likely that many users will
+want the security capabilities that it provides.
+
+See: link:http://tinkerpop.incubator.apache.org/docs/3.0.1-incubating/#_authentication[Reference Documentation - Gremlin Server Authentication]
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/upgrade/release-3.1.x-incubating.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/upgrade/release-3.1.x-incubating.asciidoc b/docs/src/upgrade/release-3.1.x-incubating.asciidoc
new file mode 100644
index 0000000..32d23f0
--- /dev/null
+++ b/docs/src/upgrade/release-3.1.x-incubating.asciidoc
@@ -0,0 +1,322 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+
+TinkerPop 3.1.0
+===============
+
+image::https://raw.githubusercontent.com/apache/incubator-tinkerpop/master/docs/static/images/gremlin-gangster.png[width=225]
+
+*A 187 On The Undercover Gremlinz*
+
+TinkerPop 3.1.0
+---------------
+
+*Release Date: November 16, 2015*
+
+Please see the link:https://github.com/apache/incubator-tinkerpop/blob/3.1.0-incubating/CHANGELOG.asciidoc#tinkerpop-310-release-date-november-16-2015[changelog] for a complete list of all the modifications that are part of this release.
+
+Additional upgrade information can be found here:
+
+* <<_tinkerpop_3_0_2,TinkerPop 3.0.2>>
+* <<_tinkerpop_3_0_1,TinkerPop 3.0.1>>
+
+Upgrading for Users
+~~~~~~~~~~~~~~~~~~~
+
+Shading Jackson
+^^^^^^^^^^^^^^^
+
+The Jackson library is now shaded to `gremlin-shaded`, which will allow Jackson to version independently without
+breaking compatibility with dependent libraries or with those who depend on TinkerPop. The downside is that if a
+library depends on TinkerPop and uses the Jackson classes, those classes will no longer exist with the standard
+Jackson package naming. They will have to shifted as follows:
+
+* `org.objenesis` becomes `org.apache.tinkerpop.shaded.objenesis`
+* `com.esotericsoftware.minlog` becomes `org.apache.tinkerpop.shaded.minlog`
+* `com.fasterxml.jackson` becomes `org.apache.tinkerpop.shaded.jackson`
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-835[TINKERPOP3-835]
+
+PartitionStrategy and VertexProperty
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+`PartitionStrategy` now supports partitioning within `VertexProperty`. The `Graph` needs to be able to support
+meta-properties for this feature to work.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-333[TINKERPOP3-333]
+
+Gremlin Server and Epoll
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Gremlin Server provides a configuration option to turn on support for Netty
+link:http://netty.io/wiki/native-transports.html[native transport] on Linux, which has been shown to help improve
+performance.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-901[TINKERPOP3-901]
+
+Rebindings Deprecated
+^^^^^^^^^^^^^^^^^^^^^
+
+The notion of "rebindings" has been deprecated in favor of the term "aliases". Alias is a better and more intuitive
+term than rebindings which should make it easier for newcomers to understand what they are for.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-913[TINKERPOP3-913],
+link:http://tinkerpop.incubator.apache.org/docs/3.1.0-incubating/#_aliases[Reference Documentation - Aliases]
+
+Configurable Driver Channelizer
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The Gremlin Driver now allows the `Channerlizer` to be supplied as a configuration, which means that custom
+implementations may be supplied.
+
+See: https://issues.apache.org/jira/browse/TINKERPOP3-680[TINKERPOP3-680]
+
+GraphSON and Strict Option
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The `GraphMLReader` now has a `strict` option on the `Builder` so that if a data type for a value is invalid in some
+way, GraphMLReader will simply skip that problem value. In that way, it is a bit more forgiving than before especially
+with empty data.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-756[TINKERPOP3-756]
+
+Transaction.close() Default Behavior
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The default behavior of `Transaction.close()` is to rollback the transaction. This is in contrast to previous versions
+where the default behavior was commit. Using rollback as the default should be thought of as a like a safer approach
+to closing where a user must now explicitly call `commit()` to persist their mutations.
+
+See link:https://issues.apache.org/jira/browse/TINKERPOP3-805[TINKERPOP3-805] for more information.
+
+ThreadLocal Transaction Settings
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The `Transaction.onReadWrite()` and `Transaction.onClose()` settings now need to be set for each thread (if another
+behavior than the default is desired). For gremlin-server users that may be changing these settings via scripts.
+If the settings are changed for a sessionless request they will now only apply to that one request. If the settings are
+changed for an in-session request they will now only apply to all future requests made in the scope of that session.
+
+See link:https://issues.apache.org/jira/browse/TINKERPOP3-885[TINKERPOP3-885]
+
+Hadoop-Gremlin
+^^^^^^^^^^^^^^
+
+* Hadoop1 is no longer supported. Hadoop2 is now the only supported Hadoop version in TinkerPop.
+* Spark and Giraph have been split out of Hadoop-Gremlin into their own respective packages (Spark-Gremlin and Giraph-Gremlin).
+* The directory where application jars are stored in HDFS is now `hadoop-gremlin-x.y.z-libs`.
+** This versioning is important so that cross-version TinkerPop use does not cause jar conflicts.
+
+See link:https://issues.apache.org/jira/browse/TINKERPOP3-616
+
+Spark-Gremlin
+^^^^^^^^^^^^^
+
+* Providers that wish to reuse a graphRDD can leverage the new `PersistedInputRDD` and `PersistedOutputRDD`.
+** This allows the graphRDD to avoid serialization into HDFS for reuse. Be sure to enabled persisted `SparkContext` (see documentation).
+
+See link:https://issues.apache.org/jira/browse/TINKERPOP3-868,
+link:https://issues.apache.org/jira/browse/TINKERPOP3-925
+
+TinkerGraph Serialization
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+TinkerGraph is serializable over Gryo, which means that it can shipped over the wire from Gremlin Server. This
+feature can be useful when working with remote subgraphs.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-728[TINKERPOP3-728]
+
+Deprecation in TinkerGraph
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The `public static String` configurations have been renamed. The old `public static` variables have been deprecated.
+If the deprecated variables were being used, then convert to the replacements as soon as possible.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-926[TINKERPOP3-926]
+
+Deprecation in Gremlin-Groovy
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The closure wrappers classes `GFunction`, `GSupplier`, `GConsumer` have been deprecated. In Groovy, a closure can be
+specified using `as Function` and thus, these wrappers are not needed. Also, the `GremlinExecutor.promoteBindings()`
+method which was previously deprecated has been removed.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-879[TINKERPOP3-879],
+link:https://issues.apache.org/jira/browse/TINKERPOP3-897[TINKERPOP3-897]
+
+Gephi Traversal Visualization
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The process for visualizing a traversal has been simplified. There is no longer a need to "name" steps that will
+represent visualization points for Gephi. It is possible to just "configure" a `visualTraversal` in the console:
+
+[source,text]
+gremlin> :remote config visualTraversal graph vg
+
+which creates a special `TraversalSource` from `graph` called `vg`. The traversals created from `vg` can be used
+to `:submit` to Gephi.
+
+See: link:http://tinkerpop.incubator.apache.org/docs/3.1.0-SNAPSHOT/#gephi-plugin[Reference Documentation - Gephi]
+
+Alterations to GraphTraversal
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There were a number of changes to `GraphTraversal`. Many of the changes came by way of deprecation, but some semantics
+have changed as well:
+
+* `ConjunctionStrategy` has been renamed to `ConnectiveStrategy` (no other behaviors changed).
+* `ConjunctionP` has been renamed to `ConnectiveP` (no other behaviors changed).
+* `DedupBijectionStrategy` has been renamed (and made more effective) as `FilterRankingStrategy`.
+* The `GraphTraversal` mutation API has change significantly with all previous methods being supported but deprecated.
+** The general pattern used now is `addE('knows').from(select('a')).to(select('b')).property('weight',1.0)`.
+* The `GraphTraversal` sack API has changed with all previous methods being supported but deprecated.
+** The old `sack(mult,'weight')` is now `sack(mult).by('weight')`.
+* `GroupStep` has been redesigned such that there is now only a key- and value-traversal. No more reduce-traversal.
+** The previous `group()`-methods have been renamed to `groupV3d0()`. To immediately upgrade, rename all your `group()`-calls to `groupV3d0()`.
+** To migrate to the new `group()`-methods, what was `group().by('age').by(outE()).by(sum(local))` is now `group().by('age').by(outE().sum())`.
+* There was a bug in `fold()`, where if a bulked traverser was provided, the traverser was only represented once.
+** This bug fix might cause a breaking change to a user query if the non-bulk behavior was being counted on. If so, used `dedup()` prior to `fold()`.
+* Both `GraphTraversal().mapKeys()` and `GraphTraversal.mapValues()` has been deprecated.
+** Use `select(keys)` and `select(columns)`. However, note that `select()` will not unroll the keys/values. Thus, `mapKeys()` => `select(keys).unfold()`.
+* The data type of `Operator` enums will now always be the highest common data type of the two given numbers, rather than the data type of the first number, as it's been before.
+
+Aliasing Remotes in the Console
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The `:remote` command in Gremlin Console has a new `alias` configuration option. This `alias` option allows
+specification of a set of key/value alias/binding pairs to apply to the remote. In this way, it becomes possible
+to refer to a variable on the server as something other than what it is referred to for purpose of the submitted
+script. For example once a `:remote` is created, this command:
+
+[source,text]
+:remote alias x g
+
+would allow "g" on the server to be referred to as "x".
+
+[source,text]
+:> x.E().label().groupCount()
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-914[TINKERPOP3-914]
+
+Upgrading for Providers
+~~~~~~~~~~~~~~~~~~~~~~~
+
+IMPORTANT: It is recommended that providers also review all the upgrade instructions specified for users. Many of the
+changes there may prove important for the provider's implementation.
+
+All providers should be aware that Jackson is now shaded to `gremlin-shaded` and could represent breaking change if
+there was usage of the dependency by way of TinkerPop, a direct dependency to Jackson may be required on the
+provider's side.
+
+Graph System Providers
+^^^^^^^^^^^^^^^^^^^^^^
+
+GraphStep Alterations
++++++++++++++++++++++
+
+* `GraphStep` is no longer in `sideEffect`-package, but now in `map`-package as traversals support mid-traversal `V()`.
+* Traversals now support mid-traversal `V()`-steps. Graph system providers should ensure that a mid-traversal `V()` can leverage any suitable index.
+
+See link:https://issues.apache.org/jira/browse/TINKERPOP3-762
+
+Decomposition of AbstractTransaction
+++++++++++++++++++++++++++++++++++++
+
+The `AbstractTransaction` class has been abstracted into two different classes supporting two different modes of
+operation: `AbstractThreadLocalTransaction` and `AbstractThreadedTransaction`, where the former should be used when
+supporting `ThreadLocal` transactions and the latter for threaded transactions. Of course, providers may still
+choose to build their own implementation on `AbstractTransaction` itself or simply implement the `Transaction`
+interface.
+
+The `AbstractTransaction` gains the following methods to potentially implement (though default implementations
+are supplied in `AbstractThreadLocalTransaction` and `AbstractThreadedTransaction`):
+
+* `doReadWrite` that should execute the read-write consumer.
+* `doClose` that should execute the close consumer.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-765[TINKERPOP3-765],
+link:https://issues.apache.org/jira/browse/TINKERPOP3-885[TINKERPOP3-885]
+
+Transaction.close() Default Behavior
+++++++++++++++++++++++++++++++++++++
+
+The default behavior for `Transaction.close()` is to rollback the transaction and is enforced by tests, which
+previously asserted the opposite (i.e. commit on close). These tests have been renamed to suite the new semantics:
+
+* `shouldCommitOnCloseByDefault` became `shouldCommitOnCloseWhenConfigured`
+* `shouldRollbackOnCloseWhenConfigured` became `shouldRollbackOnCloseByDefault`
+
+If these tests were referenced in an `OptOut`, then their names should be updated.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-805[TINKERPOP3-805]
+
+Graph Traversal Updates
++++++++++++++++++++++++
+
+There were numerous changes to the `GraphTraversal` API. Nearly all changes are backwards compatible with respective
+"deprecated" annotations. Please review the respective updates specified in the "Graph System Users" section.
+
+* `GraphStep` is no longer in `sideEffect` package. Now in `map` package.
+* Make sure mid-traversal `GraphStep` calls are folding `HasContainers` in for index-lookups.
+* Think about copying `TinkerGraphStepStrategyTest` for your implementation so you know folding is happening correctly.
+
+Element Removal
++++++++++++++++
+
+`Element.Exceptions.elementAlreadyRemoved` has been deprecated and test enforcement for consistency have been removed.
+ Providers are free to deal with deleted elements as they see fit.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-297[TINKERPOP3-297]
+
+VendorOptimizationStrategy Rename
++++++++++++++++++++++++++++++++++
+
+The `VendorOptimizationStrategy` has been renamed to `ProviderOptimizationStrategy`. This renaming is consistent
+with revised terminology for what were formerly referred to as "vendors".
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-876[TINKERPOP3-876]
+
+GraphComputer Updates
++++++++++++++++++++++
+
+`GraphComputer.configure(String key, Object value)` is now a method (with default implementation).
+This allows the user to specify engine-specific parameters to the underlying OLAP system. These parameters are not intended
+to be cross engine supported. Moreover, if there are not parameters that can be altered (beyond the standard `GraphComputer`
+methods), then the provider's `GraphComputer` implementation should simply return and do nothing.
+
+Driver Providers
+^^^^^^^^^^^^^^^^
+
+Aliases Parameter
++++++++++++++++++
+
+The "rebindings" argument to the "standard" `OpProcessor` has been renamed to "aliases". While "rebindings" is still
+supported it is recommended that the upgrade to "aliases" be made as soon as possible as support will be removed in
+the future. Gremlin Server will not accept both parameters at the same time - a request must contain either one
+parameter or the other if either is supplied.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-913[TINKERPOP3-913]
+
+ThreadLocal Transaction Settings
+++++++++++++++++++++++++++++++++
+
+If a driver configures the `Transaction.onReadWrite()` or `Transaction.onClose()` settings, note that these settings no
+longer apply to all future requests. If the settings are changed for a sessionless request they will only apply to
+that one request. If the settings are changed from an in-session request they will only apply to all future requests
+made in the scope of that session.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP3-885[TINKERPOP3-885]
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/pom.xml
----------------------------------------------------------------------
diff --git a/pom.xml b/pom.xml
index 4f29a70..c098598 100644
--- a/pom.xml
+++ b/pom.xml
@@ -698,19 +698,17 @@ limitations under the License.
<goal>process-asciidoc</goal>
</goals>
<configuration>
- <sourceDirectory>${asciidoc.input.dir}</sourceDirectory>
+ <sourceDirectory>${asciidoc.input.dir}/reference</sourceDirectory>
<sourceDocumentName>index.asciidoc</sourceDocumentName>
- <outputDirectory>${htmlsingle.output.dir}</outputDirectory>
+ <outputDirectory>${htmlsingle.output.dir}/reference</outputDirectory>
<backend>html5</backend>
<doctype>book</doctype>
<attributes>
- <imagesdir>images</imagesdir>
+ <imagesdir>../images</imagesdir>
<encoding>UTF-8</encoding>
<toc>true</toc>
<toclevels>3</toclevels>
<toc-position>left</toc-position>
- <!--<iconsdir>images/icons</iconsdir>-->
- <!-- AsciiDoctor CSS3-based theme configuration -->
<stylesdir>${asciidoctor.style.dir}</stylesdir>
<stylesheet>tinkerpop.css</stylesheet>
<source-highlighter>coderay</source-highlighter>
@@ -725,19 +723,17 @@ limitations under the License.
<goal>process-asciidoc</goal>
</goals>
<configuration>
- <sourceDirectory>${asciidoc.input.dir}</sourceDirectory>
- <sourceDocumentName>upgrade.asciidoc</sourceDocumentName>
- <outputDirectory>${htmlsingle.output.dir}</outputDirectory>
+ <sourceDirectory>${asciidoc.input.dir}/upgrade</sourceDirectory>
+ <sourceDocumentName>index.asciidoc</sourceDocumentName>
+ <outputDirectory>${htmlsingle.output.dir}/upgrade</outputDirectory>
<backend>html5</backend>
<doctype>book</doctype>
<attributes>
- <imagesdir>images</imagesdir>
+ <imagesdir>../images</imagesdir>
<encoding>UTF-8</encoding>
<toc>true</toc>
<toclevels>3</toclevels>
<toc-position>left</toc-position>
- <!--<iconsdir>images/icons</iconsdir>-->
- <!-- AsciiDoctor CSS3-based theme configuration -->
<stylesdir>${asciidoctor.style.dir}</stylesdir>
<stylesheet>tinkerpop.css</stylesheet>
<source-highlighter>coderay</source-highlighter>
@@ -752,19 +748,17 @@ limitations under the License.
<goal>process-asciidoc</goal>
</goals>
<configuration>
- <sourceDirectory>${asciidoc.input.dir}</sourceDirectory>
- <sourceDocumentName>developer.asciidoc</sourceDocumentName>
- <outputDirectory>${htmlsingle.output.dir}</outputDirectory>
+ <sourceDirectory>${asciidoc.input.dir}/dev/developer</sourceDirectory>
+ <sourceDocumentName>index.asciidoc</sourceDocumentName>
+ <outputDirectory>${htmlsingle.output.dir}/dev/developer</outputDirectory>
<backend>html5</backend>
<doctype>book</doctype>
<attributes>
- <imagesdir>images</imagesdir>
+ <imagesdir>../../images</imagesdir>
<encoding>UTF-8</encoding>
<toc>true</toc>
<toclevels>3</toclevels>
<toc-position>left</toc-position>
- <!--<iconsdir>images/icons</iconsdir>-->
- <!-- AsciiDoctor CSS3-based theme configuration -->
<stylesdir>${asciidoctor.style.dir}</stylesdir>
<stylesheet>tinkerpop.css</stylesheet>
<source-highlighter>coderay</source-highlighter>
@@ -779,16 +773,14 @@ limitations under the License.
<goal>process-asciidoc</goal>
</goals>
<configuration>
- <sourceDirectory>${asciidoc.input.dir}</sourceDirectory>
- <sourceDocumentName>tutorials-getting-started.asciidoc</sourceDocumentName>
- <outputDirectory>${htmlsingle.output.dir}</outputDirectory>
+ <sourceDirectory>${asciidoc.input.dir}/tutorials/getting-started</sourceDirectory>
+ <sourceDocumentName>index.asciidoc</sourceDocumentName>
+ <outputDirectory>${htmlsingle.output.dir}/tutorials/getting-started</outputDirectory>
<backend>html5</backend>
<doctype>article</doctype>
<attributes>
- <imagesdir>images</imagesdir>
+ <imagesdir>../../images</imagesdir>
<encoding>UTF-8</encoding>
- <!--<iconsdir>images/icons</iconsdir>-->
- <!-- AsciiDoctor CSS3-based theme configuration -->
<stylesdir>${asciidoctor.style.dir}</stylesdir>
<stylesheet>tinkerpop.css</stylesheet>
<source-highlighter>coderay</source-highlighter>
[15/22] incubator-tinkerpop git commit: Made subdirectories for
various "books" in the docs.
Posted by sp...@apache.org.
Made subdirectories for various "books" in the docs.
At this point you can generate docs with --dryRun, but the pre/post processors don't know what to do as the paths don't all point at the root of doc generation anymore. That's the next step.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/bc46d649
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/bc46d649
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/bc46d649
Branch: refs/heads/master
Commit: bc46d64982ba6e517b47baccf88a410f4d407dcc
Parents: 79b0556
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Fri Nov 20 06:53:39 2015 -0500
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Fri Nov 20 06:53:39 2015 -0500
----------------------------------------------------------------------
bin/process-docs.sh | 2 +-
docs/src/acknowledgements.asciidoc | 40 -
docs/src/conclusion.asciidoc | 26 -
docs/src/dev/developer/administration.asciidoc | 91 +
docs/src/dev/developer/contributing.asciidoc | 362 +++
docs/src/dev/developer/index.asciidoc | 33 +
docs/src/dev/developer/meetings.asciidoc | 64 +
docs/src/dev/developer/release.asciidoc | 184 ++
docs/src/developer-administration.asciidoc | 91 -
docs/src/developer-contributing.asciidoc | 362 ---
docs/src/developer-meetings.asciidoc | 64 -
docs/src/developer-release.asciidoc | 184 --
docs/src/developer.asciidoc | 33 -
docs/src/gremlin-applications.asciidoc | 1760 -------------
docs/src/implementations.asciidoc | 1764 -------------
docs/src/index.asciidoc | 39 -
docs/src/intro.asciidoc | 397 ---
docs/src/preface.asciidoc | 97 -
docs/src/reference/acknowledgements.asciidoc | 40 +
docs/src/reference/conclusion.asciidoc | 26 +
.../src/reference/gremlin-applications.asciidoc | 1760 +++++++++++++
docs/src/reference/implementations.asciidoc | 1764 +++++++++++++
docs/src/reference/index.asciidoc | 39 +
docs/src/reference/intro.asciidoc | 397 +++
docs/src/reference/preface.asciidoc | 97 +
docs/src/reference/the-graph.asciidoc | 771 ++++++
docs/src/reference/the-graphcomputer.asciidoc | 475 ++++
docs/src/reference/the-traversal.asciidoc | 2378 ++++++++++++++++++
docs/src/the-graph.asciidoc | 771 ------
docs/src/the-graphcomputer.asciidoc | 475 ----
docs/src/the-traversal.asciidoc | 2378 ------------------
docs/src/tutorials-getting-started.asciidoc | 571 -----
.../tutorials/getting-started/index.asciidoc | 571 +++++
.../upgrade-release-3.0.x-incubating.asciidoc | 180 --
.../upgrade-release-3.1.x-incubating.asciidoc | 322 ---
docs/src/upgrade.asciidoc | 39 -
docs/src/upgrade/index.asciidoc | 39 +
.../upgrade/release-3.0.x-incubating.asciidoc | 180 ++
.../upgrade/release-3.1.x-incubating.asciidoc | 322 +++
pom.xml | 38 +-
40 files changed, 9609 insertions(+), 9617 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/bin/process-docs.sh
----------------------------------------------------------------------
diff --git a/bin/process-docs.sh b/bin/process-docs.sh
index 4710317..dd1dc2a 100755
--- a/bin/process-docs.sh
+++ b/bin/process-docs.sh
@@ -24,7 +24,7 @@ if [ "$1" == "--dryRun" ]; then
mkdir -p target/postprocess-asciidoc/tmp
cp -R docs/{static,stylesheets} target/postprocess-asciidoc/
- cp docs/src/*.asciidoc target/postprocess-asciidoc/
+ cp -R docs/src/. target/postprocess-asciidoc/
ec=$?
else
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/acknowledgements.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/acknowledgements.asciidoc b/docs/src/acknowledgements.asciidoc
deleted file mode 100644
index 76cddec..0000000
--- a/docs/src/acknowledgements.asciidoc
+++ /dev/null
@@ -1,40 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-[[acknowledgements]]
-Acknowledgements
-================
-
-image:yourkit-logo.png[width=200,float=left] YourKit supports the TinkerPop open source project with its full-featured
-Java Profiler. YourKit, LLC is the creator of innovative and intelligent tools for profiling Java and .NET
-applications. YourKit's leading software products: link:http://www.yourkit.com/java/profiler/index.jsp[YourKit Java Profiler]
-and link:http://www.yourkit.com/.net/profiler/index.jsp[YourKit .NET Profiler]
-
-image:egg-logo.png[width=200,float=left] link:http://incubator.apache.org/projects/tinkerpop.html[Apache TinkerPop] is
-an effort undergoing incubation at link:http://apache.org[The Apache Software Foundation] (ASF), sponsored by the
-Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the
-infrastructure, communications, and decision making process have stabilized in a manner consistent with other
-successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of
-the code, it does indicate that the project has yet to be fully endorsed by the ASF. Apache TinkerPop is distributed
-under the Apache License v2.0.
-
-image:ketrina-tinkerpop3.png[width=150,float=right] link:http://ketrinayim.tumblr.com[Ketrina Yim] -- Designing
-Gremlin and his friends for TinkerPop was one of my first major projects as a freelancer, and it's delightful to
-see them on the Web and all over the documentation! Drawing and tweaking the characters over time is like watching
-them grow up. They've gone from sketches on paper to full-color logos, and from logos to living characters that
-cheerfully greet visitors to the TinkerPop website. And it's been a great time all throughout!
-
-...in the beginning.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/conclusion.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/conclusion.asciidoc b/docs/src/conclusion.asciidoc
deleted file mode 100644
index c5eecf3..0000000
--- a/docs/src/conclusion.asciidoc
+++ /dev/null
@@ -1,26 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-[[conclusion]]
-Conclusion
-==========
-
-image:tinkerpop-character.png[width=100,float=left] The world that we know, you and me, is but a subset of the world
-that Gremlin has weaved within The TinkerPop. Gremlin has constructed a fully connected graph and only the subset that
-makes logical sense to our traversing thoughts is the fragment we have come to know and have come to see one another
-within. But there are many more out there, within other webs of logics unfathomed. From any thought, every other
-thought, we come to realize that which is -- The TinkerPop.
-
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/dev/developer/administration.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/dev/developer/administration.asciidoc b/docs/src/dev/developer/administration.asciidoc
new file mode 100644
index 0000000..fea3749
--- /dev/null
+++ b/docs/src/dev/developer/administration.asciidoc
@@ -0,0 +1,91 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+Administration
+==============
+
+New Committers
+--------------
+
+When a candidate is identified by a PPMC member as someone who might be a good official committer to TinkerPop, the
+PPMC member should open a DISCUSS thread on the private TinkerPop mailing list. The thread should provide some
+background and reasoning for why that member believes the candidate would be a good committer. Given enough time for
+feedback on the candidate and presuming there is still positive interest in doing so, a VOTE thread on the private
+TinkerPop mailing list is started to get the official stance. As per usual, the VOTE will be made open for no less
+than 72 hours.
+
+If the VOTE closes with a successful positive vote to make the candidate a committer, then send the following email
+to the candidate and copy the private TinkerPop mailing list:
+
+[source,text]
+----
+SUBJECT: Invitation to become TinkerPop committer: [candidate name]
+
+Hello,
+
+The TinkerPop Podling Project Management Committee (PPMC) hereby offers you committer privileges to the project.
+These privileges are offered on the understanding that you'll use them reasonably and with common sense. We like to
+work on trust rather than unnecessary constraints.
+
+Being a committer enables you to more easily make changes without needing to go through the patch submission process.
+
+Being a committer does not require you to participate any more than you already do. It does tend to make one even more
+committed. You will probably find that you spend more time here.
+
+Of course, you can decline and instead remain as a contributor, participating as you do now.
+
+A. This personal invitation is a chance for you to accept or decline in private. Either way, please let us know in
+reply to the private@tinkerpop.incubator.apache.org address only.
+
+B. If you are accepting, the next step is to register an iCLA with the Apache Software Foundation:
+ 1. Details of the iCLA and the forms are found through this link: http://www.apache.org/licenses/#clas.
+
+ 2. The form (text or PDF version) provides instructions for its completion and return to the Secretary of the ASF.
+
+ 3. When you transmit the completed iCLA, request to notify the Apache TinkerPop and choose a unique Apache id.
+ Look to see if your preferred id is already taken at http://people.apache.org/committer-index.html This will
+ allow the Secretary to notify the PMC when your iCLA has been recorded.
+
+When recording of your iCLA is noticed, you will receive a follow-up message with the next steps for establishing you
+as a committer.
+----
+
+Assuming the individual accepts, the next step is to get their account created. As we are in incubation, we will
+need to contact a mentor for help with this step.
+
+Upon confirming with the new committer that their account is established, send an announcement email to the
+developer and user mailing lists:
+
+[source,text]
+----
+SUBJECT: New Committer: [committer name]
+
+The Podling Project Management Committee (PPMC) for Apache TinkerPop has asked [committer name] to become a committer
+and we are pleased to announce that he has accepted.
+
+[describe the nature of the committers work in the community]
+
+Being a committer enables easier contribution to the project since there is no need to work via the patch submission
+process. This should enable better productivity.
+----
+
+Finally, update the TinkerPop incubator project status page by:
+
+. Adding to the "News" section
+. Adding to the "Committers" section
+
+
+
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/dev/developer/contributing.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/dev/developer/contributing.asciidoc b/docs/src/dev/developer/contributing.asciidoc
new file mode 100644
index 0000000..38da30d
--- /dev/null
+++ b/docs/src/dev/developer/contributing.asciidoc
@@ -0,0 +1,362 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+Contributing
+============
+
+Contributions via GitHub pull requests are gladly accepted from their original author. By submitting any copyrighted
+material via pull request, email, or other means you agree to license the material under the project's open source
+license and warrant that you have the legal authority to do so.
+
+Getting Started
+---------------
+
+New contributors can start development with TinkerPop by first link:https://help.github.com/articles/fork-a-repo/[forking
+then cloning] the Apache TinkerPop link:https://github.com/apache/incubator-tinkerpop[GitHub repository]. Generally
+speaking it is best to tie any work done to an issue in link:https://issues.apache.org/jira/browse/TINKERPOP3[JIRA].
+Either scan through JIRA for an existing open issue to work on or create a new one.
+
+NOTE: For those who are trying to find a place to start to contribute, consider looking at unresolved issues that
+have the "trivial" priority as these issues are specifically set aside as
+link:https://issues.apache.org/jira/issues/?jql=project%20%3D%20TINKERPOP3%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Trivial%20ORDER%20BY%20key%20DESC[low-hanging fruit]
+for newcomers.
+
+After making changes, submit a link:https://help.github.com/articles/using-pull-requests/[pull request] through
+GitHub, where the name of the pull request is prefixed with the JIRA issue number. In this way, the pull request
+and its comments get tied back to the JIRA issue it references.
+
+Before issuing your pull request, please be sure of the following:
+
+. `mvn clean install` works successfully.
+. If the change requires modification to the documentation, which can be found in `docs/src`, please be sure to try to
+generate the docs. If the changes are minimal and do not include code examples, it might be sufficient to test
+generate the docs to validate formatting by just doing `bin/process-docs.sh --dryRun`. If there are code examples,
+please be sure to have Zookeeper and Hadoop running when doing a `bin/process-docs.sh`. The documentation is
+generated to `/target/docs/htmlsingle`.
+. If necessary, run the integration tests. For example, if the changes affect serialization or Gremlin Server/Driver
+operations then running the integration tests assures in addition to unit tests will definitely be necessary. After
+a successful `mvn clean install`, do `mvn verify -DskipIntegrationTests=false -pl gremlin-server`.
+
+Once a pull request is submitted it must go through <<rtc,review>> and will be merged once three TinkerPop committers
+offer positive vote and achieve Apache consensus.
+
+Building and Testing
+--------------------
+
+TinkerPop requires `Java 1.8.0_40+` for proper building and proper operations.
+
+* Build Project: `mvn clean install`
+** Specify specific tests in a TinkerPop Suite to run with the `GREMLIN_TESTS` environment variable, along with the
+Maven project list argument, e.g.:
++
+----
+export GREMLIN_TESTS='org.apache.tinkerpop.gremlin.process.traversal.step.map.PathTest$Traversals,org.apache.tinkerpop.gremlin.process.traversal.PathTest'
+mvn -Dmaven.javadoc.skip=true --projects tinkergraph-gremlin test
+----
+** Clean the `.groovy/grapes/org.apache.tinkerpop` directory on build: `mvn clean install -DcleanGrapes`
+** Turn off "heavy" logging in the "process" tests: `mvn clean install -DargLine="-DmuteTestLogs=true"`
+** The test suite for `neo4j-gremlin` is disabled by default - to turn it on: `mvn clean install -DincludeNeo4j`
+* Regenerate test data (only necessary given changes to IO classes): `mvn clean install -Dio` from `tinkergraph-gremlin` directory
+** If there are changes to the Gryo format, it may be necessary to generate the Grateful Dead dataset from GraphSON (see `IoDataGenerationTest.shouldWriteGratefulDead`)
+* Check license headers are present: `mvn apache-rat:check`
+* Build AsciiDocs (Hadoop and ZooKeeper must be running): `bin/process-docs.sh`
+** Build AsciiDocs (but don't evaluate code blocks): `bin/process-docs.sh --dryRun`
+** Process a single AsciiDoc file: +pass:[docs/preprocessor/preprocess-file.sh `pwd`/gremlin-console/target/apache-gremlin-console-*-standalone `pwd`/docs/src/xyz.asciidoc]+
+* Build JavaDocs: `mvn process-resources -Djavadoc`
+* Check for Apache License headers: `mvn apache-rat:check`
+* Check for newer dependencies: `mvn versions:display-dependency-updates` or `mvn versions:display-plugin-updates`
+* Deploy JavaDocs/AsciiDocs: `bin/publish-docs.sh svn-username`
+* Integration Tests: `mvn verify -DskipIntegrationTests=false`
+** Execute with the `-DincludeNeo4j` option to include transactional tests.
+** Execute with the `-DuseEpoll` option to try to use Netty native transport (works on Linux, but will fallback to Java NIO on other OS).
+* Performance Tests: `mvn verify -DskipPerformanceTests=false`
+
+IDE Setup with Intellij
+-----------------------
+
+This section refers specifically to setup within Intellij. TinkerPop has a module called `gremlin-shaded` which
+contains shaded dependencies for some libraries that are widely used and tend to introduce conflicts. To ensure
+that Intellij properly interprets this module after importing the Maven `pom.xml` perform the following steps:
+
+. Build `gremlin-shaded` from the command line with `mvn clean install`.
+. Right-click on the `gremlin-shaded` module in the project viewer of Intellij and select "Remove module".
+. In the "Maven Projects" Tool window and click the tool button for "Reimport All Maven projects" (go to
+`View | Tool Windows | Maven Projects` on the main menu if this panel is not activated).
+. At this point it should be possible to compile and run the tests within Intellij, but in the worst case, use
+`File | Invalidate Caches/Restart` to ensure that indices properly rebuild.
+
+Note that it maybe be necessary to re-execute these steps if the `gremlin-shaded` `pom.xml` is ever updated.
+
+Developers working on the `neo4j-gremlin` module should enabled the `include-neo4j` Maven profile in Intellij.
+This will ensure that tests will properly execute within the IDE.
+
+If Intellij complains about "duplicate sources" for the Groovy files when attempting to compile/run tests, then
+install the link:http://plugins.jetbrains.com/plugin/7442?pr=idea[GMavenPlus Intellij plugin].
+
+For Committers
+--------------
+
+The guidelines that follow apply to those with commit access to the main repository:
+
+Communication
+~~~~~~~~~~~~~
+
+TinkerPop has a link:http://groups.google.com/group/gremlin-users[user mailing list] and a
+link:http://mail-archives.apache.org/mod_mbox/incubator-tinkerpop-dev/[developer mailing list]. As a committer,
+it is a good idea to join both.
+
+It would also be helpful to join the public link:https://s.apache.org/tinkerpop[TinkerPop HipChat room] for developer
+discussion. This helps contributors to communicate in a more real-time way. Anyone can join as a guest, but for
+regular contributors it may be best to request that an Apache HipChat account be created.
+
+Release Notes
+~~~~~~~~~~~~~
+
+There is a two-pronged approach to maintaining the change log and preparing the release notes.
+
+1. For work that is documented in JIRA, run the release notes report to include all of
+the tickets targeted for a specific release. This report can be included in the
+release announcement.
+
+2. The manual change log (`CHANGELOG.asciidoc`) can be used to highlight large
+changes, describe themes (e.g. "We focused on performance improvements") or to
+give voice to undocumented changes.
+
+Given the dependence on the JIRA report for generating additions to the `CHANGELOG.asciidoc`,
+which uses the title of the issue as the line presented in the release note report, titles should
+be edited prior to release to be useful in that context. In other words, an issue title should
+be understandable as a change in the fewest words possible while still conveying the gist of the
+change.
+
+Changes that break the public APIs should be marked with a "breaking" label and should be
+distinguished from other changes in the release notes.
+
+Branches
+~~~~~~~~
+
+The "master" branch is used for the main line of development and release branches are constructed as needed
+for ongoing maintenance work. If new to the project or are returning to it after some time away, it may be good
+to send an email to the developer mailing list (or ask on HipChat) to find out what the current operating branches
+are.
+
+Other branches may be created for collaborating on features or for RFC's that other developers may want to inspect.
+It is suggested that the JIRA issue ID be used as the prefix, since that triggers certain automation, and it provides a
+way to account for the branch lifecycle, i.e. "Who's branch is this, and can I delete it?"
+
+For branches that are NOT associated with JIRA issues, developers should utilize their Apache ID as
+a branch name prefix. This provides a unique namespace, and also a way to account for the branch lifecycle.
+
+Developers should remove their own branches when they are no longer needed.
+
+Tags
+~~~~
+
+Tags are used for milestones, release candidates, and approved releases. Please refrain from creating arbitrary
+tags, as they produce permanent clutter.
+
+Issue Tracker Conventions
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+TinkerPop uses Apache JIRA as its link:https://issues.apache.org/jira/browse/TINKERPOP3[issue tracker]. JIRA is a
+very robust piece of software with many options and configurations. To simplify usage and ensure consistency across
+issues, the following conventions should be adhered to:
+
+* An issue's "status" should generally be in one of two states: `open` or `closed` (`reopened` is equivalent to `open`
+for our purposes).
+** An `open` issue is newly created, under consideration or otherwise in progress.
+** A `closed` issue is completed for purposes of release (i.e. code, testing, and documentation complete).
+** Issues in a `resolved` state should immediately be evaluated for movement to `closed` - issue become `resolved`
+by those who don't have the permissions to `close`.
+* An issue's "type" should be one of two options: `bug` or `improvement`.
+** A `bug` has a very specific meaning, referring to an error that prevents usage of TinkerPop AND does not have a
+reasonable workaround. Given that definition, a `bug` should generally have very high priority for a fix.
+** Everything else is an `improvement` in the sense that any other work is an enhancement to the current codebase.
+* The "component" should be representative of the primary area of code that it applies to and all issues should have
+this property set.
+* Issues are not assigned "labels" with two exceptions:
+** The "breaking" label which marks an issue as one that is representative of a change in the API that might
+affect users or vendors. This label is important when organizing release notes.
+** The "deprecation" label which is assigned to an issue that is about removing a deprecated portion of the API.
+* The "affects/fix version(s)" fields should be appropriately set, where the "fix version" implies the version on
+which that particular issue will completed.
+* The "priority" field can be arbitrarily applied with one exception. The "trivial" option should be reserved for
+tasks that are "easy" for a potential new contributor to jump into and do not have significant impact to urgently
+required improvements.
+
+Code Style
+~~~~~~~~~~
+
+Contributors should examine the current code base to determine what the code style patterns are and should match their
+style to what is already present. Of specific note however, TinkerPop does not use "import wildcards" - IDEs should
+be adjusted accordingly to not auto-wildcard the imports.
+
+Deprecation
+~~~~~~~~~~~
+
+When possible, committers should avoid direct "breaking" change (e.g. removing a method from a class) and favor
+deprecation. Deprecation should come with sufficient documentation and notice especially when the change involves
+public APIs that might be utilized by users or implemented by vendors:
+
+* Mark the code with the `@Deprecated` annotation.
+* Use javadoc to further document the change with the following content:
+** `@deprecated As of release x.y.z, replaced by {@link SomeOtherClass#someNewMethod()}` - if the method is not
+replaced then the comment can simply read "not replaced". Additional comments that provide more context are
+encouraged.
+** `@see <a href="https://issues.apache.org/jira/browse/TINKERPOP3-XXX">TINKERPOP3-XXX</a>` - supply a link to the
+JIRA issue for reference.
+* All deprecation should typically be tied to a JIRA issue with a "breaking" label - the issue itself does not need to
+specifically or solely be about "deprecation" but it should be documented very clearly in the comments what was
+deprecated and what the path forward should be.
+* Be sure that deprecated methods are still under test - consider using javadoc/comments in the tests themselves to
+call out this fact.
+* Create a new JIRA issue to track removal of the deprecation for future evaluation - this issue should have the
+"breaking" label as well as a "deprecation" label.
+* Update the "upgrade documentation" to reflect the API change and how the reader should resolve it.
+
+The JIRA issues that track removal of deprecated methods should be periodically evaluated to determine if it is
+prudent to schedule them into a release.
+
+Gremlin Language Test Cases
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When writing a test case for a Gremlin step, be sure to use the following conventions.
+
+* The name of the traversal generator should start with `get`, use `X` for brackets, `_` for space, and the Gremlin-Groovy sugar syntax.
+** `get_g_V_hasLabelXpersonX_groupXaX_byXageX_byXsumX_name()`
+* When creating a test for a step that has both a barrier and sideEffect form (e.g. `group()`, `groupCount()`, etc.), test both representations.
+** `get_g_V_groupCount_byXnameX()`
+** `get_g_V_groupCountXaX_byXnameX_capXaX()`
+* The name of the actual test case should be the name of the traversal generator minus the `get_` prefix.
+* The Gremlin-Groovy version of the test should use the sugar syntax in order to test sugar (as Gremlin-Java8 tests test standard syntax).
+** `g.V.age.sum`
+* Avoid using lambdas in the test case unless that is explicitly what is being tested as OLAP systems will typically not be able to execute those tests.
+* `AbstractGremlinProcessTest` has various static methods to make writing a test case easy.
+** `checkResults(Arrays.asList("marko","josh"), traversal)`
+** `checkMap(new HashMap<String,Long>() {{ put("marko",1l); }}, traversal.next())`
+
+[[rtc]]
+Review then Commit
+~~~~~~~~~~~~~~~~~~
+
+Code modifications must go through a link:http://www.apache.org/foundation/glossary.html#ReviewThenCommit[review-then-committ] (RTC)
+process before being merged into a release branch. All committers should follow the pattern below, where "you" refers to the
+committer wanting to put code into a release branch.
+
+* Make a JIRA ticket for the software problem you want to solve (i.e. a fix).
+* Fork the release branch that the fix will be put into.
+** The branch name should be the JIRA issue identifier (e.g. `TINKERPOP3-XXX`).
+* Develop your fix in your branch.
+* When your fix is complete and ready to merge, issue a link:https://git-scm.com/docs/git-request-pull[pull request].
+** Be certain that the test suite is passing.
+** If you updated documentation, be sure that the `process-docs.sh` is building the documentation correctly.
+* Before you can merge your branch into the release branch, you must have at least 3 +1 link:http://www.apache.org/foundation/glossary.html#ConsensusApproval[consensus votes].
+** Please see the Apache Software Foundations regulations regarding link:http://www.apache.org/foundation/voting.html#votes-on-code-modification[Voting on Code Modifications].
+* Votes are issued by TinkerPop committers as comments to the pull request.
+* Once 3 +1 votes are received, you are responsible for merging to the release branch and handling any merge conflicts.
+** If there is a higher version release branch that requires your fix (e.g. `3.y-1.z` fix going to a `3.y.z` release), be sure to merge to that release branch as well.
+* Be conscious of deleting your branch if it is no longer going to be used so stale branches don't pollute the repository.
+
+NOTE: These steps also generally apply to external pull requests from those who are not official Apache committers. In
+this case, the person responsible for the merge after voting is typically the first person available
+who is knowledgeable in the area that the pull request affects. Any additional coordination on merging can be handled
+via the pull request comment system.
+
+The following exceptions to the RTC (review-then-commit) model presented above are itemized below. It is up to the
+committer to self-regulate as the itemization below is not complete and only hints at the types of commits that do not
+require a review.
+
+* You are responsible for a release and need to manipulate files accordingly for the release.
+** `Gremlin.version()`, CHANGELOG dates, `pom.xml` version bumps, etc.
+* You are doing an minor change and it is obvious that an RTC is not required (would be a pointless burden to the community).
+** The fix is under the link:http://www.apache.org/foundation/glossary.html#CommitThenReview[commit-then-review] (CTR) policy and lazy consensus is sufficient, where a single -1 vote requires you to revert your changes.
+** Adding a test case, fixing spelling/grammar mistakes in the documentation, fixing LICENSE/NOTICE/etc. files, fixing a minor issue in an already merged branch.
+
+When the committer chooses CTR, it is considered good form to include something in the commit message that explains
+that CTR was invoked and the reason for doing so. For example, "Invoking CTR as this change encompasses minor
+adjustments to text formatting."
+
+Pull Request Format
+^^^^^^^^^^^^^^^^^^^
+
+When you submit a pull request, be sure it uses the following style.
+
+* The title of the pull request is the JIRA ticket number + "colon" + the title of the JIRA ticket.
+* The first line of the pull request message should contain a link to the JIRA ticket.
+* Discuss what you did to solve the problem articulated in the JIRA ticket.
+* Discuss any "extra" work done that go beyond the assumed requirements of the JIRA ticket.
+* Be sure to explain what you did to prove that the issue is resolved.
+** Test cases written.
+** Integration tests run (if required for the work accomplished).
+** Documentation building (if required for the work accomplished).
+** Any manual testing (though this should be embodied in a test case).
+* Notes about what you will do when you merge to the respective release branch (e.g. update CHANGELOG).
+** These types of "on merge tweaks" are typically done to extremely dynamic files to combat and merge conflicts.
+* If you are a TinkerPop committer, you can VOTE on your own pull request, so please do so.
+
+[[dependencies]]
+Dependencies
+~~~~~~~~~~~~
+
+There are many dependencies on other open source libraries in TinkerPop modules. When adding dependencies or
+altering the version of a dependency, developers must consider the implications that may apply to the TinkerPop
+LICENSE and NOTICE files. There are two implications to consider:
+
+. Does the dependency fit an Apache _approved_ license?
+. Given the addition or modification to a dependency, does it mean any change for TinkerPop LICENSE and NOTICE files?
+
+Understanding these implications is important for insuring that TinkerPop stays compliant with the Apache 2 license
+that it releases under.
+
+Regarding the first item, refer to the Apache Legal for a list of link:http://www.apache.org/legal/resolved.html[approved licenses]
+that are compatible with the Apache 2 license.
+
+The second item requires a bit more effort to follow. The Apache website offers a
+link:http://www.apache.org/dev/licensing-howto.html[how-to guide] on the approach to maintaining appropriate LICENSE
+and NOTICE files, but this guide is designed to offer some more specific guidance as it pertains to TinkerPop
+and its distribution.
+
+To get started, TinkerPop has both "source" and "binary" LICENSE/NOTICE files:
+
+* Source LICENSE/NOTICE relate to files packaged with the released source code distribution:
+link:https://github.com/apache/incubator-tinkerpop/blob/master/LICENSE[LICENSE] / link:https://github.com/apache/incubator-tinkerpop/blob/master/NOTICE[NOTICE]
+* Binary LICENSE/NOTICE relate to files packaged with the released binary distributions:
+** Gremlin Console link:https://github.com/apache/incubator-tinkerpop/blob/master/gremlin-console/src/main/LICENSE[LICENSE]
+/ link:https://github.com/apache/incubator-tinkerpop/blob/master/gremlin-console/src/main/NOTICE[NOTICE]
+** Gremlin Server link:https://github.com/apache/incubator-tinkerpop/blob/master/gremlin-server/src/main/LICENSE[LICENSE]
+/ link:https://github.com/apache/incubator-tinkerpop/blob/master/gremlin-server/src/main/NOTICE[NOTICE]
+
+As we don't usually include any dependencies in the source distribution (i.e. the source zip distribution), there is
+typically no need to edit source LICENSE/NOTICE when editing a TinkerPop `pom.xml`. These files only need to be edited
+if the distribution has a file added to it. Such a situation may arise from several scenarios, but it would most
+likely come from the addition of a source file from another library. In that case, edit LICENSE to include this file
+following the pattern already present there. Refer to the
+link:http://www.apache.org/dev/licensing-howto.html#mod-notice[Modifications to Notice] section from the Apache
+"how-to" to determine if changes are necessary there.
+
+The binary LICENSE/NOTICE is perhaps most impacted by changes to the various `pom.xml` files. After altering the
+`pom.xml` file of any module, build both Gremlin Console and Gremlin Server and examine the contents of both binary
+distributions, either:
+
+* target/apache-gremlin-console-x.y.z-distribution.zip
+* target/apache-gremlin-server-x.y.z-distribution.zip
+
+There is a _near_ one-to-one mapping between the jar files in those distributions and their respective LICENSE files.
+It is described as "near" because the distributions include TinkerPop jars which don't need to be represented there
+and there are entries in LICENSE that refer to libraries that are present, but shaded in the gremlin-shaded module.
+Everything else should be listed and present under the appropriate license section.
+
+There is also a good chance that the TinkerPop binary NOTICE should be updated. Check if the newly added dependency
+contains a NOTICE of its own. If so, include that NOTICE in the TinkerPop NOTICE.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/dev/developer/index.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/dev/developer/index.asciidoc b/docs/src/dev/developer/index.asciidoc
new file mode 100644
index 0000000..a9e3a94
--- /dev/null
+++ b/docs/src/dev/developer/index.asciidoc
@@ -0,0 +1,33 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+image::apache-tinkerpop-logo.png[width=500]
+
+:toc-position: left
+
+Developer Documentation
+=======================
+
+This document contains information for TinkerPop developers, contributors, and community members. It focuses on
+technical information and other internal processes related to the project.
+
+include::contributing.asciidoc[]
+
+include::release.asciidoc[]
+
+include::administration.asciidoc[]
+
+include::meetings.asciidoc[]
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/dev/developer/meetings.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/dev/developer/meetings.asciidoc b/docs/src/dev/developer/meetings.asciidoc
new file mode 100644
index 0000000..e381707
--- /dev/null
+++ b/docs/src/dev/developer/meetings.asciidoc
@@ -0,0 +1,64 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+Meetings
+========
+
+October 29, 2015
+----------------
+
+The meeting was scheduled for 1:00pm EST, started on time and was held via Google Hangout hosted by Stephen Mallette.
+Meeting adjourned at 1:45pm EST.
+
+*Committers/PPMC Members*
+
+* Stephen Mallette
+* Marko Rodriguez
+* Daniel Kuppitz
+
+*Guests*
+
+_None_
+
+*Minutes*
+
+* Reviewed the scope of 3.1.0-incubating in JIRA in the context of the upcoming release date on November 16, 2015.
+* It was noted that with the new one week code freeze policy that the real cut-off date for work is November 9, 2015.
+* There was general consensus on the call that work should be accomplished in such a way that the code review process
+not drag into the code freeze time period. In other words, pull requests to the release branch should be completed
+well ahead of the 9th.
+
+Upon review of the tickets in JIRA assigned to 3.1.0-incubating, the following were removed from the 3.1.0-incubating
+roadmap:
+
+[width="100%",cols="2,5,5",options="header"]
+|=========================================================
+|ID |Description |Removal Reasoning
+|TINKERPOP3-891 |Re-examine Sandboxing Abstractions |Low priority and time limitations.
+|TINKERPOP3-890 |Remove the concept of branch/ package. |Awaiting step library definition in 3.2.0-incubating.
+|TINKERPOP3-873 |FoldStep should default to using BulkSet for efficiency. |Awaiting step library definition in 3.2.0-incubating.
+|TINKERPOP3-864 |Operator.mean would be nice. |Awaiting step library definition in 3.2.0-incubating.
+|TINKERPOP3-862 |Add a TraversalSourceStrategy that provides "locked" values. |Low priority and time limitations.
+|TINKERPOP3-854 |Remove PropertyMapStep in favor of multi-instruction construction. |Non-trivial given time limitations.
+|TINKERPOP3-789 |Choose then Enforce Semantics for Graph.close() |Non-trivial given time limitations.
+|TINKERPOP3-768 |MatchStep in OLAP should be smart about current vertex. |Non-trivial given time limitations.
+|TINKERPOP3-705 |Make use of a MemoryRDD instead of Accumulators in SparkGraphComputer? |Non-trivial given time limitations.
+|TINKERPOP3-659 |GraphConfiguration Class |Non-trivial given time limitations.
+|TINKERPOP3-623 |More output for OLAP jobs |Not clearly defined given time limitations.
+|TINKERPOP3-620 |Commutative Step Marker interface |Awaiting step library definition in 3.2.0-incubating
+|TINKERPOP3-550 |Gremlin IO needs to support both OLTP and OLAP naturally. |Not clearly defined given time limitations.
+|TINKERPOP3-479 |Consider Providing "getOrCreate" Functionality |Not clearly defined given time limitations.
+|=========================================================
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/dev/developer/release.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/dev/developer/release.asciidoc b/docs/src/dev/developer/release.asciidoc
new file mode 100644
index 0000000..a5ddcc8
--- /dev/null
+++ b/docs/src/dev/developer/release.asciidoc
@@ -0,0 +1,184 @@
+////
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+////
+Release Process
+===============
+
+This document describes the steps required to release a version of TinkerPop. The release is handled by a "release
+manager" (a committer fulfills this role), who ensures that the steps in this document are executed. The process is
+multi-phased and can therefore take several weeks to complete given the time needed for Apache voting and community
+feedback. Once a release point has been identified, the following phases represent the flow of "release":
+
+* Pre-flight check.
+* Optionally, produce a release candidate for community feedback.
+* Submit the official release for PMC vote.
+* Submit the official release for Incubator vote.
+* Release and promote.
+
+NOTE: It might be helpful to use this document as generated from the currently release as opposed to one generate
+from a previous version or from recent `SNAPSHOT`. When using one generated for release, all the "versions" in the
+commands end up being set to the version that is being released, making cut and paste of those commands less labor
+intensive and error prone.
+
+Pre-flight Check
+----------------
+
+The "pre-flight check" is a list of things performed by the release manager during the weeks leading up to a scheduled
+day to release. These checks will help to ensure that that release day goes smoothly by identifying problems up early
+and communicating with other members of the community.
+
+. Fourteen days before release, issue an email to the dev mailing list to remind the community of the pending release.
+.. Note any important issues open in JIRA in that post.
+.. Request review and update of the "upgrade documentation" and CHANGELOG.
+. Seven days before release, announce the code freeze on the dev mailing list to remind the community that the branch
+under release is protected. Tweaks to documentation and other odds and ends related to release are still allowed
+during this period.
+. At some point during the week:
+.. Run the full integration test suite: `mvn clean install -DskipIntegrationTests=false -DincludeNeo4j`
+.. Deploy a final SNAPSHOT to the snapshot repository.
+.. Review LICENSE and NOTICE files to make sure that no <<dependencies,changes are needed>>.
+.. Review javadoc filters on the "Core API" docs to be sure nothing needs to change.
+.. Review JIRA tickets in the release and ensure that:
+... All tickets categorized by having a "Component" assigned.
+... All tickets are either of type "Bug" or "Enhancement".
+. When all documentation changes are in place, use `bin/publish-docs.sh` to deploy a final `SNAPSHOT` representation
+of the docs and thus validate that there are no issues with the documentation generation process. Request review
+of the published documentation on the dev mailing list.
+
+Release Candidate
+-----------------
+
+A release candidate is an unofficial release that is represented by a tagged version in the Git repository. It is
+offered in cases where there is significant change in a particular version and the potential for upgrades and problems
+might be high.
+
+. `mvn clean install -DincludeNeo4j`
+.. `mvn verify -DskipIntegrationTests=false -DincludeNeo4j`
+.. `mvn verify -DskipPerformanceTests=false`
+. `bin/publish-docs.sh <username>` - note that under a release candidate the documentation is published as SNAPSHOT
+. `mvn versions:set -DnewVersion=x.y.z -DgenerateBackupPoms=false` to update the project files to reference a non-SNAPSHOT version
+. `git diff` and review the updated files (expect all `pom.xml` files and this README)
+. `git commit -a -m "TinkerPop x.y.z release"` and `git push`
+. `git tag -a -m "TinkerPop x.y.z release" x.y.z` and `git push --tags`
+. `mvn clean install -Dmaven.test.skip=true`
+. `mvn versions:set -DnewVersion=x.y.z-SNAPSHOT -DgenerateBackupPoms=false` to go back to SNAPSHOT
+. `git commit -a -m "Returned to x.y.z-SNAPSHOT"` and `git push`
+. Announce the release candidate to `dev` mailing list and await feedback
+. Repeat as required or proceed to the next phase
+
+PMC Vote
+--------
+
+A positive vote for a particular release from the TinkerPop PMC is required to move to the following phase.
+
+. By this point, the testing performed during the code freeze should have validated the release. If however there
+are additional tests to perform that the release manager feels are relevant, they should be performed now. In other
+words, there is no need to rebuild the `SNAPSHOT` yet another time unless there are circumstances that would call its
+validity into question.
+. Update `CHANGELOG.asciidoc`:
+.. Update the release date
+.. Generate the JIRA release notes report for the current version and append them to the `CHANGELOG.asciidoc`.
+... Use an "advanced" search to filter out JIRA issues already released on other versions. For example: `fixVersion
+= 3.1.0-incubating AND fixVersion not in (3.0.2-incubating, 3.0.1-incubating`.
+... Consider use of an "Excel" export to organize, sort and prepare the JIRA tickets to be pasted to `CHANGELOG.asciidoc`
+... Be sure to include a link to other versions in the `CHANGELOG.asciidoc` that were previously released while the
+current release was under development as this new release will have those changes included within it. Please see
+3.1.0-incubating for an example.
+.. Organize "breaking" changes to be clearly marked (use JIRA and the "breaking" label to identify those)
+. Update "upgrade documentation":
+.. Update the release date.
+.. Update the link to CHANGELOG.asciidoc
+. `mvn versions:set -DnewVersion=x.y.z -DgenerateBackupPoms=false` to update project files to reference the non-SNAPSHOT version
+. `git diff` and review the updated files (expect all `pom.xml` files and this README)
+. `git commit -a -m "TinkerPop x.y.z release"` and `git push`
+. `mvn clean install -Dmaven.test.skip=true` - need to build first so that the right version of the console is used with `bin/publish-docs.sh`
+. `bin/process-docs.sh` and validate the generated documentation locally
+. `bin/publish-docs.sh <username>` - Note that this step requires no additional processing as the previous step.
+handled document generation and this step now merely needs to upload what was generated.
+. `mvn deploy -Papache-release -DcreateChecksum=true -Dmaven.test.skip=true` - deploy signed artifacts with checksums to link:https://repository.apache.org/[Apache Nexus]. Review (artifacts versions, file sizes, anything that might be out of place - request another committer to review as well) but do NOT close/release the staging repository at this time.
+. Review generated artifacts to be sure they have both javadocs and asciidocs present
+. Upload artifacts to `https://dist.apache.org/repos/dist/dev/incubator/tinkerpop` for `[VOTE]` review.
+.. `svn co --depth empty https://dist.apache.org/repos/dist/dev/incubator/tinkerpop/ dev` and `mkdir dev/x.y.z`
+.. `cp ~/.m2/repository/org/apache/tinkerpop/gremlin-console/x.y.z/gremlin-console-x.y.z-distribution.zip* dev/x.y.z`
+.. `cp ~/.m2/repository/org/apache/tinkerpop/gremlin-server/x.y.z/gremlin-server-x.y.z-distribution.zip* dev/x.y.z`
+.. `cp ~/.m2/repository/org/apache/tinkerpop/tinkerpop/x.y.z/tinkerpop-x.y.z-source-release.zip* dev/x.y.z`
+.. `cd dev/x.y.z`
+.. pass:[<code>ls * | xargs -n1 -I {} echo "mv apache-{} {}" | sed -e 's/distribution/bin/' -e 's/source-release/src/' -e s'/^\(.*\) \(.*\) \(.*\)$/\1 \3 \2/' | /bin/bash</code>]
+.. `cd ..; svn add x.y.z/; svn ci -m "TinkerPop x.y.z release"`
+. Execute `bin/validate-distribution.sh` and any other relevant testing.
+. `git tag -a -m "TinkerPop x.y.z release" x.y.z` and `git push --tags`
+. Perform JIRA administration tasks:
+.. "Release" the current version and set the "release date"
+.. If there is to be a follow on release in the current line of code, create that new version specifying the "start date"
+. Submit for `[VOTE]` at `dev@tinkerpop.incubator.apache.org` (see email template below)
+. *Wait for vote acceptance* (72 hours)
+
+Incubator Vote
+--------------
+
+A positive vote for a particular release from the Apache Incubator is required to move to the following phase.
+
+. Submit for `[VOTE]` at `general@incubator.apache.org` (see email template below)
+.. Include the vote tally: "Apache TinkerPop (http://tinkerpop.incubator.apache.org/) would like to release TinkerPop x.y.z. We had a dev@ VOTE which resulted in a tally of +1 (3), 0 (0), and -1 (0). We now present our artifacts for vote by Incubator."
+. *Wait for vote acceptance* (72 hours)
+
+Release & Promote
+-----------------
+
+. Close the staging repository at link:https://repository.apache.org/[Apache Nexus]) and then release.
+. `svn co --depth empty https://dist.apache.org/repos/dist/dev/incubator/tinkerpop dev; svn up dev/x.y.z`
+. `svn co --depth empty https://dist.apache.org/repos/dist/release/incubator/tinkerpop release; mkdir release/x.y.z`
+. `cd release; svn add x.y.z/; svn ci -m "TinkerPop x.y.z release"`
+. Update homepage with references to latest distribution and to other internal links elsewhere on the page.
+. Wait for Apache Central to sync the jars and src (link:http://repo1.maven.org/maven2/org/apache/tinkerpop/tinkerpop/[http://repo1.maven.org/maven2/org/apache/tinkerpop/tinkerpop/]).
+. Announce release on `dev@`/`gremlin-users@` mailing lists and tweet from `@apachetinkerpop`
+
+Example `[VOTE]` email:
+
+```
+[VOTE] TinkerPop x.y.z Release
+
+Hello,
+
+The release artifacts can be found at this location:
+ https://dist.apache.org/repos/dist/dev/incubator/tinkerpop/x.y.z/
+
+The source distribution is provided by:
+ apache-tinkerpop-x.y.z-src.zip
+
+Two binary distributions are provided for user convenience:
+ apache-gremlin-console-x.y.z-bin.zip
+ apache-gremlin-server-x.y.z-bin.zip
+
+The online docs can be found here:
+ http://tinkerpop.incubator.apache.org/docs/x.y.z/ (user docs)
+ http://tinkerpop.incubator.apache.org/docs/x.y.z/upgrade.html (upgrade docs)
+ http://tinkerpop.incubator.apache.org/javadocs/x.y.z/core/ (core javadoc)
+ http://tinkerpop.incubator.apache.org/javadocs/x.y.z/full/ (full javadoc)
+
+The tag in Apache Git can be found here:
+ https://git-wip-us.apache.org/repos/asf?p=incubator-tinkerpop.git;...
+
+The release notes are available here:
+ https://github.com/apache/incubator-tinkerpop/blob/master/CHANGELOG.asciidoc#...
+
+The [VOTE] will be open for the next 72 hours --- closing <DayOfTheWeek> (<Month> <Day> <Year>) at <Time> <TimeZone>.
+
+My vote is +1.
+
+Thank you very much,
+<TinkerPop Committer Name>
+```
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/developer-administration.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/developer-administration.asciidoc b/docs/src/developer-administration.asciidoc
deleted file mode 100644
index fea3749..0000000
--- a/docs/src/developer-administration.asciidoc
+++ /dev/null
@@ -1,91 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-Administration
-==============
-
-New Committers
---------------
-
-When a candidate is identified by a PPMC member as someone who might be a good official committer to TinkerPop, the
-PPMC member should open a DISCUSS thread on the private TinkerPop mailing list. The thread should provide some
-background and reasoning for why that member believes the candidate would be a good committer. Given enough time for
-feedback on the candidate and presuming there is still positive interest in doing so, a VOTE thread on the private
-TinkerPop mailing list is started to get the official stance. As per usual, the VOTE will be made open for no less
-than 72 hours.
-
-If the VOTE closes with a successful positive vote to make the candidate a committer, then send the following email
-to the candidate and copy the private TinkerPop mailing list:
-
-[source,text]
-----
-SUBJECT: Invitation to become TinkerPop committer: [candidate name]
-
-Hello,
-
-The TinkerPop Podling Project Management Committee (PPMC) hereby offers you committer privileges to the project.
-These privileges are offered on the understanding that you'll use them reasonably and with common sense. We like to
-work on trust rather than unnecessary constraints.
-
-Being a committer enables you to more easily make changes without needing to go through the patch submission process.
-
-Being a committer does not require you to participate any more than you already do. It does tend to make one even more
-committed. You will probably find that you spend more time here.
-
-Of course, you can decline and instead remain as a contributor, participating as you do now.
-
-A. This personal invitation is a chance for you to accept or decline in private. Either way, please let us know in
-reply to the private@tinkerpop.incubator.apache.org address only.
-
-B. If you are accepting, the next step is to register an iCLA with the Apache Software Foundation:
- 1. Details of the iCLA and the forms are found through this link: http://www.apache.org/licenses/#clas.
-
- 2. The form (text or PDF version) provides instructions for its completion and return to the Secretary of the ASF.
-
- 3. When you transmit the completed iCLA, request to notify the Apache TinkerPop and choose a unique Apache id.
- Look to see if your preferred id is already taken at http://people.apache.org/committer-index.html This will
- allow the Secretary to notify the PMC when your iCLA has been recorded.
-
-When recording of your iCLA is noticed, you will receive a follow-up message with the next steps for establishing you
-as a committer.
-----
-
-Assuming the individual accepts, the next step is to get their account created. As we are in incubation, we will
-need to contact a mentor for help with this step.
-
-Upon confirming with the new committer that their account is established, send an announcement email to the
-developer and user mailing lists:
-
-[source,text]
-----
-SUBJECT: New Committer: [committer name]
-
-The Podling Project Management Committee (PPMC) for Apache TinkerPop has asked [committer name] to become a committer
-and we are pleased to announce that he has accepted.
-
-[describe the nature of the committers work in the community]
-
-Being a committer enables easier contribution to the project since there is no need to work via the patch submission
-process. This should enable better productivity.
-----
-
-Finally, update the TinkerPop incubator project status page by:
-
-. Adding to the "News" section
-. Adding to the "Committers" section
-
-
-
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/developer-contributing.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/developer-contributing.asciidoc b/docs/src/developer-contributing.asciidoc
deleted file mode 100644
index 38da30d..0000000
--- a/docs/src/developer-contributing.asciidoc
+++ /dev/null
@@ -1,362 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-Contributing
-============
-
-Contributions via GitHub pull requests are gladly accepted from their original author. By submitting any copyrighted
-material via pull request, email, or other means you agree to license the material under the project's open source
-license and warrant that you have the legal authority to do so.
-
-Getting Started
----------------
-
-New contributors can start development with TinkerPop by first link:https://help.github.com/articles/fork-a-repo/[forking
-then cloning] the Apache TinkerPop link:https://github.com/apache/incubator-tinkerpop[GitHub repository]. Generally
-speaking it is best to tie any work done to an issue in link:https://issues.apache.org/jira/browse/TINKERPOP3[JIRA].
-Either scan through JIRA for an existing open issue to work on or create a new one.
-
-NOTE: For those who are trying to find a place to start to contribute, consider looking at unresolved issues that
-have the "trivial" priority as these issues are specifically set aside as
-link:https://issues.apache.org/jira/issues/?jql=project%20%3D%20TINKERPOP3%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Trivial%20ORDER%20BY%20key%20DESC[low-hanging fruit]
-for newcomers.
-
-After making changes, submit a link:https://help.github.com/articles/using-pull-requests/[pull request] through
-GitHub, where the name of the pull request is prefixed with the JIRA issue number. In this way, the pull request
-and its comments get tied back to the JIRA issue it references.
-
-Before issuing your pull request, please be sure of the following:
-
-. `mvn clean install` works successfully.
-. If the change requires modification to the documentation, which can be found in `docs/src`, please be sure to try to
-generate the docs. If the changes are minimal and do not include code examples, it might be sufficient to test
-generate the docs to validate formatting by just doing `bin/process-docs.sh --dryRun`. If there are code examples,
-please be sure to have Zookeeper and Hadoop running when doing a `bin/process-docs.sh`. The documentation is
-generated to `/target/docs/htmlsingle`.
-. If necessary, run the integration tests. For example, if the changes affect serialization or Gremlin Server/Driver
-operations then running the integration tests assures in addition to unit tests will definitely be necessary. After
-a successful `mvn clean install`, do `mvn verify -DskipIntegrationTests=false -pl gremlin-server`.
-
-Once a pull request is submitted it must go through <<rtc,review>> and will be merged once three TinkerPop committers
-offer positive vote and achieve Apache consensus.
-
-Building and Testing
---------------------
-
-TinkerPop requires `Java 1.8.0_40+` for proper building and proper operations.
-
-* Build Project: `mvn clean install`
-** Specify specific tests in a TinkerPop Suite to run with the `GREMLIN_TESTS` environment variable, along with the
-Maven project list argument, e.g.:
-+
-----
-export GREMLIN_TESTS='org.apache.tinkerpop.gremlin.process.traversal.step.map.PathTest$Traversals,org.apache.tinkerpop.gremlin.process.traversal.PathTest'
-mvn -Dmaven.javadoc.skip=true --projects tinkergraph-gremlin test
-----
-** Clean the `.groovy/grapes/org.apache.tinkerpop` directory on build: `mvn clean install -DcleanGrapes`
-** Turn off "heavy" logging in the "process" tests: `mvn clean install -DargLine="-DmuteTestLogs=true"`
-** The test suite for `neo4j-gremlin` is disabled by default - to turn it on: `mvn clean install -DincludeNeo4j`
-* Regenerate test data (only necessary given changes to IO classes): `mvn clean install -Dio` from `tinkergraph-gremlin` directory
-** If there are changes to the Gryo format, it may be necessary to generate the Grateful Dead dataset from GraphSON (see `IoDataGenerationTest.shouldWriteGratefulDead`)
-* Check license headers are present: `mvn apache-rat:check`
-* Build AsciiDocs (Hadoop and ZooKeeper must be running): `bin/process-docs.sh`
-** Build AsciiDocs (but don't evaluate code blocks): `bin/process-docs.sh --dryRun`
-** Process a single AsciiDoc file: +pass:[docs/preprocessor/preprocess-file.sh `pwd`/gremlin-console/target/apache-gremlin-console-*-standalone `pwd`/docs/src/xyz.asciidoc]+
-* Build JavaDocs: `mvn process-resources -Djavadoc`
-* Check for Apache License headers: `mvn apache-rat:check`
-* Check for newer dependencies: `mvn versions:display-dependency-updates` or `mvn versions:display-plugin-updates`
-* Deploy JavaDocs/AsciiDocs: `bin/publish-docs.sh svn-username`
-* Integration Tests: `mvn verify -DskipIntegrationTests=false`
-** Execute with the `-DincludeNeo4j` option to include transactional tests.
-** Execute with the `-DuseEpoll` option to try to use Netty native transport (works on Linux, but will fallback to Java NIO on other OS).
-* Performance Tests: `mvn verify -DskipPerformanceTests=false`
-
-IDE Setup with Intellij
------------------------
-
-This section refers specifically to setup within Intellij. TinkerPop has a module called `gremlin-shaded` which
-contains shaded dependencies for some libraries that are widely used and tend to introduce conflicts. To ensure
-that Intellij properly interprets this module after importing the Maven `pom.xml` perform the following steps:
-
-. Build `gremlin-shaded` from the command line with `mvn clean install`.
-. Right-click on the `gremlin-shaded` module in the project viewer of Intellij and select "Remove module".
-. In the "Maven Projects" Tool window and click the tool button for "Reimport All Maven projects" (go to
-`View | Tool Windows | Maven Projects` on the main menu if this panel is not activated).
-. At this point it should be possible to compile and run the tests within Intellij, but in the worst case, use
-`File | Invalidate Caches/Restart` to ensure that indices properly rebuild.
-
-Note that it maybe be necessary to re-execute these steps if the `gremlin-shaded` `pom.xml` is ever updated.
-
-Developers working on the `neo4j-gremlin` module should enabled the `include-neo4j` Maven profile in Intellij.
-This will ensure that tests will properly execute within the IDE.
-
-If Intellij complains about "duplicate sources" for the Groovy files when attempting to compile/run tests, then
-install the link:http://plugins.jetbrains.com/plugin/7442?pr=idea[GMavenPlus Intellij plugin].
-
-For Committers
---------------
-
-The guidelines that follow apply to those with commit access to the main repository:
-
-Communication
-~~~~~~~~~~~~~
-
-TinkerPop has a link:http://groups.google.com/group/gremlin-users[user mailing list] and a
-link:http://mail-archives.apache.org/mod_mbox/incubator-tinkerpop-dev/[developer mailing list]. As a committer,
-it is a good idea to join both.
-
-It would also be helpful to join the public link:https://s.apache.org/tinkerpop[TinkerPop HipChat room] for developer
-discussion. This helps contributors to communicate in a more real-time way. Anyone can join as a guest, but for
-regular contributors it may be best to request that an Apache HipChat account be created.
-
-Release Notes
-~~~~~~~~~~~~~
-
-There is a two-pronged approach to maintaining the change log and preparing the release notes.
-
-1. For work that is documented in JIRA, run the release notes report to include all of
-the tickets targeted for a specific release. This report can be included in the
-release announcement.
-
-2. The manual change log (`CHANGELOG.asciidoc`) can be used to highlight large
-changes, describe themes (e.g. "We focused on performance improvements") or to
-give voice to undocumented changes.
-
-Given the dependence on the JIRA report for generating additions to the `CHANGELOG.asciidoc`,
-which uses the title of the issue as the line presented in the release note report, titles should
-be edited prior to release to be useful in that context. In other words, an issue title should
-be understandable as a change in the fewest words possible while still conveying the gist of the
-change.
-
-Changes that break the public APIs should be marked with a "breaking" label and should be
-distinguished from other changes in the release notes.
-
-Branches
-~~~~~~~~
-
-The "master" branch is used for the main line of development and release branches are constructed as needed
-for ongoing maintenance work. If new to the project or are returning to it after some time away, it may be good
-to send an email to the developer mailing list (or ask on HipChat) to find out what the current operating branches
-are.
-
-Other branches may be created for collaborating on features or for RFC's that other developers may want to inspect.
-It is suggested that the JIRA issue ID be used as the prefix, since that triggers certain automation, and it provides a
-way to account for the branch lifecycle, i.e. "Who's branch is this, and can I delete it?"
-
-For branches that are NOT associated with JIRA issues, developers should utilize their Apache ID as
-a branch name prefix. This provides a unique namespace, and also a way to account for the branch lifecycle.
-
-Developers should remove their own branches when they are no longer needed.
-
-Tags
-~~~~
-
-Tags are used for milestones, release candidates, and approved releases. Please refrain from creating arbitrary
-tags, as they produce permanent clutter.
-
-Issue Tracker Conventions
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-TinkerPop uses Apache JIRA as its link:https://issues.apache.org/jira/browse/TINKERPOP3[issue tracker]. JIRA is a
-very robust piece of software with many options and configurations. To simplify usage and ensure consistency across
-issues, the following conventions should be adhered to:
-
-* An issue's "status" should generally be in one of two states: `open` or `closed` (`reopened` is equivalent to `open`
-for our purposes).
-** An `open` issue is newly created, under consideration or otherwise in progress.
-** A `closed` issue is completed for purposes of release (i.e. code, testing, and documentation complete).
-** Issues in a `resolved` state should immediately be evaluated for movement to `closed` - issue become `resolved`
-by those who don't have the permissions to `close`.
-* An issue's "type" should be one of two options: `bug` or `improvement`.
-** A `bug` has a very specific meaning, referring to an error that prevents usage of TinkerPop AND does not have a
-reasonable workaround. Given that definition, a `bug` should generally have very high priority for a fix.
-** Everything else is an `improvement` in the sense that any other work is an enhancement to the current codebase.
-* The "component" should be representative of the primary area of code that it applies to and all issues should have
-this property set.
-* Issues are not assigned "labels" with two exceptions:
-** The "breaking" label which marks an issue as one that is representative of a change in the API that might
-affect users or vendors. This label is important when organizing release notes.
-** The "deprecation" label which is assigned to an issue that is about removing a deprecated portion of the API.
-* The "affects/fix version(s)" fields should be appropriately set, where the "fix version" implies the version on
-which that particular issue will completed.
-* The "priority" field can be arbitrarily applied with one exception. The "trivial" option should be reserved for
-tasks that are "easy" for a potential new contributor to jump into and do not have significant impact to urgently
-required improvements.
-
-Code Style
-~~~~~~~~~~
-
-Contributors should examine the current code base to determine what the code style patterns are and should match their
-style to what is already present. Of specific note however, TinkerPop does not use "import wildcards" - IDEs should
-be adjusted accordingly to not auto-wildcard the imports.
-
-Deprecation
-~~~~~~~~~~~
-
-When possible, committers should avoid direct "breaking" change (e.g. removing a method from a class) and favor
-deprecation. Deprecation should come with sufficient documentation and notice especially when the change involves
-public APIs that might be utilized by users or implemented by vendors:
-
-* Mark the code with the `@Deprecated` annotation.
-* Use javadoc to further document the change with the following content:
-** `@deprecated As of release x.y.z, replaced by {@link SomeOtherClass#someNewMethod()}` - if the method is not
-replaced then the comment can simply read "not replaced". Additional comments that provide more context are
-encouraged.
-** `@see <a href="https://issues.apache.org/jira/browse/TINKERPOP3-XXX">TINKERPOP3-XXX</a>` - supply a link to the
-JIRA issue for reference.
-* All deprecation should typically be tied to a JIRA issue with a "breaking" label - the issue itself does not need to
-specifically or solely be about "deprecation" but it should be documented very clearly in the comments what was
-deprecated and what the path forward should be.
-* Be sure that deprecated methods are still under test - consider using javadoc/comments in the tests themselves to
-call out this fact.
-* Create a new JIRA issue to track removal of the deprecation for future evaluation - this issue should have the
-"breaking" label as well as a "deprecation" label.
-* Update the "upgrade documentation" to reflect the API change and how the reader should resolve it.
-
-The JIRA issues that track removal of deprecated methods should be periodically evaluated to determine if it is
-prudent to schedule them into a release.
-
-Gremlin Language Test Cases
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-When writing a test case for a Gremlin step, be sure to use the following conventions.
-
-* The name of the traversal generator should start with `get`, use `X` for brackets, `_` for space, and the Gremlin-Groovy sugar syntax.
-** `get_g_V_hasLabelXpersonX_groupXaX_byXageX_byXsumX_name()`
-* When creating a test for a step that has both a barrier and sideEffect form (e.g. `group()`, `groupCount()`, etc.), test both representations.
-** `get_g_V_groupCount_byXnameX()`
-** `get_g_V_groupCountXaX_byXnameX_capXaX()`
-* The name of the actual test case should be the name of the traversal generator minus the `get_` prefix.
-* The Gremlin-Groovy version of the test should use the sugar syntax in order to test sugar (as Gremlin-Java8 tests test standard syntax).
-** `g.V.age.sum`
-* Avoid using lambdas in the test case unless that is explicitly what is being tested as OLAP systems will typically not be able to execute those tests.
-* `AbstractGremlinProcessTest` has various static methods to make writing a test case easy.
-** `checkResults(Arrays.asList("marko","josh"), traversal)`
-** `checkMap(new HashMap<String,Long>() {{ put("marko",1l); }}, traversal.next())`
-
-[[rtc]]
-Review then Commit
-~~~~~~~~~~~~~~~~~~
-
-Code modifications must go through a link:http://www.apache.org/foundation/glossary.html#ReviewThenCommit[review-then-committ] (RTC)
-process before being merged into a release branch. All committers should follow the pattern below, where "you" refers to the
-committer wanting to put code into a release branch.
-
-* Make a JIRA ticket for the software problem you want to solve (i.e. a fix).
-* Fork the release branch that the fix will be put into.
-** The branch name should be the JIRA issue identifier (e.g. `TINKERPOP3-XXX`).
-* Develop your fix in your branch.
-* When your fix is complete and ready to merge, issue a link:https://git-scm.com/docs/git-request-pull[pull request].
-** Be certain that the test suite is passing.
-** If you updated documentation, be sure that the `process-docs.sh` is building the documentation correctly.
-* Before you can merge your branch into the release branch, you must have at least 3 +1 link:http://www.apache.org/foundation/glossary.html#ConsensusApproval[consensus votes].
-** Please see the Apache Software Foundations regulations regarding link:http://www.apache.org/foundation/voting.html#votes-on-code-modification[Voting on Code Modifications].
-* Votes are issued by TinkerPop committers as comments to the pull request.
-* Once 3 +1 votes are received, you are responsible for merging to the release branch and handling any merge conflicts.
-** If there is a higher version release branch that requires your fix (e.g. `3.y-1.z` fix going to a `3.y.z` release), be sure to merge to that release branch as well.
-* Be conscious of deleting your branch if it is no longer going to be used so stale branches don't pollute the repository.
-
-NOTE: These steps also generally apply to external pull requests from those who are not official Apache committers. In
-this case, the person responsible for the merge after voting is typically the first person available
-who is knowledgeable in the area that the pull request affects. Any additional coordination on merging can be handled
-via the pull request comment system.
-
-The following exceptions to the RTC (review-then-commit) model presented above are itemized below. It is up to the
-committer to self-regulate as the itemization below is not complete and only hints at the types of commits that do not
-require a review.
-
-* You are responsible for a release and need to manipulate files accordingly for the release.
-** `Gremlin.version()`, CHANGELOG dates, `pom.xml` version bumps, etc.
-* You are doing an minor change and it is obvious that an RTC is not required (would be a pointless burden to the community).
-** The fix is under the link:http://www.apache.org/foundation/glossary.html#CommitThenReview[commit-then-review] (CTR) policy and lazy consensus is sufficient, where a single -1 vote requires you to revert your changes.
-** Adding a test case, fixing spelling/grammar mistakes in the documentation, fixing LICENSE/NOTICE/etc. files, fixing a minor issue in an already merged branch.
-
-When the committer chooses CTR, it is considered good form to include something in the commit message that explains
-that CTR was invoked and the reason for doing so. For example, "Invoking CTR as this change encompasses minor
-adjustments to text formatting."
-
-Pull Request Format
-^^^^^^^^^^^^^^^^^^^
-
-When you submit a pull request, be sure it uses the following style.
-
-* The title of the pull request is the JIRA ticket number + "colon" + the title of the JIRA ticket.
-* The first line of the pull request message should contain a link to the JIRA ticket.
-* Discuss what you did to solve the problem articulated in the JIRA ticket.
-* Discuss any "extra" work done that go beyond the assumed requirements of the JIRA ticket.
-* Be sure to explain what you did to prove that the issue is resolved.
-** Test cases written.
-** Integration tests run (if required for the work accomplished).
-** Documentation building (if required for the work accomplished).
-** Any manual testing (though this should be embodied in a test case).
-* Notes about what you will do when you merge to the respective release branch (e.g. update CHANGELOG).
-** These types of "on merge tweaks" are typically done to extremely dynamic files to combat and merge conflicts.
-* If you are a TinkerPop committer, you can VOTE on your own pull request, so please do so.
-
-[[dependencies]]
-Dependencies
-~~~~~~~~~~~~
-
-There are many dependencies on other open source libraries in TinkerPop modules. When adding dependencies or
-altering the version of a dependency, developers must consider the implications that may apply to the TinkerPop
-LICENSE and NOTICE files. There are two implications to consider:
-
-. Does the dependency fit an Apache _approved_ license?
-. Given the addition or modification to a dependency, does it mean any change for TinkerPop LICENSE and NOTICE files?
-
-Understanding these implications is important for insuring that TinkerPop stays compliant with the Apache 2 license
-that it releases under.
-
-Regarding the first item, refer to the Apache Legal for a list of link:http://www.apache.org/legal/resolved.html[approved licenses]
-that are compatible with the Apache 2 license.
-
-The second item requires a bit more effort to follow. The Apache website offers a
-link:http://www.apache.org/dev/licensing-howto.html[how-to guide] on the approach to maintaining appropriate LICENSE
-and NOTICE files, but this guide is designed to offer some more specific guidance as it pertains to TinkerPop
-and its distribution.
-
-To get started, TinkerPop has both "source" and "binary" LICENSE/NOTICE files:
-
-* Source LICENSE/NOTICE relate to files packaged with the released source code distribution:
-link:https://github.com/apache/incubator-tinkerpop/blob/master/LICENSE[LICENSE] / link:https://github.com/apache/incubator-tinkerpop/blob/master/NOTICE[NOTICE]
-* Binary LICENSE/NOTICE relate to files packaged with the released binary distributions:
-** Gremlin Console link:https://github.com/apache/incubator-tinkerpop/blob/master/gremlin-console/src/main/LICENSE[LICENSE]
-/ link:https://github.com/apache/incubator-tinkerpop/blob/master/gremlin-console/src/main/NOTICE[NOTICE]
-** Gremlin Server link:https://github.com/apache/incubator-tinkerpop/blob/master/gremlin-server/src/main/LICENSE[LICENSE]
-/ link:https://github.com/apache/incubator-tinkerpop/blob/master/gremlin-server/src/main/NOTICE[NOTICE]
-
-As we don't usually include any dependencies in the source distribution (i.e. the source zip distribution), there is
-typically no need to edit source LICENSE/NOTICE when editing a TinkerPop `pom.xml`. These files only need to be edited
-if the distribution has a file added to it. Such a situation may arise from several scenarios, but it would most
-likely come from the addition of a source file from another library. In that case, edit LICENSE to include this file
-following the pattern already present there. Refer to the
-link:http://www.apache.org/dev/licensing-howto.html#mod-notice[Modifications to Notice] section from the Apache
-"how-to" to determine if changes are necessary there.
-
-The binary LICENSE/NOTICE is perhaps most impacted by changes to the various `pom.xml` files. After altering the
-`pom.xml` file of any module, build both Gremlin Console and Gremlin Server and examine the contents of both binary
-distributions, either:
-
-* target/apache-gremlin-console-x.y.z-distribution.zip
-* target/apache-gremlin-server-x.y.z-distribution.zip
-
-There is a _near_ one-to-one mapping between the jar files in those distributions and their respective LICENSE files.
-It is described as "near" because the distributions include TinkerPop jars which don't need to be represented there
-and there are entries in LICENSE that refer to libraries that are present, but shaded in the gremlin-shaded module.
-Everything else should be listed and present under the appropriate license section.
-
-There is also a good chance that the TinkerPop binary NOTICE should be updated. Check if the newly added dependency
-contains a NOTICE of its own. If so, include that NOTICE in the TinkerPop NOTICE.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bc46d649/docs/src/developer-meetings.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/developer-meetings.asciidoc b/docs/src/developer-meetings.asciidoc
deleted file mode 100644
index e381707..0000000
--- a/docs/src/developer-meetings.asciidoc
+++ /dev/null
@@ -1,64 +0,0 @@
-////
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-////
-Meetings
-========
-
-October 29, 2015
-----------------
-
-The meeting was scheduled for 1:00pm EST, started on time and was held via Google Hangout hosted by Stephen Mallette.
-Meeting adjourned at 1:45pm EST.
-
-*Committers/PPMC Members*
-
-* Stephen Mallette
-* Marko Rodriguez
-* Daniel Kuppitz
-
-*Guests*
-
-_None_
-
-*Minutes*
-
-* Reviewed the scope of 3.1.0-incubating in JIRA in the context of the upcoming release date on November 16, 2015.
-* It was noted that with the new one week code freeze policy that the real cut-off date for work is November 9, 2015.
-* There was general consensus on the call that work should be accomplished in such a way that the code review process
-not drag into the code freeze time period. In other words, pull requests to the release branch should be completed
-well ahead of the 9th.
-
-Upon review of the tickets in JIRA assigned to 3.1.0-incubating, the following were removed from the 3.1.0-incubating
-roadmap:
-
-[width="100%",cols="2,5,5",options="header"]
-|=========================================================
-|ID |Description |Removal Reasoning
-|TINKERPOP3-891 |Re-examine Sandboxing Abstractions |Low priority and time limitations.
-|TINKERPOP3-890 |Remove the concept of branch/ package. |Awaiting step library definition in 3.2.0-incubating.
-|TINKERPOP3-873 |FoldStep should default to using BulkSet for efficiency. |Awaiting step library definition in 3.2.0-incubating.
-|TINKERPOP3-864 |Operator.mean would be nice. |Awaiting step library definition in 3.2.0-incubating.
-|TINKERPOP3-862 |Add a TraversalSourceStrategy that provides "locked" values. |Low priority and time limitations.
-|TINKERPOP3-854 |Remove PropertyMapStep in favor of multi-instruction construction. |Non-trivial given time limitations.
-|TINKERPOP3-789 |Choose then Enforce Semantics for Graph.close() |Non-trivial given time limitations.
-|TINKERPOP3-768 |MatchStep in OLAP should be smart about current vertex. |Non-trivial given time limitations.
-|TINKERPOP3-705 |Make use of a MemoryRDD instead of Accumulators in SparkGraphComputer? |Non-trivial given time limitations.
-|TINKERPOP3-659 |GraphConfiguration Class |Non-trivial given time limitations.
-|TINKERPOP3-623 |More output for OLAP jobs |Not clearly defined given time limitations.
-|TINKERPOP3-620 |Commutative Step Marker interface |Awaiting step library definition in 3.2.0-incubating
-|TINKERPOP3-550 |Gremlin IO needs to support both OLTP and OLAP naturally. |Not clearly defined given time limitations.
-|TINKERPOP3-479 |Consider Providing "getOrCreate" Functionality |Not clearly defined given time limitations.
-|=========================================================
\ No newline at end of file