You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by sp...@apache.org on 2018/04/27 12:36:15 UTC
[1/4] tinkerpop git commit: Fix title formatting in olap/spark/yarn
recipe CTR
Repository: tinkerpop
Updated Branches:
refs/heads/master 08d98ad82 -> 219093692
Fix title formatting in olap/spark/yarn recipe CTR
Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/0694cd7c
Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/0694cd7c
Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/0694cd7c
Branch: refs/heads/master
Commit: 0694cd7cf7368752ddac041218a76c2db921983b
Parents: 09fd327
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Fri Apr 27 08:33:03 2018 -0400
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Fri Apr 27 08:33:03 2018 -0400
----------------------------------------------------------------------
docs/src/recipes/olap-spark-yarn.asciidoc | 20 ++++++++------------
1 file changed, 8 insertions(+), 12 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/0694cd7c/docs/src/recipes/olap-spark-yarn.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/recipes/olap-spark-yarn.asciidoc b/docs/src/recipes/olap-spark-yarn.asciidoc
index 1543829..54ecf77 100644
--- a/docs/src/recipes/olap-spark-yarn.asciidoc
+++ b/docs/src/recipes/olap-spark-yarn.asciidoc
@@ -15,8 +15,7 @@ See the License for the specific language governing permissions and
limitations under the License.
////
[[olap-spark-yarn]]
-OLAP traversals with Spark on YARN
-----------------------------------
+== OLAP traversals with Spark on YARN
TinkerPop's combination of link:http://tinkerpop.apache.org/docs/x.y.z/reference/#sparkgraphcomputer[SparkGraphComputer]
and link:http://tinkerpop.apache.org/docs/x.y.z/reference/#_properties_files[HadoopGraph] allows for running
@@ -26,8 +25,7 @@ where Spark runs locally or where the cluster is managed by a Spark server. Howe
via the http://hadoop.apache.org/[Hadoop 2.x] Resource Manager (YARN), which requires `SparkGraphComputer` to be
configured differently. This recipe describes this configuration.
-Approach
-~~~~~~~~
+=== Approach
Most configuration problems of TinkerPop with Spark on YARN stem from three reasons:
@@ -40,8 +38,8 @@ The current recipe follows a minimalist approach in which no dependencies are ad
included in the TinkerPop binary distribution. The Hadoop cluster's Spark installation is completely ignored. This
approach minimizes the chance of dependency version conflicts.
-Prerequisites
-~~~~~~~~~~~~~
+=== Prerequisites
+
This recipe is suitable for both a real external and a local pseudo Hadoop cluster. While the recipe is maintained
for the vanilla Hadoop pseudo-cluster, it has been reported to work on real clusters with Hadoop distributions
from various vendors.
@@ -79,8 +77,7 @@ export HADOOP_GREMLIN_LIBS=$GREMLIN_HOME/empty
bin/gremlin.sh
----
-Running the job
-~~~~~~~~~~~~~~~
+=== Running the job
You can now run a gremlin OLAP query with Spark on YARN:
@@ -118,8 +115,7 @@ the YARN Resource Manager UI (e.g. \http://rm.your.domain:8088/cluster), provide
`yarn.log-aggregation-enable` property set to `true`. See the Spark documentation for
https://spark.apache.org/docs/latest/running-on-yarn.html#debugging-your-application[additional hints].
-Explanation
-~~~~~~~~~~~
+=== Explanation
This recipe does not require running the `bin/hadoop/init-tp-spark.sh` script described in the
link:http://tinkerpop.apache.org/docs/x.y.z/reference/#sparkgraphcomputer[reference documentation] and thus is also
@@ -138,8 +134,8 @@ The `gremlin.spark.persistContext` property is explained in the reference docume
link:http://tinkerpop.apache.org/docs/x.y.z/reference/#sparkgraphcomputer[SparkGraphComputer]: it helps in getting
follow-up OLAP queries answered faster, because you skip the overhead for getting resources from YARN.
-Additional configuration options
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+=== Additional configuration options
+
This recipe does most of the graph configuration in the Gremlin Console so that environment variables can be used and
the chance of configuration mistakes is minimal. Once you have your setup working, it is probably easier to make a copy
of the `conf/hadoop/hadoop-gryo.properties` file and put the property values specific to your environment there. This is
[3/4] tinkerpop git commit: Merge branch 'tp32' into tp33
Posted by sp...@apache.org.
Merge branch 'tp32' into tp33
Conflicts:
gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/Order.java
Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/71bc37fd
Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/71bc37fd
Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/71bc37fd
Branch: refs/heads/master
Commit: 71bc37fdfe00bb8c6b75d8bf81834713abf8cd22
Parents: a38d68f 5fea198
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Fri Apr 27 08:35:55 2018 -0400
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Fri Apr 27 08:35:55 2018 -0400
----------------------------------------------------------------------
docs/src/recipes/olap-spark-yarn.asciidoc | 20 ++++++++------------
.../gremlin/process/traversal/Order.java | 9 ++++++++-
2 files changed, 16 insertions(+), 13 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/71bc37fd/docs/src/recipes/olap-spark-yarn.asciidoc
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/71bc37fd/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/Order.java
----------------------------------------------------------------------
diff --cc gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/Order.java
index 847cc57,c9111f0..0b88bba
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/Order.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/Order.java
@@@ -67,7 -66,82 +67,14 @@@ public enum Order implements Comparator
public Order reversed() {
return incr;
}
- }, shuffle {
+ },
+
+ /**
- * @since 3.0.0-incubating
- * @deprecated As of release 3.1.1-incubating, replaced by {@link org.apache.tinkerpop.gremlin.structure.Column#keys}.
- */
- @Deprecated
- keyIncr {
- @Override
- public int compare(final Object first, final Object second) {
- return Comparator.<Comparable>naturalOrder().compare(((Map.Entry<Comparable, ?>) first).getKey(), ((Map.Entry<Comparable, ?>) second).getKey());
- }
-
- @Override
- public Order reversed() {
- return keyDecr;
- }
- },
-
- /**
- * @since 3.0.0-incubating
- * @deprecated As of release 3.1.1-incubating, replaced by {@link org.apache.tinkerpop.gremlin.structure.Column#values}.
- */
- @Deprecated
- valueIncr {
- @Override
- public int compare(final Object first, final Object second) {
- return Comparator.<Comparable>naturalOrder().compare(((Map.Entry<?, Comparable>) first).getValue(), ((Map.Entry<?, Comparable>) second).getValue());
- }
-
- @Override
- public Order reversed() {
- return valueDecr;
- }
- },
-
- /**
- * @since 3.0.0-incubating
- * @deprecated As of release 3.1.1-incubating, replaced by {@link org.apache.tinkerpop.gremlin.structure.Column#keys}.
- */
- @Deprecated
- keyDecr {
- @Override
- public int compare(final Object first, final Object second) {
- return Comparator.<Comparable>reverseOrder().compare(((Map.Entry<Comparable, ?>) first).getKey(), ((Map.Entry<Comparable, ?>) second).getKey());
- }
-
- @Override
- public Order reversed() {
- return keyIncr;
- }
- },
-
- /**
- * @since 3.0.0-incubating
- * @deprecated As of release 3.1.1-incubating, replaced by {@link org.apache.tinkerpop.gremlin.structure.Column#values}.
- */
- @Deprecated
- valueDecr {
- @Override
- public int compare(final Object first, final Object second) {
- return Comparator.<Comparable>reverseOrder().compare(((Map.Entry<?, Comparable>) first).getValue(), ((Map.Entry<?, Comparable>) second).getValue());
- }
-
- @Override
- public Order reversed() {
- return valueIncr;
- }
- },
-
- /**
+ * Order in a random fashion.
+ *
+ * @since 3.0.0-incubating
+ */
+ shuffle {
@Override
public int compare(final Object first, final Object second) {
return RANDOM.nextBoolean() ? -1 : 1;
[2/4] tinkerpop git commit: Added javadoc in Order CTR
Posted by sp...@apache.org.
Added javadoc in Order CTR
Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/5fea198c
Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/5fea198c
Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/5fea198c
Branch: refs/heads/master
Commit: 5fea198c836c9886e7a8cc8abc9ead44956ad2b6
Parents: 0694cd7
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Fri Apr 27 08:34:34 2018 -0400
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Fri Apr 27 08:34:34 2018 -0400
----------------------------------------------------------------------
.../tinkerpop/gremlin/process/traversal/Order.java | 13 ++++++++++++-
1 file changed, 12 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/5fea198c/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/Order.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/Order.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/Order.java
index 3710396..c9111f0 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/Order.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/Order.java
@@ -67,6 +67,7 @@ public enum Order implements Comparator<Object> {
return incr;
}
},
+
/**
* @since 3.0.0-incubating
* @deprecated As of release 3.1.1-incubating, replaced by {@link org.apache.tinkerpop.gremlin.structure.Column#keys}.
@@ -83,6 +84,7 @@ public enum Order implements Comparator<Object> {
return keyDecr;
}
},
+
/**
* @since 3.0.0-incubating
* @deprecated As of release 3.1.1-incubating, replaced by {@link org.apache.tinkerpop.gremlin.structure.Column#values}.
@@ -99,6 +101,7 @@ public enum Order implements Comparator<Object> {
return valueDecr;
}
},
+
/**
* @since 3.0.0-incubating
* @deprecated As of release 3.1.1-incubating, replaced by {@link org.apache.tinkerpop.gremlin.structure.Column#keys}.
@@ -115,6 +118,7 @@ public enum Order implements Comparator<Object> {
return keyIncr;
}
},
+
/**
* @since 3.0.0-incubating
* @deprecated As of release 3.1.1-incubating, replaced by {@link org.apache.tinkerpop.gremlin.structure.Column#values}.
@@ -130,7 +134,14 @@ public enum Order implements Comparator<Object> {
public Order reversed() {
return valueIncr;
}
- }, shuffle {
+ },
+
+ /**
+ * Order in a random fashion.
+ *
+ * @since 3.0.0-incubating
+ */
+ shuffle {
@Override
public int compare(final Object first, final Object second) {
return RANDOM.nextBoolean() ? -1 : 1;
[4/4] tinkerpop git commit: Merge branch 'tp33'
Posted by sp...@apache.org.
Merge branch 'tp33'
Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/21909369
Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/21909369
Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/21909369
Branch: refs/heads/master
Commit: 2190936927df79e1e7c7ffc11a5b6677651610f6
Parents: 08d98ad 71bc37f
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Fri Apr 27 08:36:07 2018 -0400
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Fri Apr 27 08:36:07 2018 -0400
----------------------------------------------------------------------
docs/src/recipes/olap-spark-yarn.asciidoc | 20 ++++++++------------
.../gremlin/process/traversal/Order.java | 9 ++++++++-
2 files changed, 16 insertions(+), 13 deletions(-)
----------------------------------------------------------------------