You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by sp...@apache.org on 2015/11/05 02:56:19 UTC
[01/50] [abbrv] incubator-tinkerpop git commit: Modifications to
release process
Repository: incubator-tinkerpop
Updated Branches:
refs/heads/TINKERPOP3-923 3326fcc66 -> 7e9aef4f5
Modifications to release process
Based on things learned in the 3.0.2-incubating release it made some sense to move some steps around. Most specifically, I moved staging the nexus repo to PMC Vote stage as that saves rebuilding later after incubator general VOTE.
I'm going to CTR this one but @okram can you just take a quick review on this commit to make sure i'm not hurting myself in some way for next release.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/553f91fc
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/553f91fc
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/553f91fc
Branch: refs/heads/TINKERPOP3-923
Commit: 553f91fcf89f14bb99c60d5e055d65e2b36d01f5
Parents: 1478c43
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Thu Oct 29 15:12:49 2015 -0400
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Thu Oct 29 15:12:49 2015 -0400
----------------------------------------------------------------------
docs/src/developer-release.asciidoc | 9 ++++-----
1 file changed, 4 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/553f91fc/docs/src/developer-release.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/developer-release.asciidoc b/docs/src/developer-release.asciidoc
index e2b7eeb..ef7349c 100644
--- a/docs/src/developer-release.asciidoc
+++ b/docs/src/developer-release.asciidoc
@@ -92,9 +92,10 @@ A positive vote for a particular release from the TinkerPop PMC is required to m
. `git diff` and review the updated files (expect all `pom.xml` files and this README)
. `git commit -a -m "TinkerPop x.y.z release"` and `git push`
. `git tag -a -m "TinkerPop x.y.z release" x.y.z` and `git push --tags`
-. `mvn clean install -Dmaven.test.skip=true`
+. `mvn clean install -Dmaven.test.skip=true` - need to build first so that the right version of the console is used with `bin/publish-docs.sh`
. `bin/publish-docs.sh <username>`
-. `mvn install -Papache-release -DcreateChecksum=true -Dmaven.test.skip=true`
+. `mvn install -Papache-release -Djavadoc -DcreateChecksum=true -Dmaven.test.skip=true`
+. `mvn deploy -Papache-release -DcreateChecksum=true -Dmaven.test.skip=true`- deploy signed artifacts with checksums to link:https://repository.apache.org/[Apache Nexus]. Review (artifacts versions, file sizes, anything that might be out of place - request another committer to review as well) but do NOT close/release the staging repository at this time.
. Review generated artifacts to be sure they have both javadocs and asciidocs present
. Upload artifacts to `https://dist.apache.org/repos/dist/dev/incubator/tinkerpop` for `[VOTE]` review.
.. `svn co --depth empty https://dist.apache.org/repos/dist/dev/incubator/tinkerpop/ dev` and `mkdir dev/x.y.z`
@@ -123,9 +124,7 @@ A positive vote for a particular release from the Apache Incubator is required t
Release & Promote
-----------------
-. `mvn clean install -Djavadoc -Dmaven.test.skip=true; bin/process-docs.sh` - rebuild source and docs of tagged release
-. `mvn deploy -Papache-release -DcreateChecksum=true -Dmaven.test.skip=true`- deploy signed artifacts with checksums to Apache Nexus
-. Review and close the staging repository (Apache Nexus at link:https://repository.apache.org/[https://repository.apache.org/]) - check that versions are correct and that all artifacts are present
+. Close the staging repository at link:https://repository.apache.org/[Apache Nexus]) and then release.
. `svn co --depth empty https://dist.apache.org/repos/dist/dev/incubator/tinkerpop dev; svn up dev/x.y.z`
. `svn co --depth empty https://dist.apache.org/repos/dist/release/incubator/tinkerpop release; mkdir release/x.y.z`
. `ls dev/x.y.z/ | grep '\-\(distribution\|source\-release\)\.zip' | sed -e 's/\(^[^ ]\*\)-distribution\([^ ]*\)/cp dev\/x.y.z\/\0 release\/x.y.z\/\1-bin\2/' -e 's/\(^[^ ]\*\)-source-release\([^ ]*\)/cp dev\/x.y.z\/\0 release\/x.y.z\/\1-src\2/' | /bin/sh`
[18/50] [abbrv] incubator-tinkerpop git commit: modified method/type
signature of SumGlobalStep after getting some feedback from @okram
Posted by sp...@apache.org.
modified method/type signature of SumGlobalStep after getting some feedback from @okram
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/eda16b50
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/eda16b50
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/eda16b50
Branch: refs/heads/TINKERPOP3-923
Commit: eda16b504fda57f58d8d90a60421bde386528bba
Parents: d49deaf
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Sat Oct 31 18:42:09 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Sat Oct 31 18:42:09 2015 +0100
----------------------------------------------------------------------
.../gremlin/process/traversal/step/map/SumGlobalStep.java | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/eda16b50/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumGlobalStep.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumGlobalStep.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumGlobalStep.java
index 8816390..14573d1 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumGlobalStep.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumGlobalStep.java
@@ -43,7 +43,7 @@ import static org.apache.tinkerpop.gremlin.process.traversal.NumberHelper.mul;
/**
* @author Marko A. Rodriguez (http://markorodriguez.com)
*/
-public final class SumGlobalStep extends ReducingBarrierStep<Number, Number> implements MapReducer {
+public final class SumGlobalStep<S extends Number> extends ReducingBarrierStep<S, S> implements MapReducer {
private static final Set<TraverserRequirement> REQUIREMENTS = EnumSet.of(
TraverserRequirement.BULK,
@@ -52,7 +52,7 @@ public final class SumGlobalStep extends ReducingBarrierStep<Number, Number> imp
public SumGlobalStep(final Traversal.Admin traversal) {
super(traversal);
- this.setSeedSupplier(new ConstantSupplier<>(0));
+ this.setSeedSupplier(new ConstantSupplier<>((S) Integer.valueOf(0)));
this.setBiFunction(SumGlobalBiFunction.instance());
}
@@ -69,7 +69,7 @@ public final class SumGlobalStep extends ReducingBarrierStep<Number, Number> imp
/////
- private static class SumGlobalBiFunction<S extends Number> implements BiFunction<Number, Traverser<S>, Number>, Serializable {
+ private static class SumGlobalBiFunction<S extends Number> implements BiFunction<S, Traverser<S>, S>, Serializable {
private static final SumGlobalBiFunction INSTANCE = new SumGlobalBiFunction<>();
@@ -78,8 +78,8 @@ public final class SumGlobalStep extends ReducingBarrierStep<Number, Number> imp
}
@Override
- public Number apply(final Number mutatingSeed, final Traverser<S> traverser) {
- return add(mutatingSeed, mul(traverser.get(), traverser.bulk()));
+ public S apply(final S mutatingSeed, final Traverser<S> traverser) {
+ return (S) add(mutatingSeed, mul(traverser.get(), traverser.bulk()));
}
public static <S extends Number> SumGlobalBiFunction<S> instance() {
[31/50] [abbrv] incubator-tinkerpop git commit: Integer.MAX_VALUE
should be Long.MAX_VALUE. CTR as its a trivial tweak.
Posted by sp...@apache.org.
Integer.MAX_VALUE should be Long.MAX_VALUE. CTR as its a trivial tweak.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/06d747f0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/06d747f0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/06d747f0
Branch: refs/heads/TINKERPOP3-923
Commit: 06d747f094eef6d3f42d9b322bb3c2c806b5d3fc
Parents: aadd422
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Tue Nov 3 09:17:27 2015 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Tue Nov 3 09:17:27 2015 -0700
----------------------------------------------------------------------
.../gremlin/hadoop/structure/hdfs/HadoopElementIterator.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/06d747f0/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/hdfs/HadoopElementIterator.java
----------------------------------------------------------------------
diff --git a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/hdfs/HadoopElementIterator.java b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/hdfs/HadoopElementIterator.java
index 3602f85..a9f287b 100644
--- a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/hdfs/HadoopElementIterator.java
+++ b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/hdfs/HadoopElementIterator.java
@@ -52,7 +52,7 @@ public abstract class HadoopElementIterator<E extends Element> implements Iterat
this.graph = graph;
final Configuration configuration = ConfUtil.makeHadoopConfiguration(this.graph.configuration());
for (final Path path2 : HDFSTools.getAllFilePaths(FileSystem.get(configuration), path, HiddenFileFilter.instance())) {
- this.readers.add(inputFormat.createRecordReader(new FileSplit(path2, 0, Integer.MAX_VALUE, new String[]{}), new TaskAttemptContextImpl(configuration, new TaskAttemptID())));
+ this.readers.add(inputFormat.createRecordReader(new FileSplit(path2, 0, Long.MAX_VALUE, new String[]{}), new TaskAttemptContextImpl(configuration, new TaskAttemptID())));
}
}
@@ -63,7 +63,7 @@ public abstract class HadoopElementIterator<E extends Element> implements Iterat
final Configuration configuration = ConfUtil.makeHadoopConfiguration(this.graph.configuration());
final InputFormat<NullWritable, VertexWritable> inputFormat = this.graph.configuration().getGraphInputFormat().getConstructor().newInstance();
for (final Path path : HDFSTools.getAllFilePaths(FileSystem.get(configuration), new Path(graph.configuration().getInputLocation()), HiddenFileFilter.instance())) {
- this.readers.add(inputFormat.createRecordReader(new FileSplit(path, 0, Integer.MAX_VALUE, new String[]{}), new TaskAttemptContextImpl(configuration, new TaskAttemptID())));
+ this.readers.add(inputFormat.createRecordReader(new FileSplit(path, 0, Long.MAX_VALUE, new String[]{}), new TaskAttemptContextImpl(configuration, new TaskAttemptID())));
}
}
} catch (Exception e) {
[47/50] [abbrv] incubator-tinkerpop git commit: Fixed broken link in
contributors guide for newcomers.
Posted by sp...@apache.org.
Fixed broken link in contributors guide for newcomers.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/b2daf097
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/b2daf097
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/b2daf097
Branch: refs/heads/TINKERPOP3-923
Commit: b2daf097e3c62440e7da2637ccc6fce8aaad114a
Parents: 863bdab
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Wed Nov 4 14:51:07 2015 -0500
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Wed Nov 4 14:51:07 2015 -0500
----------------------------------------------------------------------
docs/src/developer-contributing.asciidoc | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/b2daf097/docs/src/developer-contributing.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/developer-contributing.asciidoc b/docs/src/developer-contributing.asciidoc
index fceedb2..fcdade0 100644
--- a/docs/src/developer-contributing.asciidoc
+++ b/docs/src/developer-contributing.asciidoc
@@ -247,7 +247,7 @@ When writing a test case for a Gremlin step, be sure to use the following conven
** `checkResults(Arrays.asList("marko","josh"), traversal)`
** `checkMap(new HashMap<String,Long>() {{ put("marko",1l); }}, traversal.next())`
-[rtc]
+[[rtc]]
Review then Commit
~~~~~~~~~~~~~~~~~~
[37/50] [abbrv] incubator-tinkerpop git commit: Merge branch 'master'
into TINKERPOP3-904
Posted by sp...@apache.org.
Merge branch 'master' into TINKERPOP3-904
Resolved Conflicts:
CHANGELOG.asciidoc
tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/a2b94c4e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/a2b94c4e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/a2b94c4e
Branch: refs/heads/TINKERPOP3-923
Commit: a2b94c4e4e25a04bdbd8b926fdc9d949ed6e42d2
Parents: 63c8cb8 558c04e
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Wed Nov 4 13:51:18 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Wed Nov 4 13:51:18 2015 +0100
----------------------------------------------------------------------
CHANGELOG.asciidoc | 1 +
.../upgrade-release-3.1.x-incubating.asciidoc | 3 +-
.../gremlin/process/traversal/NumberHelper.java | 12 +++-
.../traversal/dsl/graph/GraphTraversal.java | 16 ++---
.../traversal/step/map/MaxGlobalStep.java | 14 +++--
.../traversal/step/map/MaxLocalStep.java | 5 +-
.../traversal/step/map/MeanGlobalStep.java | 61 +++++++++++---------
.../traversal/step/map/MeanLocalStep.java | 11 ++--
.../traversal/step/map/MinGlobalStep.java | 14 +++--
.../traversal/step/map/MinLocalStep.java | 5 +-
.../traversal/step/map/SumGlobalStep.java | 25 ++++----
.../traversal/step/map/SumLocalStep.java | 5 +-
.../traversal/step/map/MeanGlobalStepTest.java | 14 +++++
.../traversal/step/map/MeanLocalStepTest.java | 14 +++++
.../step/sideEffect/GroovyGroupTest.groovy | 2 +-
.../process/computer/GraphComputerTest.java | 29 +++++-----
.../process/traversal/step/map/SumTest.java | 2 +-
.../traversal/step/sideEffect/GroupTest.java | 22 ++++---
...ComputerVerificationStrategyProcessTest.java | 3 +-
.../groovy/plugin/HadoopGremlinPluginTest.java | 2 +-
.../spark/structure/io/InputRDDTest.java | 2 +-
21 files changed, 164 insertions(+), 98 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/a2b94c4e/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --cc CHANGELOG.asciidoc
index ef0d032,5ad89e5..8d74212
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@@ -25,7 -25,7 +25,8 @@@ image::https://raw.githubusercontent.co
TinkerPop 3.1.0 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Optimized `BulkLoaderVertexProgram`. It now uses `EventStrategy` to monitor what the underlying `BulkLoader` implementation does (e.g. whether it creates a new vertex or returns an existing).
+ * Integrated `NumberHelper` in `SumStep`, `MinStep`, `MaxStep` and `MeanStep` (local and global step variants).
* Bumped to Neo4j 2.3.0.
* Added `PersistedInputRDD` and `PersistedOutputRDD` which enables `SparkGraphComputer` to store the graph RDD in the context between jobs (no HDFS serialization required).
* Renamed the `public static String` configuration variable names of TinkerGraph (deprecated old variables).
[12/50] [abbrv] incubator-tinkerpop git commit: Testing to make sure
InputOutputHelper translator works for registered RDDs.
Posted by sp...@apache.org.
Testing to make sure InputOutputHelper translator works for registered RDDs.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/cf0c4562
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/cf0c4562
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/cf0c4562
Branch: refs/heads/TINKERPOP3-923
Commit: cf0c4562d4b992343080e9b5cb5f38fae33c0c3e
Parents: cb97238
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Fri Oct 30 14:37:43 2015 -0600
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Fri Oct 30 14:37:43 2015 -0600
----------------------------------------------------------------------
.../gremlin/spark/structure/io/InputOutputHelper.java | 2 +-
.../gremlin/spark/structure/io/ExampleInputRDD.java | 13 ++++++++-----
.../gremlin/spark/structure/io/ExampleOutputRDD.java | 6 +++++-
3 files changed, 14 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cf0c4562/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputHelper.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputHelper.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputHelper.java
index b1e1b0a..1e279de 100644
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputHelper.java
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputHelper.java
@@ -68,7 +68,7 @@ public final class InputOutputHelper {
final BaseConfiguration newConfiguration = new BaseConfiguration();
newConfiguration.copy(org.apache.tinkerpop.gremlin.hadoop.structure.io.InputOutputHelper.getOutputGraph(configuration, resultGraph, persist).configuration());
if (resultGraph.equals(GraphComputer.ResultGraph.NEW) && hadoopConfiguration.containsKey(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD)) {
- newConfiguration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, InputOutputHelper.getInputFormat((Class) Class.forName(hadoopConfiguration.getProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD).toString())).getCanonicalName());
+ newConfiguration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, InputOutputHelper.getInputFormat((Class) Class.forName(hadoopConfiguration.getString(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD))).getCanonicalName());
if (newConfiguration.getString(Constants.GREMLIN_HADOOP_INPUT_LOCATION, "").endsWith("/" + Constants.HIDDEN_G)) { // Spark RDDs are not namespaced the same as Hadoop
newConfiguration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, newConfiguration.getString(Constants.GREMLIN_HADOOP_INPUT_LOCATION).replace("/" + Constants.HIDDEN_G, ""));
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cf0c4562/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleInputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleInputRDD.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleInputRDD.java
index 8417d1c..86c7610 100644
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleInputRDD.java
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleInputRDD.java
@@ -22,7 +22,6 @@ import org.apache.commons.configuration.Configuration;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
-import org.apache.tinkerpop.gremlin.spark.structure.io.InputRDD;
import org.apache.tinkerpop.gremlin.structure.T;
import org.apache.tinkerpop.gremlin.structure.Vertex;
import org.apache.tinkerpop.gremlin.structure.util.star.StarGraph;
@@ -36,13 +35,17 @@ import java.util.List;
*/
public final class ExampleInputRDD implements InputRDD {
+ static {
+ InputOutputHelper.registerInputOutputPair(ExampleInputRDD.class, ExampleOutputRDD.class);
+ }
+
@Override
public JavaPairRDD<Object, VertexWritable> readGraphRDD(final Configuration configuration, final JavaSparkContext sparkContext) {
final List<Vertex> list = new ArrayList<>();
- list.add(StarGraph.open().addVertex(T.id, 1l, T.label,"person","age", 29));
- list.add(StarGraph.open().addVertex(T.id, 2l, T.label,"person","age", 27));
- list.add(StarGraph.open().addVertex(T.id, 4l, T.label,"person","age", 32));
- list.add(StarGraph.open().addVertex(T.id, 6l, T.label,"person","age", 35));
+ list.add(StarGraph.open().addVertex(T.id, 1l, T.label, "person", "age", 29));
+ list.add(StarGraph.open().addVertex(T.id, 2l, T.label, "person", "age", 27));
+ list.add(StarGraph.open().addVertex(T.id, 4l, T.label, "person", "age", 32));
+ list.add(StarGraph.open().addVertex(T.id, 6l, T.label, "person", "age", 35));
return sparkContext.parallelize(list).mapToPair(vertex -> new Tuple2<>(vertex.id(), new VertexWritable(vertex)));
}
}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cf0c4562/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleOutputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleOutputRDD.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleOutputRDD.java
index 9a9e5ef..103ec20 100644
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleOutputRDD.java
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleOutputRDD.java
@@ -21,7 +21,6 @@ package org.apache.tinkerpop.gremlin.spark.structure.io;
import org.apache.commons.configuration.Configuration;
import org.apache.spark.api.java.JavaPairRDD;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
-import org.apache.tinkerpop.gremlin.spark.structure.io.OutputRDD;
import org.apache.tinkerpop.gremlin.structure.Vertex;
import java.util.Iterator;
@@ -32,6 +31,11 @@ import static org.junit.Assert.assertEquals;
* @author Marko A. Rodriguez (http://markorodriguez.com)
*/
public final class ExampleOutputRDD implements OutputRDD {
+
+ static {
+ InputOutputHelper.registerInputOutputPair(ExampleInputRDD.class, ExampleOutputRDD.class);
+ }
+
@Override
public void writeGraphRDD(final Configuration configuration, final JavaPairRDD<Object, VertexWritable> graphRDD) {
int totalAge = 0;
[44/50] [abbrv] incubator-tinkerpop git commit: Merge branch 'master'
into TINKERPOP3-904
Posted by sp...@apache.org.
Merge branch 'master' into TINKERPOP3-904
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/c4fcae6a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/c4fcae6a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/c4fcae6a
Branch: refs/heads/TINKERPOP3-923
Commit: c4fcae6a942fa67b457d14ef6ac10d98b732164b
Parents: 77fac19 2edefa6
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Wed Nov 4 20:08:20 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Wed Nov 4 20:08:20 2015 +0100
----------------------------------------------------------------------
CHANGELOG.asciidoc | 3 +
docs/src/implementations.asciidoc | 9 ++
.../process/traversal/step/map/MatchStep.java | 46 ++++++---
.../traversal/step/map/MatchStepTest.java | 82 ++++++++++++++-
.../process/computer/SparkGraphComputer.java | 32 ++++++
.../process/computer/LocalPropertyTest.java | 100 +++++++++++++++++++
.../spark/structure/io/InputOutputRDDTest.java | 3 +-
.../spark/structure/io/InputRDDTest.java | 3 +-
.../spark/structure/io/OutputRDDTest.java | 5 +-
.../io/PersistedInputOutputRDDTest.java | 13 ++-
10 files changed, 270 insertions(+), 26 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/c4fcae6a/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --cc CHANGELOG.asciidoc
index 8d74212,085bed7..9f0e18e
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@@ -25,8 -25,10 +25,11 @@@ image::https://raw.githubusercontent.co
TinkerPop 3.1.0 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Optimized `BulkLoaderVertexProgram`. It now uses `EventStrategy` to monitor what the underlying `BulkLoader` implementation does (e.g. whether it creates a new vertex or returns an existing).
* Integrated `NumberHelper` in `SumStep`, `MinStep`, `MaxStep` and `MeanStep` (local and global step variants).
+ * `CountMatchAlgorithm`, in OLAP, now biases traversal selection towards those traversals that start at the current traverser location to reduce message passing.
+ * Fixed a file stream bug in Hadoop OLTP that showed up if the streamed file was more than 2G of data.
+ * Added the ability to set thread local properties in `SparkGraphComputer` when using a persistent context.
* Bumped to Neo4j 2.3.0.
* Added `PersistedInputRDD` and `PersistedOutputRDD` which enables `SparkGraphComputer` to store the graph RDD in the context between jobs (no HDFS serialization required).
* Renamed the `public static String` configuration variable names of TinkerGraph (deprecated old variables).
[40/50] [abbrv] incubator-tinkerpop git commit: relaxed a test
condition to use only KryoSerializer as it seems that GryoSerializer is
causing problems on certain OSs. CTR.
Posted by sp...@apache.org.
relaxed a test condition to use only KryoSerializer as it seems that GryoSerializer is causing problems on certain OSs. CTR.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/ebec0953
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/ebec0953
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/ebec0953
Branch: refs/heads/TINKERPOP3-923
Commit: ebec0953d86ea748e8b1cacebb921642f19954a5
Parents: b338f96
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Wed Nov 4 09:39:23 2015 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Wed Nov 4 09:39:23 2015 -0700
----------------------------------------------------------------------
.../gremlin/spark/structure/io/InputOutputRDDTest.java | 3 ++-
.../gremlin/spark/structure/io/InputRDDTest.java | 3 ++-
.../gremlin/spark/structure/io/OutputRDDTest.java | 5 +++--
.../structure/io/PersistedInputOutputRDDTest.java | 13 ++++++-------
4 files changed, 13 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/ebec0953/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputRDDTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputRDDTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputRDDTest.java
index 1feeb61..3691aba 100644
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputRDDTest.java
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputRDDTest.java
@@ -20,6 +20,7 @@ package org.apache.tinkerpop.gremlin.spark.structure.io;
import org.apache.commons.configuration.BaseConfiguration;
import org.apache.commons.configuration.Configuration;
+import org.apache.spark.serializer.KryoSerializer;
import org.apache.tinkerpop.gremlin.hadoop.Constants;
import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
@@ -40,7 +41,7 @@ public class InputOutputRDDTest {
public void shouldReadFromWriteToArbitraryRDD() throws Exception {
final Configuration configuration = new BaseConfiguration();
configuration.setProperty("spark.master", "local[4]");
- configuration.setProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
+ configuration.setProperty("spark.serializer", KryoSerializer.class.getCanonicalName());
configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, ExampleInputRDD.class.getCanonicalName());
configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, ExampleOutputRDD.class.getCanonicalName());
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/ebec0953/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDDTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDDTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDDTest.java
index e8bd44a..5ba3b12 100644
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDDTest.java
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDDTest.java
@@ -20,6 +20,7 @@ package org.apache.tinkerpop.gremlin.spark.structure.io;
import org.apache.commons.configuration.BaseConfiguration;
import org.apache.commons.configuration.Configuration;
+import org.apache.spark.serializer.KryoSerializer;
import org.apache.tinkerpop.gremlin.hadoop.Constants;
import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat;
@@ -40,7 +41,7 @@ public class InputRDDTest {
public void shouldReadFromArbitraryRDD() {
final Configuration configuration = new BaseConfiguration();
configuration.setProperty("spark.master", "local[4]");
- configuration.setProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
+ configuration.setProperty("spark.serializer", KryoSerializer.class.getCanonicalName());
configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, ExampleInputRDD.class.getCanonicalName());
configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT, GryoOutputFormat.class.getCanonicalName());
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/ebec0953/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputRDDTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputRDDTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputRDDTest.java
index 43dcdbd..10eecb3 100644
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputRDDTest.java
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputRDDTest.java
@@ -20,6 +20,7 @@ package org.apache.tinkerpop.gremlin.spark.structure.io;
import org.apache.commons.configuration.BaseConfiguration;
import org.apache.commons.configuration.Configuration;
+import org.apache.spark.serializer.KryoSerializer;
import org.apache.tinkerpop.gremlin.hadoop.Constants;
import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat;
@@ -27,8 +28,8 @@ import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
import org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
import org.apache.tinkerpop.gremlin.process.traversal.engine.ComputerTraversalEngine;
-import org.apache.tinkerpop.gremlin.spark.process.computer.SparkHadoopGraphProvider;
import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;
+import org.apache.tinkerpop.gremlin.spark.process.computer.SparkHadoopGraphProvider;
import org.apache.tinkerpop.gremlin.structure.Graph;
import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
import org.junit.Test;
@@ -42,7 +43,7 @@ public class OutputRDDTest {
public void shouldWriteToArbitraryRDD() throws Exception {
final Configuration configuration = new BaseConfiguration();
configuration.setProperty("spark.master", "local[4]");
- configuration.setProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
+ configuration.setProperty("spark.serializer", KryoSerializer.class.getCanonicalName());
configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/ebec0953/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedInputOutputRDDTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedInputOutputRDDTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedInputOutputRDDTest.java
index 6ee7aaa..6aeb864 100644
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedInputOutputRDDTest.java
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedInputOutputRDDTest.java
@@ -24,6 +24,7 @@ import org.apache.commons.configuration.Configuration;
import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.serializer.KryoSerializer;
import org.apache.tinkerpop.gremlin.hadoop.Constants;
import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat;
@@ -36,9 +37,6 @@ import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSo
import org.apache.tinkerpop.gremlin.process.traversal.engine.ComputerTraversalEngine;
import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;
import org.apache.tinkerpop.gremlin.spark.process.computer.SparkHadoopGraphProvider;
-import org.apache.tinkerpop.gremlin.spark.structure.io.PersistedInputRDD;
-import org.apache.tinkerpop.gremlin.spark.structure.io.PersistedOutputRDD;
-import org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer;
import org.apache.tinkerpop.gremlin.structure.Graph;
import org.apache.tinkerpop.gremlin.structure.io.IoCore;
import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
@@ -55,11 +53,12 @@ import static org.junit.Assert.*;
public class PersistedInputOutputRDDTest {
@Test
+
public void shouldNotPersistRDDAcrossJobs() throws Exception {
final String rddName = "target/test-output/" + UUID.randomUUID();
final Configuration configuration = new BaseConfiguration();
configuration.setProperty("spark.master", "local[4]");
- configuration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
+ configuration.setProperty("spark.serializer", KryoSerializer.class.getCanonicalName());
configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
@@ -88,7 +87,7 @@ public class PersistedInputOutputRDDTest {
final String rddName = "target/test-output/" + UUID.randomUUID();
final Configuration configuration = new BaseConfiguration();
configuration.setProperty("spark.master", "local[4]");
- configuration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
+ configuration.setProperty("spark.serializer", KryoSerializer.class.getCanonicalName());
configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
@@ -130,7 +129,7 @@ public class PersistedInputOutputRDDTest {
final String rddName = "target/test-output/" + UUID.randomUUID().toString();
final Configuration readConfiguration = new BaseConfiguration();
readConfiguration.setProperty("spark.master", "local[4]");
- readConfiguration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
+ readConfiguration.setProperty("spark.serializer", KryoSerializer.class.getCanonicalName());
readConfiguration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
readConfiguration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
readConfiguration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
@@ -177,7 +176,7 @@ public class PersistedInputOutputRDDTest {
final String rddName = "target/test-output/" + UUID.randomUUID().toString();
final Configuration readConfiguration = new BaseConfiguration();
readConfiguration.setProperty("spark.master", "local[4]");
- readConfiguration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
+ readConfiguration.setProperty("spark.serializer", KryoSerializer.class.getCanonicalName());
readConfiguration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
readConfiguration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
readConfiguration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
[42/50] [abbrv] incubator-tinkerpop git commit: Added "Getting
Started" to the developer docs for new contributors.
Posted by sp...@apache.org.
Added "Getting Started" to the developer docs for new contributors.
Includes notes on low-hanging fruit = trivial priority in jira as discussed on the dev mailing list.
https://pony-poc.apache.org/thread.html/da7265b799b4bfca98f0094cc9df83d3c67e1de507aaeb6c1015e32a@1445013066@%3Cdev.tinkerpop.apache.org%3E
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/bfb99f3b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/bfb99f3b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/bfb99f3b
Branch: refs/heads/TINKERPOP3-923
Commit: bfb99f3bbac6e811b3a880f13fa6cc9e2bb64640
Parents: ebec095
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Wed Nov 4 13:46:02 2015 -0500
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Wed Nov 4 13:46:02 2015 -0500
----------------------------------------------------------------------
docs/src/developer-contributing.asciidoc | 27 +++++++++++++++++++++++----
1 file changed, 23 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/bfb99f3b/docs/src/developer-contributing.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/developer-contributing.asciidoc b/docs/src/developer-contributing.asciidoc
index 951c1b0..acf5dc8 100644
--- a/docs/src/developer-contributing.asciidoc
+++ b/docs/src/developer-contributing.asciidoc
@@ -17,10 +17,26 @@ limitations under the License.
Contributing
============
-Contributions via GitHub pull requests are gladly accepted from their original
-author. By submitting any copyrighted material via pull request, email, or other means
-you agree to license the material under the project's open source license and
-warrant that you have the legal authority to do so.
+Contributions via GitHub pull requests are gladly accepted from their original author. By submitting any copyrighted
+material via pull request, email, or other means you agree to license the material under the project's open source
+license and warrant that you have the legal authority to do so.
+
+Getting Started
+---------------
+
+New contributors can start development with TinkerPop by first link:https://help.github.com/articles/fork-a-repo/[forking
+then cloning] the Apache TinkerPop link:https://github.com/apache/incubator-tinkerpop[GitHub repository]. Generally
+speaking it is best to tie any work done to an issue in link:https://issues.apache.org/jira/browse/TINKERPOP3[JIRA].
+Either scan through JIRA for an existing open issue to work on or create a new one.
+
+NOTE: For those who are trying to find a place to start to contribute, consider looking at unresolved issues that
+have the "trivial" priority as these issues are specifically set aside as
+link:https://issues.apache.org/jira/issues/?jql=project%20%3D%20TINKERPOP3%20AND%20resolution%20%3D%20Unresolved%20AND%20priority%20%3D%20Trivial%20ORDER%20BY%20key%20DESC[low-hanging fruit]
+for newcomers.
+
+After making changes, submit a link:https://help.github.com/articles/using-pull-requests/[pull request] through
+GitHub, where the name of the pull request is prefixed with the JIRA issue number. In this way, the pull request
+and its comments get tied back to the JIRA issue it references.
Building and Testing
--------------------
@@ -161,6 +177,9 @@ as one that is representative of a change in the API that might affect users or
organizing release notes.
* The "affects/fix version(s)" fields should be appropriately set, where the "fix version" implies the version on
which that particular issue will completed.
+* The "priority" field can be arbitrarily applied with one exception. The "trivial" option should be reserved for
+tasks that are "easy" for a potential new contributor to jump into and do not have significant impact to urgently
+required improvements.
Code Style
~~~~~~~~~~
[49/50] [abbrv] incubator-tinkerpop git commit: Merge remote-tracking
branch 'origin/master' into TINKERPOP3-923
Posted by sp...@apache.org.
Merge remote-tracking branch 'origin/master' into TINKERPOP3-923
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/2b7dc773
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/2b7dc773
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/2b7dc773
Branch: refs/heads/TINKERPOP3-923
Commit: 2b7dc773bd6a8df837b6d4da36c27b5c2d9c04c7
Parents: 3326fcc 151f64b
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Wed Nov 4 18:08:57 2015 -0500
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Wed Nov 4 18:08:57 2015 -0500
----------------------------------------------------------------------
CHANGELOG.asciidoc | 11 +
docs/src/developer-contributing.asciidoc | 60 ++++--
docs/src/developer-meetings.asciidoc | 64 ++++++
docs/src/developer-release.asciidoc | 13 +-
docs/src/developer.asciidoc | 4 +-
docs/src/gremlin-applications.asciidoc | 7 +
docs/src/implementations.asciidoc | 45 ++--
docs/src/the-graphcomputer.asciidoc | 4 +
docs/src/the-traversal.asciidoc | 36 +++-
.../upgrade-release-3.1.x-incubating.asciidoc | 25 ++-
docs/static/images/gremlin-standing-strong.png | Bin 0 -> 30985 bytes
.../process/computer/GiraphGraphComputer.java | 16 +-
.../computer/io/GiraphVertexInputFormat.java | 70 ------
.../computer/io/GiraphVertexOutputFormat.java | 65 ------
.../process/computer/io/GiraphVertexReader.java | 67 ------
.../process/computer/io/GiraphVertexWriter.java | 57 -----
.../structure/io/GiraphVertexInputFormat.java | 70 ++++++
.../structure/io/GiraphVertexOutputFormat.java | 65 ++++++
.../giraph/structure/io/GiraphVertexReader.java | 67 ++++++
.../giraph/structure/io/GiraphVertexWriter.java | 57 +++++
.../GephiTraversalVisualizationStrategy.groovy | 2 +-
.../gremlin/process/computer/GraphComputer.java | 19 +-
.../computer/bulkloading/BulkLoader.java | 31 +++
.../bulkloading/BulkLoaderVertexProgram.java | 115 ++++++++--
.../bulkloading/IncrementalBulkLoader.java | 14 +-
.../traversal/TraversalVertexProgram.java | 4 +-
.../gremlin/process/traversal/NumberHelper.java | 12 +-
.../traversal/dsl/graph/GraphTraversal.java | 23 +-
.../dsl/graph/GraphTraversalSource.java | 10 +-
.../gremlin/process/traversal/dsl/graph/__.java | 9 +-
.../process/traversal/step/map/GraphStep.java | 149 +++++++++++++
.../process/traversal/step/map/MatchStep.java | 46 ++--
.../traversal/step/map/MaxGlobalStep.java | 14 +-
.../traversal/step/map/MaxLocalStep.java | 5 +-
.../traversal/step/map/MeanGlobalStep.java | 61 +++---
.../traversal/step/map/MeanLocalStep.java | 11 +-
.../traversal/step/map/MinGlobalStep.java | 14 +-
.../traversal/step/map/MinLocalStep.java | 5 +-
.../traversal/step/map/SumGlobalStep.java | 25 ++-
.../traversal/step/map/SumLocalStep.java | 5 +-
.../traversal/step/sideEffect/GraphStep.java | 101 ---------
.../step/util/TraversalComparator.java | 5 +-
.../strategy/decoration/ConnectiveStrategy.java | 3 +-
.../strategy/decoration/ElementIdStrategy.java | 2 +-
.../strategy/decoration/PartitionStrategy.java | 2 +-
.../strategy/decoration/SubgraphStrategy.java | 2 +-
.../finalization/LazyBarrierStrategy.java | 2 +-
.../ComputerVerificationStrategy.java | 39 +++-
.../process/traversal/util/TraversalHelper.java | 3 +
.../traversal/step/map/GraphStepTest.java | 41 ++++
.../traversal/step/map/MatchStepTest.java | 82 ++++++-
.../traversal/step/map/MeanGlobalStepTest.java | 14 ++
.../traversal/step/map/MeanLocalStepTest.java | 14 ++
.../ElementIdStrategyTraverseTest.java | 4 +-
.../PartitionStrategyTraverseTest.java | 4 +-
.../SubgraphStrategyTraverseTest.java | 5 +-
.../traversal/step/map/GroovyGraphTest.groovy | 50 +++++
.../traversal/step/map/GroovyOrderTest.groovy | 10 +
.../step/sideEffect/GroovyGroupTest.groovy | 2 +-
.../process/GroovyProcessComputerSuite.java | 2 +
.../process/GroovyProcessStandardSuite.java | 2 +
.../gremlin/process/ProcessComputerSuite.java | 2 +
.../gremlin/process/ProcessStandardSuite.java | 2 +
.../process/computer/GraphComputerTest.java | 34 +--
.../traversal/step/map/AddVertexTest.java | 2 +-
.../process/traversal/step/map/GraphTest.java | 109 ++++++++++
.../process/traversal/step/map/OrderTest.java | 58 ++++-
.../process/traversal/step/map/SumTest.java | 2 +-
.../traversal/step/sideEffect/GroupTest.java | 22 +-
...ComputerVerificationStrategyProcessTest.java | 3 +-
.../tinkerpop/gremlin/hadoop/Constants.java | 1 -
.../structure/hdfs/HadoopElementIterator.java | 4 +-
.../hadoop/structure/io/InputOutputHelper.java | 22 ++
.../hadoop/structure/util/HadoopHelper.java | 50 -----
.../groovy/plugin/HadoopGremlinPluginTest.java | 2 +-
neo4j-gremlin/pom.xml | 28 ++-
.../step/sideEffect/Neo4jGraphStep.java | 10 +-
.../optimization/Neo4jGraphStepStrategy.java | 13 +-
.../gremlin/neo4j/structure/Neo4jGraph.java | 1 +
.../gremlin/neo4j/structure/Neo4jHelper.java | 16 +-
.../traversal/strategy/Neo4jStrategySuite.java | 44 ++++
.../traversal/strategy/Neo4jStrategyTest.java | 32 +++
.../Neo4jGraphStepStrategyTest.java | 76 +++++++
.../process/computer/SparkGraphComputer.java | 102 +++++++--
.../process/computer/io/InputFormatRDD.java | 47 ----
.../spark/process/computer/io/InputRDD.java | 41 ----
.../process/computer/io/OutputFormatRDD.java | 49 -----
.../spark/process/computer/io/OutputRDD.java | 31 ---
.../spark/structure/io/InputFormatRDD.java | 47 ++++
.../spark/structure/io/InputOutputHelper.java | 81 +++++++
.../gremlin/spark/structure/io/InputRDD.java | 41 ++++
.../spark/structure/io/OutputFormatRDD.java | 49 +++++
.../gremlin/spark/structure/io/OutputRDD.java | 31 +++
.../spark/structure/io/PersistedInputRDD.java | 60 ++++++
.../spark/structure/io/PersistedOutputRDD.java | 41 ++++
.../process/computer/LocalPropertyTest.java | 100 +++++++++
.../process/computer/io/ExampleInputRDD.java | 47 ----
.../process/computer/io/ExampleOutputRDD.java | 45 ----
.../process/computer/io/InputOutputRDDTest.java | 59 -----
.../spark/process/computer/io/InputRDDTest.java | 54 -----
.../process/computer/io/OutputRDDTest.java | 62 ------
.../spark/structure/io/ExampleInputRDD.java | 51 +++++
.../spark/structure/io/ExampleOutputRDD.java | 50 +++++
.../spark/structure/io/InputOutputRDDTest.java | 60 ++++++
.../spark/structure/io/InputRDDTest.java | 55 +++++
.../spark/structure/io/OutputRDDTest.java | 63 ++++++
.../io/PersistedInputOutputRDDTest.java | 216 +++++++++++++++++++
.../step/sideEffect/TinkerGraphStep.java | 10 +-
.../optimization/TinkerGraphStepStrategy.java | 13 +-
.../tinkergraph/structure/TinkerGraph.java | 1 +
.../strategy/TinkerGraphStrategySuite.java | 44 ++++
.../strategy/TinkerGraphStrategyTest.java | 32 +++
.../TinkerGraphStepStrategyTest.java | 33 ++-
.../structure/TinkerGraphPlayTest.java | 41 ++--
114 files changed, 2814 insertions(+), 1141 deletions(-)
----------------------------------------------------------------------
[04/50] [abbrv] incubator-tinkerpop git commit: TINKERPOP3-935;
Added a missing table for the close op of the SessionOpProcessor
Posted by sp...@apache.org.
TINKERPOP3-935; Added a missing table for the close op of the SessionOpProcessor
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/d93f1f76
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/d93f1f76
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/d93f1f76
Branch: refs/heads/TINKERPOP3-923
Commit: d93f1f7630eb1fab01dc3d73cb35a2b60a39b379
Parents: ac0924d
Author: Dylan Millikin <dy...@brightzone.fr>
Authored: Thu Oct 29 23:48:23 2015 +0100
Committer: Dylan Millikin <dy...@brightzone.fr>
Committed: Thu Oct 29 23:48:23 2015 +0100
----------------------------------------------------------------------
docs/src/gremlin-applications.asciidoc | 7 +++++++
1 file changed, 7 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d93f1f76/docs/src/gremlin-applications.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/gremlin-applications.asciidoc b/docs/src/gremlin-applications.asciidoc
index 34b760d..88684ab 100644
--- a/docs/src/gremlin-applications.asciidoc
+++ b/docs/src/gremlin-applications.asciidoc
@@ -1350,6 +1350,13 @@ session state is not shared among them.
|language |String |The flavor used (e.g. `gremlin-groovy`)
|=========================================================
+'`close` operation arguments'
+[width="100%",cols="2,2,9",options="header"]
+|=========================================================
+|Key |Type |Description
+|session |String | *Required* The session identifier for the session to close.
+|=========================================================
+
Authentication
^^^^^^^^^^^^^^
[26/50] [abbrv] incubator-tinkerpop git commit: Optimized BLVP
Posted by sp...@apache.org.
Optimized BLVP
BLVP now uses EventStrategy to monitor what the actual BulkLoader implementation does (e.g. whether it creates a new vertex or just returns an existing one).
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/8b8222da
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/8b8222da
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/8b8222da
Branch: refs/heads/TINKERPOP3-923
Commit: 8b8222da7fed57fa1e7e0232ab788944716404f7
Parents: 19b7ae0
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Tue Nov 3 15:14:12 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Tue Nov 3 15:14:12 2015 +0100
----------------------------------------------------------------------
.../computer/bulkloading/BulkLoader.java | 31 +++++
.../bulkloading/BulkLoaderVertexProgram.java | 116 ++++++++++++++++---
.../bulkloading/IncrementalBulkLoader.java | 14 ++-
.../structure/TinkerGraphPlayTest.java | 30 ++---
4 files changed, 158 insertions(+), 33 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/8b8222da/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoader.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoader.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoader.java
index dd5eaf4..2640d2e 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoader.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoader.java
@@ -41,6 +41,22 @@ public interface BulkLoader {
public Vertex getOrCreateVertex(final Vertex vertex, final Graph graph, final GraphTraversalSource g);
/**
+ * Creates a clone of the given edge between the given in- and out-vertices.
+ *
+ * @param edge The edge to be cloned.
+ * @param outVertex The out-vertex in the given graph..
+ * @param inVertex The in-vertex in the given graph.
+ * @param graph The graph that holds the cloned edge after this method was called.
+ * @param g A standard traversal source for the given graph.
+ * @return The cloned edge.
+ */
+ public default Edge createEdge(final Edge edge, final Vertex outVertex, final Vertex inVertex, final Graph graph, final GraphTraversalSource g) {
+ final Edge result = outVertex.addEdge(edge.label(), inVertex);
+ edge.properties().forEachRemaining(property -> result.property(property.key(), property.value()));
+ return result;
+ }
+
+ /**
* Gets or creates a clone of the given edge between the given in- and out-vertices.
*
* @param edge The edge to be cloned.
@@ -53,6 +69,21 @@ public interface BulkLoader {
public Edge getOrCreateEdge(final Edge edge, final Vertex outVertex, final Vertex inVertex, final Graph graph, final GraphTraversalSource g);
/**
+ * Creates a clone of the given property for the given vertex.
+ *
+ * @param property The property to be cloned.
+ * @param vertex The vertex in the given graph..
+ * @param graph The graph that holds the given vertex.
+ * @param g A standard traversal source for the given graph.
+ * @return The cloned property.
+ */
+ public default VertexProperty createVertexProperty(final VertexProperty<?> property, final Vertex vertex, final Graph graph, final GraphTraversalSource g) {
+ final VertexProperty result = vertex.property(property.key(), property.value());
+ property.properties().forEachRemaining(metaProperty -> result.property(metaProperty.key(), metaProperty.value()));
+ return result;
+ }
+
+ /**
* Gets or creates a clone of the given property for the given vertex.
*
* @param property The property to be cloned.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/8b8222da/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgram.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgram.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgram.java
index 99bb1f5..2950bfd 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgram.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgram.java
@@ -31,8 +31,12 @@ import org.apache.tinkerpop.gremlin.process.computer.VertexProgram;
import org.apache.tinkerpop.gremlin.process.computer.util.AbstractVertexProgramBuilder;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__;
+import org.apache.tinkerpop.gremlin.process.traversal.step.util.event.MutationListener;
+import org.apache.tinkerpop.gremlin.process.traversal.strategy.decoration.EventStrategy;
import org.apache.tinkerpop.gremlin.structure.Direction;
+import org.apache.tinkerpop.gremlin.structure.Edge;
import org.apache.tinkerpop.gremlin.structure.Graph;
+import org.apache.tinkerpop.gremlin.structure.Property;
import org.apache.tinkerpop.gremlin.structure.Vertex;
import org.apache.tinkerpop.gremlin.structure.VertexProperty;
import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
@@ -48,7 +52,6 @@ import java.util.HashSet;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
-import java.util.concurrent.atomic.AtomicLong;
/**
* @author Daniel Kuppitz (http://gremlin.guru)
@@ -74,12 +77,7 @@ public class BulkLoaderVertexProgram implements VertexProgram<Tuple> {
private GraphTraversalSource g;
private long intermediateBatchSize;
- private static final ThreadLocal<AtomicLong> counter = new ThreadLocal<AtomicLong>() {
- @Override
- protected AtomicLong initialValue() {
- return new AtomicLong();
- }
- };
+ private BulkLoadingListener listener;
private BulkLoaderVertexProgram() {
messageScope = MessageScope.Local.of(__::inE);
@@ -116,17 +114,19 @@ public class BulkLoaderVertexProgram implements VertexProgram<Tuple> {
* @param close Whether to close the current graph instance after calling commit() or not.
*/
private void commit(final boolean close) {
- if (!close && (intermediateBatchSize == 0L || counter.get().incrementAndGet() % intermediateBatchSize != 0))
+ if (!close && (intermediateBatchSize == 0L || listener.mutations() < intermediateBatchSize))
return;
if (null != graph) {
if (graph.features().graph().supportsTransactions()) {
- LOGGER.info("Committing transaction on Graph instance: {} [{}]", graph, counter.get().get());
+ LOGGER.info("Committing transaction on Graph instance: {} [{} mutations]", graph, listener.mutations());
try {
graph.tx().commit();
LOGGER.debug("Committed transaction on Graph instance: {}", graph);
+ listener.resetCounter();
} catch (Exception e) {
LOGGER.error("Failed to commit transaction on Graph instance: {}", graph);
graph.tx().rollback();
+ listener.resetCounter();
throw e;
}
}
@@ -144,7 +144,6 @@ public class BulkLoaderVertexProgram implements VertexProgram<Tuple> {
@Override
public void setup(final Memory memory) {
- counter.get().set(0L);
}
@Override
@@ -172,7 +171,8 @@ public class BulkLoaderVertexProgram implements VertexProgram<Tuple> {
graph = GraphFactory.open(configuration.subset(WRITE_GRAPH_CFG_KEY));
LOGGER.info("Opened Graph instance: {}", graph);
try {
- g = graph.traversal();
+ listener = new BulkLoadingListener();
+ g = GraphTraversalSource.build().with(EventStrategy.build().addListener(listener).create()).create(graph);
} catch (Exception e) {
try {
graph.close();
@@ -205,12 +205,15 @@ public class BulkLoaderVertexProgram implements VertexProgram<Tuple> {
private void executeInternal(final Vertex sourceVertex, final Messenger<Tuple> messenger, final Memory memory) {
if (memory.isInitialIteration()) {
+ this.listener.resetStats();
// get or create the vertex
final Vertex targetVertex = bulkLoader.getOrCreateVertex(sourceVertex, graph, g);
// write all the properties of the vertex to the newly created vertex
final Iterator<VertexProperty<Object>> vpi = sourceVertex.properties();
- while (vpi.hasNext()) {
- bulkLoader.getOrCreateVertexProperty(vpi.next(), targetVertex, graph, g);
+ if (this.listener.isNewVertex()) {
+ vpi.forEachRemaining(vp -> bulkLoader.createVertexProperty(vp, targetVertex, graph, g));
+ } else {
+ vpi.forEachRemaining(vp -> bulkLoader.getOrCreateVertexProperty(vp, targetVertex, graph, g));
}
this.commit(false);
if (!bulkLoader.useUserSuppliedIds()) {
@@ -221,9 +224,14 @@ public class BulkLoaderVertexProgram implements VertexProgram<Tuple> {
} else if (memory.getIteration() == 1) {
if (bulkLoader.useUserSuppliedIds()) {
final Vertex outV = bulkLoader.getVertex(sourceVertex, graph, g);
+ final boolean incremental = outV.edges(Direction.OUT).hasNext();
sourceVertex.edges(Direction.OUT).forEachRemaining(edge -> {
final Vertex inV = bulkLoader.getVertex(edge.inVertex(), graph, g);
- bulkLoader.getOrCreateEdge(edge, outV, inV, graph, g);
+ if (incremental) {
+ bulkLoader.getOrCreateEdge(edge, outV, inV, graph, g);
+ } else {
+ bulkLoader.createEdge(edge, outV, inV, graph, g);
+ }
this.commit(false);
});
} else {
@@ -407,4 +415,84 @@ public class BulkLoaderVertexProgram implements VertexProgram<Tuple> {
}
};
}
+
+ static class BulkLoadingListener implements MutationListener {
+
+ private long counter;
+ private boolean isNewVertex;
+
+ public BulkLoadingListener() {
+ this.counter = 0L;
+ this.isNewVertex = false;
+ ;
+ }
+
+ public boolean isNewVertex() {
+ return this.isNewVertex;
+ }
+
+ public long mutations() {
+ return this.counter;
+ }
+
+ public void resetStats() {
+ this.isNewVertex = false;
+ }
+
+ public void resetCounter() {
+ this.counter = 0L;
+ }
+
+ @Override
+ public void vertexAdded(final Vertex vertex) {
+ this.isNewVertex = true;
+ this.counter++;
+ }
+
+ @Override
+ public void vertexRemoved(final Vertex vertex) {
+ this.counter++;
+ }
+
+ @Override
+ public void vertexPropertyChanged(final Vertex element, final Property oldValue, final Object setValue,
+ final Object... vertexPropertyKeyValues) {
+ this.counter++;
+ }
+
+ @Override
+ public void vertexPropertyRemoved(final VertexProperty vertexProperty) {
+ this.counter++;
+ }
+
+ @Override
+ public void edgeAdded(final Edge edge) {
+ this.counter++;
+ }
+
+ @Override
+ public void edgeRemoved(final Edge edge) {
+ this.counter++;
+ }
+
+ @Override
+ public void edgePropertyChanged(final Edge element, final Property oldValue, final Object setValue) {
+ this.counter++;
+ }
+
+ @Override
+ public void edgePropertyRemoved(final Edge element, final Property property) {
+ this.counter++;
+ }
+
+ @Override
+ public void vertexPropertyPropertyChanged(final VertexProperty element, final Property oldValue, final Object setValue) {
+ this.counter++;
+ }
+
+ @Override
+ public void vertexPropertyPropertyRemoved(final VertexProperty element, final Property property) {
+ this.counter++;
+ }
+ }
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/8b8222da/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/IncrementalBulkLoader.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/IncrementalBulkLoader.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/IncrementalBulkLoader.java
index f03cd18..8219515 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/IncrementalBulkLoader.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/IncrementalBulkLoader.java
@@ -51,8 +51,10 @@ public class IncrementalBulkLoader implements BulkLoader {
return iterator.hasNext()
? iterator.next()
: useUserSuppliedIds()
- ? graph.addVertex(T.id, vertex.id(), T.label, vertex.label())
- : graph.addVertex(T.label, vertex.label(), getVertexIdProperty(), vertex.id());
+ ? g.addV(vertex.label()).property(T.id, vertex.id()).next()
+ : g.addV(vertex.label()).property(getVertexIdProperty(), vertex.id()).next();
+ //? graph.addVertex(T.id, vertex.id(), T.label, vertex.label())
+ //: graph.addVertex(T.label, vertex.label(), getVertexIdProperty(), vertex.id());
}
/**
@@ -71,8 +73,7 @@ public class IncrementalBulkLoader implements BulkLoader {
}
});
} else {
- e = outVertex.addEdge(edge.label(), inVertex);
- edge.properties().forEachRemaining(property -> e.property(property.key(), property.value()));
+ e = createEdge(edge, outVertex, inVertex, graph, g);
}
return e;
}
@@ -84,7 +85,10 @@ public class IncrementalBulkLoader implements BulkLoader {
public VertexProperty getOrCreateVertexProperty(final VertexProperty<?> property, final Vertex vertex, final Graph graph, final GraphTraversalSource g) {
final VertexProperty<?> vp;
final VertexProperty<?> existing = vertex.property(property.key());
- if (!existing.isPresent() || !existing.value().equals(property.value())) {
+ if (!existing.isPresent()) {
+ return createVertexProperty(property, vertex, graph, g);
+ }
+ if (!existing.value().equals(property.value())) {
vp = vertex.property(property.key(), property.value());
} else {
vp = existing;
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/8b8222da/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java
----------------------------------------------------------------------
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java
index 06994e2..e7c54ce 100644
--- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java
+++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java
@@ -18,6 +18,7 @@
*/
package org.apache.tinkerpop.gremlin.tinkergraph.structure;
+import org.apache.tinkerpop.gremlin.process.computer.bulkloading.BulkLoaderVertexProgram;
import org.apache.tinkerpop.gremlin.process.traversal.Operator;
import org.apache.tinkerpop.gremlin.process.traversal.P;
import org.apache.tinkerpop.gremlin.process.traversal.Scope;
@@ -30,10 +31,12 @@ import org.apache.tinkerpop.gremlin.structure.Graph;
import org.apache.tinkerpop.gremlin.structure.T;
import org.apache.tinkerpop.gremlin.structure.Vertex;
import org.apache.tinkerpop.gremlin.structure.io.graphml.GraphMLIo;
+import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
import org.apache.tinkerpop.gremlin.util.TimeUtil;
import org.junit.Ignore;
import org.junit.Test;
+import java.io.File;
import java.util.Arrays;
import java.util.List;
import java.util.function.Supplier;
@@ -192,20 +195,19 @@ public class TinkerGraphPlayTest {
@Test
@Ignore
public void testPlayDK() throws Exception {
- final Graph graph = TinkerFactory.createModern();
- final GraphTraversalSource g = graph.traversal();
- Traversal traversal = g.V().where(out().and().in()).profile().cap(TraversalMetrics.METRICS_KEY);
- //traversal.forEachRemaining(System.out::println);
- System.out.println(traversal.toString());
- traversal.asAdmin().applyStrategies();
- System.out.println(traversal.toString());
- traversal.forEachRemaining(System.out::println);
- traversal = g.V().where(and(out(), in())).profile().cap(TraversalMetrics.METRICS_KEY);
- //traversal.forEachRemaining(System.out::println);
- System.out.println(traversal.toString());
- traversal.asAdmin().applyStrategies();
- System.out.println(traversal.toString());
- //System.out.println(traversal.toString());
+
+ new File("/tmp/tinkergraph2.kryo").deleteOnExit();
+ new File("/tmp/tinkergraph3.kryo").deleteOnExit();
+
+ final Graph graph1 = TinkerFactory.createModern();
+ final Graph graph2 = GraphFactory.open("/tmp/graph2.properties");
+ TinkerFactory.generateModern((TinkerGraph) graph2);
+ graph2.close();
+
+ System.out.println("graph1 -> graph2");
+ graph1.compute().workers(1).program(BulkLoaderVertexProgram.build().userSuppliedIds(true).writeGraph("/tmp/graph2.properties").create(graph1)).submit().get();
+ System.out.println("graph1 -> graph3");
+ graph1.compute().workers(1).program(BulkLoaderVertexProgram.build().userSuppliedIds(true).writeGraph("/tmp/graph3.properties").create(graph1)).submit().get();
}
@Test
[20/50] [abbrv] incubator-tinkerpop git commit: Merge branch
'TINKERPOP3-899' of https://github.com/apache/incubator-tinkerpop
Posted by sp...@apache.org.
Merge branch 'TINKERPOP3-899' of https://github.com/apache/incubator-tinkerpop
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/19b7ae0f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/19b7ae0f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/19b7ae0f
Branch: refs/heads/TINKERPOP3-923
Commit: 19b7ae0fc2018375ac148481c247b5ccf88e1d29
Parents: 6169fff 1a31b78
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Sun Nov 1 12:22:04 2015 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Sun Nov 1 12:22:04 2015 -0700
----------------------------------------------------------------------
CHANGELOG.asciidoc | 3 ++-
neo4j-gremlin/pom.xml | 28 +++++++++++++++++++-
.../gremlin/neo4j/structure/Neo4jHelper.java | 16 ++++++-----
3 files changed, 38 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
[14/50] [abbrv] incubator-tinkerpop git commit: Merge branch
'TINKERPOP3-935' of https://github.com/PommeVerte/incubator-tinkerpop
Posted by sp...@apache.org.
Merge branch 'TINKERPOP3-935' of https://github.com/PommeVerte/incubator-tinkerpop
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/6169fffd
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/6169fffd
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/6169fffd
Branch: refs/heads/TINKERPOP3-923
Commit: 6169fffd27b1994c2bbf75ece6d6f252ef2fadcd
Parents: 9300485 d93f1f7
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Fri Oct 30 21:33:39 2015 -0400
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Fri Oct 30 21:33:39 2015 -0400
----------------------------------------------------------------------
docs/src/gremlin-applications.asciidoc | 7 +++++++
1 file changed, 7 insertions(+)
----------------------------------------------------------------------
[07/50] [abbrv] incubator-tinkerpop git commit: This is a masterpiece
here. PersistedXXXRDD is now a Spark RDD class where the inputLocation
(outputLocation) are the names of the RDD. No HDFS is used between jobs as
the graphRDD is stored in the SparkSer
Posted by sp...@apache.org.
This is a masterpiece here. PersistedXXXRDD is now a Spark RDD class where the inputLocation (outputLocation) are the names of the RDD. No HDFS is used between jobs as the graphRDD is stored in the SparkServer using a persisted context. Added test cases, renamed GraphComputer.config() to configure() to be consistent with the naming conventions of GraphComputer methods. Also made it default as most implementaitons won't need it and there is no point to require a random return this. Updated docs accordingly.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/528ba027
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/528ba027
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/528ba027
Branch: refs/heads/TINKERPOP3-923
Commit: 528ba027a098bc722211767b11b7dc010fb2cba1
Parents: 16b5005
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Fri Oct 30 11:48:42 2015 -0600
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Fri Oct 30 11:48:42 2015 -0600
----------------------------------------------------------------------
CHANGELOG.asciidoc | 3 +-
docs/src/implementations.asciidoc | 32 ++--
docs/src/the-traversal.asciidoc | 11 +-
.../upgrade-release-3.1.x-incubating.asciidoc | 8 +
.../process/computer/GiraphGraphComputer.java | 2 +-
.../gremlin/process/computer/GraphComputer.java | 5 +-
.../process/computer/GraphComputerTest.java | 2 +-
.../tinkerpop/gremlin/hadoop/Constants.java | 3 -
.../process/computer/SparkGraphComputer.java | 78 ++++-----
.../process/computer/io/PersistedInputRDD.java | 60 +++++++
.../process/computer/io/PersistedOutputRDD.java | 41 +++++
.../process/computer/util/SparkHelper.java | 49 ------
.../io/PersistedInputOutputRDDTest.java | 168 +++++++++++++++++++
.../io/SparkContextPersistenceTest.java | 126 --------------
.../process/computer/TinkerGraphComputer.java | 5 -
15 files changed, 347 insertions(+), 246 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/528ba027/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index 8bd6e51..7ee21ad 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -25,8 +25,9 @@ image::https://raw.githubusercontent.com/apache/incubator-tinkerpop/master/docs/
TinkerPop 3.1.0 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Added `PersistedInputRDD` and `PersistedOutputRDD` which enables `SparkGraphComputer` to store the graph RDD in the context between jobs.
* Renamed the `public static String` configuration variable names of TinkerGraph (deprecated old variables).
-* Added `GraphComputer.config(key,value)` to allow engine-specific configurations.
+* Added `GraphComputer.configure(key,value)` to allow engine-specific configurations.
* `GraphStep` is no longer in the `sideEffect`-package and is now in `map`-package (breaking change).
* Added suppport for mid-traversal `V()`-steps (`GraphStep` semantics updated).
* Fixed `Number` handling in `Operator` enums. Prior this change a lot of operations on mixed `Number` types returned a wrong result (wrong data type).
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/528ba027/docs/src/implementations.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/implementations.asciidoc b/docs/src/implementations.asciidoc
index 90ade3f..8110c5f 100644
--- a/docs/src/implementations.asciidoc
+++ b/docs/src/implementations.asciidoc
@@ -1196,11 +1196,16 @@ image::spark-algorithm.png[width=775]
|gremlin.spark.persistContext |Whether to create a new `SparkContext` for every `SparkGraphComputer` or to reuse an existing one.
|========================================================
-IMPORTANT: If the provider/user wishes to not use Hadoop `InputFormats`, it is possible to leverage Spark's RDD
+If the provider/user wishes to not use Hadoop `InputFormats`, it is possible to leverage Spark's RDD
constructs directly. There is a `gremlin.spark.graphInputRDD` configuration that references a `Class<? extends
InputRDD>`. An `InputRDD` provides a read method that takes a `SparkContext` and returns a graphRDD. Likewise, use
`gremlin.spark.graphOutputRDD` and the respective `OutputRDD`.
+It is possible to persist the graph RDD between jobs within the `SparkContext` (e.g. SparkServer) by leveraging `PersistedOutputRDD`.
+Note that `gremlin.spark.persistContext` should be set to `true` or else the persisted RDD will be destroyed when the `SparkContext` closes.
+The persisted RDD is named by the `gremlin.hadoop.outputLocation` configuration (i.e. named in `SparkContext.getPersistedRDDs()`).
+Finally, `PersistedInputRDD` is used with respective `gremlin.hadoop.inputLocation` to retrieve the persisted RDD from the `SparkContext`.
+
Loading with BulkLoaderVertexProgram
++++++++++++++++++++++++++++++++++++
@@ -1211,14 +1216,14 @@ Grateful Dead graph from HadoopGraph into TinkerGraph over Spark:
[gremlin-groovy]
----
hdfs.copyFromLocal('data/grateful-dead.kryo', 'data/grateful-dead.kryo')
-wgConf = 'conf/tinkergraph-gryo.properties'
-grateful = GraphFactory.open('conf/hadoop/hadoop-grateful-gryo.properties')
+readGraph = GraphFactory.open('conf/hadoop/hadoop-grateful-gryo.properties')
+writeGraph = 'conf/tinkergraph-gryo.properties'
blvp = BulkLoaderVertexProgram.build().
keepOriginalIds(false).
- writeGraph(wgConf).create(grateful)
-grateful.compute(SparkGraphComputer).workers(1).program(blvp).submit().get()
+ writeGraph(writeGraph).create(readGraph)
+readGraph.compute(SparkGraphComputer).workers(1).program(blvp).submit().get()
:set max-iteration 10
-graph = GraphFactory.open(wgConf)
+graph = GraphFactory.open(writeGraph)
g = graph.traversal()
g.V().valueMap()
graph.close()
@@ -1233,7 +1238,6 @@ graph.close()
#
gremlin.graph=org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph
gremlin.hadoop.graphInputFormat=org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat
-gremlin.hadoop.graphOutputFormat=org.apache.hadoop.mapreduce.lib.output.NullOutputFormat
gremlin.hadoop.memoryOutputFormat=org.apache.hadoop.mapreduce.lib.output.SequenceFileOutputFormat
gremlin.hadoop.inputLocation=data/grateful-dead.kryo
gremlin.hadoop.outputLocation=output
@@ -1245,7 +1249,7 @@ gremlin.hadoop.jarsInDistributedCache=true
#
spark.master=local[1]
spark.executor.memory=1g
-spark.serializer=org.apache.spark.serializer.KryoSerializer
+spark.serializer=org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer
----
[source,properties]
@@ -1257,7 +1261,7 @@ gremlin.tinkergraph.graphFormat=gryo
gremlin.tinkergraph.graphLocation=/tmp/tinkergraph.kryo
----
-NOTE: The path to TinkerGraph needs to be included in the `HADOOP_GREMLIN_LIBS` for the above example to work.
+IMPORTANT: The path to TinkerGraph jars needs to be included in the `HADOOP_GREMLIN_LIBS` for the above example to work.
[[giraphgraphcomputer]]
GiraphGraphComputer
@@ -1341,14 +1345,14 @@ the Grateful Dead graph from HadoopGraph into TinkerGraph over Giraph:
[gremlin-groovy]
----
hdfs.copyFromLocal('data/grateful-dead.kryo', 'data/grateful-dead.kryo')
-wgConf = 'conf/tinkergraph-gryo.properties'
-grateful = GraphFactory.open('conf/hadoop/hadoop-grateful-gryo.properties')
+readGraph = GraphFactory.open('conf/hadoop/hadoop-grateful-gryo.properties')
+writeGraph = 'conf/tinkergraph-gryo.properties'
blvp = BulkLoaderVertexProgram.build().
keepOriginalIds(false).
- writeGraph(wgConf).create(grateful)
-grateful.compute(GiraphGraphComputer).workers(1).program(blvp).submit().get()
+ writeGraph(writeGraph).create(readGraph)
+readGraph.compute(GiraphGraphComputer).workers(1).program(blvp).submit().get()
:set max-iteration 10
-graph = GraphFactory.open(wgConf)
+graph = GraphFactory.open(writeGraph)
g = graph.traversal()
g.V().valueMap()
graph.close()
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/528ba027/docs/src/the-traversal.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/the-traversal.asciidoc b/docs/src/the-traversal.asciidoc
index 42365fa..74e859e 100644
--- a/docs/src/the-traversal.asciidoc
+++ b/docs/src/the-traversal.asciidoc
@@ -2174,12 +2174,9 @@ public final class TinkerGraphStepStrategy extends AbstractTraversalStrategy<Tra
if (traversal.getEngine().isComputer())
return;
- final Step<?, ?> startStep = traversal.getStartStep();
- if (startStep instanceof GraphStep) {
- final GraphStep<?> originalGraphStep = (GraphStep) startStep;
- final TinkerGraphStep<?> tinkerGraphStep = new TinkerGraphStep<>(originalGraphStep);
- TraversalHelper.replaceStep(startStep, (Step) tinkerGraphStep, traversal);
-
+ TraversalHelper.getStepsOfClass(GraphStep.class, traversal).forEach(originalGraphStep -> {
+ final TinkerGraphStep<?,?> tinkerGraphStep = new TinkerGraphStep<>(originalGraphStep);
+ TraversalHelper.replaceStep(originalGraphStep, (Step) tinkerGraphStep, traversal);
Step<?, ?> currentStep = tinkerGraphStep.getNextStep();
while (currentStep instanceof HasContainerHolder) {
((HasContainerHolder) currentStep).getHasContainers().forEach(tinkerGraphStep::addHasContainer);
@@ -2187,7 +2184,7 @@ public final class TinkerGraphStepStrategy extends AbstractTraversalStrategy<Tra
traversal.removeStep(currentStep);
currentStep = currentStep.getNextStep();
}
- }
+ });
}
public static TinkerGraphStepStrategy instance() {
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/528ba027/docs/src/upgrade-release-3.1.x-incubating.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/upgrade-release-3.1.x-incubating.asciidoc b/docs/src/upgrade-release-3.1.x-incubating.asciidoc
index b50dd7e..77a448f 100644
--- a/docs/src/upgrade-release-3.1.x-incubating.asciidoc
+++ b/docs/src/upgrade-release-3.1.x-incubating.asciidoc
@@ -164,3 +164,11 @@ The `VendorOptimizationStrategy` has been renamed to `ProviderOptimizationStrate
with revised terminology for what were formerly referred to as "vendors".
See link:https://issues.apache.org/jira/browse/TINKERPOP3-876[TINKERPOP3-876] for more information.
+
+GraphComputer Updates
++++++++++++++++++++++
+
+`GraphComputer.configure(String key, Object value)` is now a method. This allows the user to specify engine-specific
+parameters to the underlying OLAP system. These parameters are not intended to be cross engine supported. Moreover, if
+there are not parameters that can be altered (beyond the standard `GraphComputer` methods), then the provider's `GraphComputer`
+implementation should simply return and do nothing.
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/528ba027/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
----------------------------------------------------------------------
diff --git a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
index 1e5dbd8..4c33220 100644
--- a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
+++ b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
@@ -93,7 +93,7 @@ public final class GiraphGraphComputer extends AbstractHadoopGraphComputer imple
}
@Override
- public GraphComputer config(final String key, final Object value) {
+ public GraphComputer configure(final String key, final Object value) {
this.giraphConfiguration.set(key, value.toString());
this.useWorkerThreadsInConfiguration = this.giraphConfiguration.getInt(GiraphConstants.MAX_WORKERS, -666) != -666 || this.giraphConfiguration.getInt(GiraphConstants.NUM_COMPUTE_THREADS.getKey(), -666) != -666;
return this;
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/528ba027/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputer.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputer.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputer.java
index bfb7bde..547af9e 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputer.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputer.java
@@ -108,12 +108,15 @@ public interface GraphComputer {
* Typically, the other fluent methods in {@link GraphComputer} should be used to configure the computation.
* However, for some custom configuration in the underlying engine, this method should be used.
* Different GraphComputer implementations will have different key/values and thus, parameters placed here are generally not universal to all GraphComputer implementations.
+ * The default implementation simply does nothing and returns the {@link GraphComputer} unchanged.
*
* @param key the key of the configuration
* @param value the value of the configuration
* @return the updated GraphComputer with newly set key/value configuration
*/
- public GraphComputer config(final String key, final Object value);
+ public default GraphComputer configure(final String key, final Object value) {
+ return this;
+ }
/**
* Submit the {@link VertexProgram} and the set of {@link MapReduce} jobs for execution by the {@link GraphComputer}.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/528ba027/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputerTest.java
----------------------------------------------------------------------
diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputerTest.java b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputerTest.java
index 0026296..acbdc81 100644
--- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputerTest.java
+++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputerTest.java
@@ -127,7 +127,7 @@ public class GraphComputerTest extends AbstractGremlinProcessTest {
}
@Override
- public GraphComputer config(final String key, final Object value) {
+ public GraphComputer configure(final String key, final Object value) {
return null;
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/528ba027/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/Constants.java
----------------------------------------------------------------------
diff --git a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/Constants.java b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/Constants.java
index 7264001..469e9b0 100644
--- a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/Constants.java
+++ b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/Constants.java
@@ -52,7 +52,4 @@ public final class Constants {
public static final String GREMLIN_SPARK_GRAPH_INPUT_RDD = "gremlin.spark.graphInputRDD";
public static final String GREMLIN_SPARK_GRAPH_OUTPUT_RDD = "gremlin.spark.graphOutputRDD";
public static final String GREMLIN_SPARK_PERSIST_CONTEXT = "gremlin.spark.persistContext";
- public static final String GREMLIN_SPARK_GRAPH_INPUT_RDD_NAME = "gremlin.spark.graphInputRDD.name";
- public static final String GREMLIN_SPARK_GRAPH_OUTPUT_RDD_NAME = "gremlin.spark.graphOutputRDD.name";
-
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/528ba027/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
index ef2ae6f..c9e95e3 100644
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
@@ -29,8 +29,8 @@ import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
import org.apache.spark.api.java.JavaPairRDD;
-import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.launcher.SparkLauncher;
import org.apache.tinkerpop.gremlin.hadoop.Constants;
import org.apache.tinkerpop.gremlin.hadoop.process.computer.AbstractHadoopGraphComputer;
import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration;
@@ -50,7 +50,6 @@ import org.apache.tinkerpop.gremlin.spark.process.computer.io.InputRDD;
import org.apache.tinkerpop.gremlin.spark.process.computer.io.OutputFormatRDD;
import org.apache.tinkerpop.gremlin.spark.process.computer.io.OutputRDD;
import org.apache.tinkerpop.gremlin.spark.process.computer.payload.ViewIncomingPayload;
-import org.apache.tinkerpop.gremlin.spark.process.computer.util.SparkHelper;
import java.io.File;
import java.io.IOException;
@@ -74,36 +73,45 @@ public final class SparkGraphComputer extends AbstractHadoopGraphComputer {
@Override
public GraphComputer workers(final int workers) {
super.workers(workers);
- if (this.sparkConfiguration.getString("spark.master").startsWith("local")) {
- this.sparkConfiguration.setProperty("spark.master", "local[" + this.workers + "]");
+ if (this.sparkConfiguration.getString(SparkLauncher.SPARK_MASTER).startsWith("local")) {
+ this.sparkConfiguration.setProperty(SparkLauncher.SPARK_MASTER, "local[" + this.workers + "]");
}
return this;
}
@Override
- public GraphComputer config(final String key, final Object value) {
+ public GraphComputer configure(final String key, final Object value) {
this.sparkConfiguration.setProperty(key, value);
return this;
}
@Override
- public Future<ComputerResult> submit() {
+ protected void validateStatePriorToExecution() {
super.validateStatePriorToExecution();
+ if (this.sparkConfiguration.containsKey(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD) && this.sparkConfiguration.containsKey(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT))
+ this.logger.warn("Both " + Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD + " and " + Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT + " were specified, ignoring " + Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT);
+ if (this.sparkConfiguration.containsKey(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD) && this.sparkConfiguration.containsKey(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT))
+ this.logger.warn("Both " + Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD + " and " + Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT + " were specified, ignoring " + Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT);
+ }
+
+ @Override
+ public Future<ComputerResult> submit() {
+ this.validateStatePriorToExecution();
// apache and hadoop configurations that are used throughout the graph computer computation
final org.apache.commons.configuration.Configuration apacheConfiguration = new HadoopConfiguration(this.sparkConfiguration);
apacheConfiguration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT_HAS_EDGES, this.persist.equals(GraphComputer.Persist.EDGES));
final Configuration hadoopConfiguration = ConfUtil.makeHadoopConfiguration(apacheConfiguration);
- if (null == hadoopConfiguration.get(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD_NAME, null)) {
- if (FileInputFormat.class.isAssignableFrom(hadoopConfiguration.getClass(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, InputFormat.class))) {
- try {
- final String inputLocation = FileSystem.get(hadoopConfiguration).getFileStatus(new Path(hadoopConfiguration.get(Constants.GREMLIN_HADOOP_INPUT_LOCATION))).getPath().toString();
- apacheConfiguration.setProperty(Constants.MAPREDUCE_INPUT_FILEINPUTFORMAT_INPUTDIR, inputLocation);
- hadoopConfiguration.set(Constants.MAPREDUCE_INPUT_FILEINPUTFORMAT_INPUTDIR, inputLocation);
- } catch (final IOException e) {
- throw new IllegalStateException(e.getMessage(), e);
- }
+ if (hadoopConfiguration.get(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, null) == null && // if an InputRDD is specified, then ignore InputFormat
+ hadoopConfiguration.get(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, null) != null &&
+ FileInputFormat.class.isAssignableFrom(hadoopConfiguration.getClass(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, InputFormat.class))) {
+ try {
+ final String inputLocation = FileSystem.get(hadoopConfiguration).getFileStatus(new Path(hadoopConfiguration.get(Constants.GREMLIN_HADOOP_INPUT_LOCATION))).getPath().toString();
+ apacheConfiguration.setProperty(Constants.MAPREDUCE_INPUT_FILEINPUTFORMAT_INPUTDIR, inputLocation);
+ hadoopConfiguration.set(Constants.MAPREDUCE_INPUT_FILEINPUTFORMAT_INPUTDIR, inputLocation);
+ } catch (final IOException e) {
+ throw new IllegalStateException(e.getMessage(), e);
}
- } // else WARN that both an INPUT_FORMAT and INPUT_RDD_NAME were provided?
+ }
// create the completable future
return CompletableFuture.<ComputerResult>supplyAsync(() -> {
@@ -111,12 +119,11 @@ public final class SparkGraphComputer extends AbstractHadoopGraphComputer {
SparkMemory memory = null;
// delete output location
final String outputLocation = hadoopConfiguration.get(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, null);
- if (null != outputLocation) {
- try {
+ try {
+ if (null != outputLocation && FileSystem.get(hadoopConfiguration).exists(new Path(outputLocation)))
FileSystem.get(hadoopConfiguration).delete(new Path(outputLocation), true);
- } catch (final IOException e) {
- throw new IllegalStateException(e.getMessage(), e);
- }
+ } catch (final IOException e) {
+ throw new IllegalStateException(e.getMessage(), e);
}
// wire up a spark context
final SparkConf sparkConfiguration = new SparkConf();
@@ -132,21 +139,14 @@ public final class SparkGraphComputer extends AbstractHadoopGraphComputer {
this.loadJars(sparkContext, hadoopConfiguration);
// create a message-passing friendly rdd from the input rdd
final JavaPairRDD<Object, VertexWritable> graphRDD;
- if (null != sparkConfiguration.get(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD_NAME, null)) {
- if (!SparkHelper.getPersistedRDD(sparkContext, sparkConfiguration.get(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD_NAME)).isPresent())
- throw new IllegalArgumentException("The provided graphRDD name is not in the persisted RDDs of the SparkContext: " + sparkConfiguration.get(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD_NAME, null));
- final JavaRDD rdd = SparkHelper.getPersistedRDD(sparkContext, sparkConfiguration.get(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD_NAME)).get().toJavaRDD();
- graphRDD = JavaPairRDD.fromJavaRDD(rdd).cache();
- } else {
- try {
- graphRDD = hadoopConfiguration.getClass(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, InputFormatRDD.class, InputRDD.class)
- .newInstance()
- .readGraphRDD(apacheConfiguration, sparkContext)
- .setName(sparkConfiguration.get(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD_NAME, "graphRDD"))
- .cache();
- } catch (final InstantiationException | IllegalAccessException e) {
- throw new IllegalStateException(e.getMessage(), e);
- }
+ try {
+ graphRDD = hadoopConfiguration.getClass(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, InputFormatRDD.class, InputRDD.class)
+ .newInstance()
+ .readGraphRDD(apacheConfiguration, sparkContext)
+ .setName(sparkConfiguration.get(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, "graphRDD"))
+ .cache();
+ } catch (final InstantiationException | IllegalAccessException e) {
+ throw new IllegalStateException(e.getMessage(), e);
}
JavaPairRDD<Object, ViewIncomingPayload<Object>> viewIncomingRDD = null;
@@ -176,7 +176,9 @@ public final class SparkGraphComputer extends AbstractHadoopGraphComputer {
}
}
// write the graph rdd using the output rdd
- if (!this.persist.equals(GraphComputer.Persist.NOTHING)) {
+ if ((hadoopConfiguration.get(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT, null) != null ||
+ hadoopConfiguration.get(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, null) != null) &&
+ !this.persist.equals(GraphComputer.Persist.NOTHING)) {
try {
hadoopConfiguration.getClass(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, OutputFormatRDD.class, OutputRDD.class)
.newInstance()
@@ -201,7 +203,7 @@ public final class SparkGraphComputer extends AbstractHadoopGraphComputer {
mapReduce.storeState(newApacheConfiguration);
// map
final JavaPairRDD mapRDD = SparkExecutor.executeMap((JavaPairRDD) mapReduceGraphRDD, mapReduce, newApacheConfiguration).setName("mapRDD");
- // combine TODO: is this really needed
+ // combine TODO: is this really needed?
// reduce
final JavaPairRDD reduceRDD = (mapReduce.doStage(MapReduce.Stage.REDUCE)) ? SparkExecutor.executeReduce(mapRDD, mapReduce, newApacheConfiguration).setName("reduceRDD") : null;
// write the map reduce output back to disk (memory)
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/528ba027/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputRDD.java
new file mode 100644
index 0000000..ad521b3
--- /dev/null
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputRDD.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tinkerpop.gremlin.spark.process.computer.io;
+
+import org.apache.commons.configuration.Configuration;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.rdd.RDD;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+import scala.Tuple2;
+import scala.collection.Iterator;
+
+import java.util.Optional;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public final class PersistedInputRDD implements InputRDD {
+
+ @Override
+ public JavaPairRDD<Object, VertexWritable> readGraphRDD(final Configuration configuration, final JavaSparkContext sparkContext) {
+ final String inputRDDName = configuration.getString(Constants.GREMLIN_HADOOP_INPUT_LOCATION, null);
+ if (null == inputRDDName)
+ throw new IllegalArgumentException(PersistedInputRDD.class.getSimpleName() + " requires " + Constants.GREMLIN_HADOOP_INPUT_LOCATION + " in order to retrieve the named graphRDD from the SparkContext");
+ if (!PersistedInputRDD.getPersistedRDD(sparkContext, inputRDDName).isPresent())
+ throw new IllegalArgumentException("The provided graphRDD name is not in the persisted RDDs of the SparkContext: " + inputRDDName);
+ return JavaPairRDD.fromJavaRDD((JavaRDD) PersistedInputRDD.getPersistedRDD(sparkContext, inputRDDName).get().toJavaRDD());
+ }
+
+ public static Optional<RDD<?>> getPersistedRDD(final JavaSparkContext sparkContext, final String rddName) {
+ final Iterator<Tuple2<Object, RDD<?>>> iterator = JavaSparkContext.toSparkContext(sparkContext).
+ getPersistentRDDs().
+ toList().iterator();
+ while (iterator.hasNext()) {
+ final Tuple2<Object, RDD<?>> tuple2 = iterator.next();
+ if (tuple2._2().toString().contains(rddName))
+ return Optional.of(tuple2._2());
+ }
+ return Optional.empty();
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/528ba027/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedOutputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedOutputRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedOutputRDD.java
new file mode 100644
index 0000000..1832ca3
--- /dev/null
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedOutputRDD.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tinkerpop.gremlin.spark.process.computer.io;
+
+import org.apache.commons.configuration.Configuration;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public final class PersistedOutputRDD implements OutputRDD {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(PersistedOutputRDD.class);
+
+ @Override
+ public void writeGraphRDD(final Configuration configuration, final JavaPairRDD<Object, VertexWritable> graphRDD) {
+ if (!configuration.getBoolean(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, false))
+ LOGGER.warn("The SparkContext should be persisted in order for the RDD to persist across jobs. To do so, set " + Constants.GREMLIN_SPARK_PERSIST_CONTEXT + " to true");
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/528ba027/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/util/SparkHelper.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/util/SparkHelper.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/util/SparkHelper.java
deleted file mode 100644
index da57fb0..0000000
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/util/SparkHelper.java
+++ /dev/null
@@ -1,49 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.tinkerpop.gremlin.spark.process.computer.util;
-
-import org.apache.spark.api.java.JavaSparkContext;
-import org.apache.spark.rdd.RDD;
-import scala.Tuple2;
-import scala.collection.Iterator;
-
-import java.util.Optional;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public final class SparkHelper {
-
- private SparkHelper() {
-
- }
-
- public static Optional<RDD<?>> getPersistedRDD(final JavaSparkContext sparkContext, final String rddName) {
- final Iterator<Tuple2<Object, RDD<?>>> iterator = JavaSparkContext.toSparkContext(sparkContext).
- getPersistentRDDs().
- toList().iterator();
- while (iterator.hasNext()) {
- final Tuple2<Object, RDD<?>> tuple2 = iterator.next();
- if (tuple2._2().toString().contains(rddName))
- return Optional.of(tuple2._2());
- }
- return Optional.empty();
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/528ba027/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputOutputRDDTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputOutputRDDTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputOutputRDDTest.java
new file mode 100644
index 0000000..332d80d
--- /dev/null
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputOutputRDDTest.java
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tinkerpop.gremlin.spark.process.computer.io;
+
+import org.apache.commons.configuration.BaseConfiguration;
+import org.apache.commons.configuration.Configuration;
+import org.apache.spark.SparkConf;
+import org.apache.spark.SparkContext;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat;
+import org.apache.tinkerpop.gremlin.hadoop.structure.util.ConfUtil;
+import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
+import org.apache.tinkerpop.gremlin.process.computer.bulkloading.BulkLoaderVertexProgram;
+import org.apache.tinkerpop.gremlin.process.computer.ranking.pagerank.PageRankVertexProgram;
+import org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram;
+import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
+import org.apache.tinkerpop.gremlin.process.traversal.engine.ComputerTraversalEngine;
+import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;
+import org.apache.tinkerpop.gremlin.spark.process.computer.SparkHadoopGraphProvider;
+import org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer;
+import org.apache.tinkerpop.gremlin.structure.Graph;
+import org.apache.tinkerpop.gremlin.structure.io.IoCore;
+import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
+import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
+import org.junit.Test;
+
+import java.util.UUID;
+
+import static org.junit.Assert.*;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public class PersistedInputOutputRDDTest {
+
+ @Test
+ public void shouldNotPersistRDDAcrossJobs() throws Exception {
+ final String rddName = "target/test-output/" + UUID.randomUUID();
+ final Configuration configuration = new BaseConfiguration();
+ configuration.setProperty("spark.master", "local[4]");
+ configuration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
+ configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, PersistedOutputRDD.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
+ configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, rddName);
+ configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, false); // because the spark context is NOT persisted, neither is the RDD
+ Graph graph = GraphFactory.open(configuration);
+ graph.compute(SparkGraphComputer.class)
+ .result(GraphComputer.ResultGraph.NEW)
+ .persist(GraphComputer.Persist.EDGES)
+ .program(TraversalVertexProgram.build()
+ .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
+ "gremlin-groovy",
+ "g.V()").create(graph)).submit().get();
+ ////////
+ SparkConf sparkConfiguration = new SparkConf();
+ sparkConfiguration.setAppName("shouldNotPersistRDDAcrossJobs");
+ ConfUtil.makeHadoopConfiguration(configuration).forEach(entry -> sparkConfiguration.set(entry.getKey(), entry.getValue()));
+ JavaSparkContext sparkContext = new JavaSparkContext(SparkContext.getOrCreate(sparkConfiguration));
+ assertFalse(PersistedInputRDD.getPersistedRDD(sparkContext, rddName).isPresent());
+ }
+
+ @Test
+ public void shouldPersistRDDAcrossJobs() throws Exception {
+ final String rddName = "target/test-output/" + UUID.randomUUID();
+ final Configuration configuration = new BaseConfiguration();
+ configuration.setProperty("spark.master", "local[4]");
+ configuration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
+ configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, PersistedOutputRDD.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, rddName);
+ configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
+ configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, true);
+ Graph graph = GraphFactory.open(configuration);
+ graph.compute(SparkGraphComputer.class)
+ .result(GraphComputer.ResultGraph.NEW)
+ .persist(GraphComputer.Persist.EDGES)
+ .program(TraversalVertexProgram.build()
+ .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
+ "gremlin-groovy",
+ "g.V()").create(graph)).submit().get();
+ ////////
+ SparkConf sparkConfiguration = new SparkConf();
+ sparkConfiguration.setAppName("shouldPersistRDDAcrossJobs");
+ ConfUtil.makeHadoopConfiguration(configuration).forEach(entry -> sparkConfiguration.set(entry.getKey(), entry.getValue()));
+ JavaSparkContext sparkContext = new JavaSparkContext(SparkContext.getOrCreate(sparkConfiguration));
+ assertTrue(PersistedInputRDD.getPersistedRDD(sparkContext, rddName).isPresent());
+ ///////
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, PersistedInputRDD.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, rddName);
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, null);
+ configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, null);
+ graph = GraphFactory.open(configuration);
+ graph.compute(SparkGraphComputer.class)
+ .result(GraphComputer.ResultGraph.NEW)
+ .persist(GraphComputer.Persist.NOTHING)
+ .program(TraversalVertexProgram.build()
+ .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
+ "gremlin-groovy",
+ "g.V()").create(graph)).submit().get();
+ }
+
+ @Test
+ public void testBulkLoaderVertexProgramChain() throws Exception {
+ final String rddName = "target/test-output/" + UUID.randomUUID().toString();
+ final Configuration readConfiguration = new BaseConfiguration();
+ readConfiguration.setProperty("spark.master", "local[4]");
+ readConfiguration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
+ readConfiguration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
+ readConfiguration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, PersistedOutputRDD.class.getCanonicalName());
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, rddName);
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
+ readConfiguration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, true);
+ Graph pageRankGraph = GraphFactory.open(readConfiguration);
+ ///////////////
+ final Configuration writeConfiguration = new BaseConfiguration();
+ writeConfiguration.setProperty(Graph.GRAPH, TinkerGraph.class.getCanonicalName());
+ writeConfiguration.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo");
+ writeConfiguration.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, "target/test-output/tinkergraph.kryo");
+ final Graph bulkLoaderGraph = pageRankGraph.compute(SparkGraphComputer.class).persist(GraphComputer.Persist.NOTHING).program(PageRankVertexProgram.build().create(pageRankGraph)).submit().get().graph();
+ bulkLoaderGraph.compute(SparkGraphComputer.class)
+ .persist(GraphComputer.Persist.NOTHING)
+ .workers(1)
+ .configure(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, PersistedInputRDD.class.getCanonicalName())
+ .configure(Constants.GREMLIN_HADOOP_INPUT_LOCATION, rddName)
+ .configure(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, null)
+ .configure(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, null)
+ .program(BulkLoaderVertexProgram.build().userSuppliedIds(true).writeGraph(writeConfiguration).create(bulkLoaderGraph))
+ .submit().get();
+ ////
+ final Graph graph = TinkerGraph.open();
+ final GraphTraversalSource g = graph.traversal();
+ graph.io(IoCore.gryo()).readGraph("target/test-output/tinkergraph.kryo");
+
+ assertEquals(6l, g.V().count().next().longValue());
+ assertEquals(6l, g.E().count().next().longValue());
+ assertEquals("marko", g.V().has("name", "marko").values("name").next());
+ assertEquals(6l, g.V().values(PageRankVertexProgram.PAGE_RANK).count().next().longValue());
+ assertEquals(0l, g.E().values(PageRankVertexProgram.PAGE_RANK).count().next().longValue());
+ assertEquals(6l, g.V().values(PageRankVertexProgram.EDGE_COUNT).count().next().longValue());
+ assertEquals(0l, g.E().values(PageRankVertexProgram.EDGE_COUNT).count().next().longValue());
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/528ba027/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/SparkContextPersistenceTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/SparkContextPersistenceTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/SparkContextPersistenceTest.java
deleted file mode 100644
index cc0957f..0000000
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/SparkContextPersistenceTest.java
+++ /dev/null
@@ -1,126 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.tinkerpop.gremlin.spark.process.computer.io;
-
-import org.apache.commons.configuration.BaseConfiguration;
-import org.apache.commons.configuration.Configuration;
-import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat;
-import org.apache.spark.SparkConf;
-import org.apache.spark.SparkContext;
-import org.apache.spark.api.java.JavaSparkContext;
-import org.apache.tinkerpop.gremlin.hadoop.Constants;
-import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat;
-import org.apache.tinkerpop.gremlin.hadoop.structure.util.ConfUtil;
-import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
-import org.apache.tinkerpop.gremlin.process.computer.bulkloading.BulkLoaderVertexProgram;
-import org.apache.tinkerpop.gremlin.process.computer.ranking.pagerank.PageRankVertexProgram;
-import org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram;
-import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
-import org.apache.tinkerpop.gremlin.process.traversal.engine.ComputerTraversalEngine;
-import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;
-import org.apache.tinkerpop.gremlin.spark.process.computer.SparkHadoopGraphProvider;
-import org.apache.tinkerpop.gremlin.spark.process.computer.util.SparkHelper;
-import org.apache.tinkerpop.gremlin.structure.Graph;
-import org.apache.tinkerpop.gremlin.structure.io.IoCore;
-import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
-import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
-import org.junit.Test;
-
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertTrue;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public class SparkContextPersistenceTest {
-
- @Test
- public void shouldPersistRDDAcrossJobs() throws Exception {
- final Configuration configuration = new BaseConfiguration();
- configuration.setProperty("spark.master", "local[4]");
- configuration.setProperty("spark.serializer", "org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer");
- configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
- configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
- configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
- configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, NullOutputFormat.class.getCanonicalName());
- configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
- configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD_NAME, "a-random-name-for-testing");
- configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, true);
- Graph graph = GraphFactory.open(configuration);
- graph.compute(SparkGraphComputer.class)
- .result(GraphComputer.ResultGraph.NEW)
- .persist(GraphComputer.Persist.NOTHING)
- .program(TraversalVertexProgram.build()
- .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
- "gremlin-groovy",
- "g.V()").create(graph)).submit().get();
- ////////
- SparkConf sparkConfiguration = new SparkConf();
- sparkConfiguration.setAppName("shouldPersistRDDAcrossJobs");
- ConfUtil.makeHadoopConfiguration(configuration).forEach(entry -> sparkConfiguration.set(entry.getKey(), entry.getValue()));
- JavaSparkContext sparkContext = new JavaSparkContext(SparkContext.getOrCreate(sparkConfiguration));
- assertTrue(SparkHelper.getPersistedRDD(sparkContext, "a-random-name-for-testing").isPresent());
- ///////
- configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD_NAME, "a-random-name-for-testing");
- configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD_NAME, null);
- graph = GraphFactory.open(configuration);
- graph.compute(SparkGraphComputer.class)
- .result(GraphComputer.ResultGraph.NEW)
- .persist(GraphComputer.Persist.NOTHING)
- .program(TraversalVertexProgram.build()
- .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
- "gremlin-groovy",
- "g.V()").create(graph)).submit().get();
- }
-
- @Test
- public void testBulkLoaderVertexProgramChain() throws Exception {
- final Configuration readConfiguration = new BaseConfiguration();
- readConfiguration.setProperty("spark.master", "local[4]");
- readConfiguration.setProperty("spark.serializer", "org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer");
- readConfiguration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
- readConfiguration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
- readConfiguration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
- readConfiguration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, NullOutputFormat.class.getCanonicalName());
- readConfiguration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, "target/test-output");
- readConfiguration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
- readConfiguration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD_NAME, "a-random-name-for-testing");
- readConfiguration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, true);
- Graph graph = GraphFactory.open(readConfiguration);
-
- ///////////////
- final Configuration writeConfiguration = new BaseConfiguration();
- writeConfiguration.setProperty(Graph.GRAPH, "org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph");
- writeConfiguration.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo");
- writeConfiguration.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, "target/test-output/tinkergraph.kryo");
- final Graph secondGraph = graph.compute(SparkGraphComputer.class).persist(GraphComputer.Persist.NOTHING).program(PageRankVertexProgram.build().create(graph)).submit().get().graph();
- secondGraph.compute(SparkGraphComputer.class)
- .persist(GraphComputer.Persist.NOTHING)
- .workers(1)
- .config(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD_NAME, "a-random-name-for-testing")
- .config(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD_NAME, null)
- .program(BulkLoaderVertexProgram.build().userSuppliedIds(true).writeGraph(writeConfiguration).create(secondGraph))
- .submit().get();
- final Graph finalGraph = TinkerGraph.open();
- finalGraph.io(IoCore.gryo()).readGraph("target/test-output/tinkergraph.kryo");
- assertEquals(6l, finalGraph.traversal().V().count().next().longValue());
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/528ba027/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/process/computer/TinkerGraphComputer.java
----------------------------------------------------------------------
diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/process/computer/TinkerGraphComputer.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/process/computer/TinkerGraphComputer.java
index 7092f99..07ad0c8 100644
--- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/process/computer/TinkerGraphComputer.java
+++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/process/computer/TinkerGraphComputer.java
@@ -92,11 +92,6 @@ public final class TinkerGraphComputer implements GraphComputer {
}
@Override
- public GraphComputer config(final String key, final Object value) {
- return this;
- }
-
- @Override
public Future<ComputerResult> submit() {
// a graph computer can only be executed once
if (this.executed)
[29/50] [abbrv] incubator-tinkerpop git commit: cleaned up mean() and
added test cases to verify that we get the correct return types
Posted by sp...@apache.org.
cleaned up mean() and added test cases to verify that we get the correct return types
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/af4f5dc1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/af4f5dc1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/af4f5dc1
Branch: refs/heads/TINKERPOP3-923
Commit: af4f5dc1d945da632998f4c490e5b946edb4d64f
Parents: baf2ae0
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Tue Nov 3 16:11:27 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Tue Nov 3 16:11:27 2015 +0100
----------------------------------------------------------------------
.../process/traversal/dsl/graph/GraphTraversal.java | 12 ++++++------
.../traversal/step/map/MeanGlobalStepTest.java | 14 ++++++++++++++
.../process/traversal/step/map/MeanLocalStepTest.java | 14 ++++++++++++++
3 files changed, 34 insertions(+), 6 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/af4f5dc1/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversal.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversal.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversal.java
index f1028a2..97d0c53 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversal.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversal.java
@@ -617,7 +617,7 @@ public interface GraphTraversal<S, E> extends Traversal<S, E> {
}
public default <E2 extends Number> GraphTraversal<S, E2> sum(final Scope scope) {
- return this.asAdmin().addStep(scope.equals(Scope.global) ? new SumGlobalStep(this.asAdmin()) : new SumLocalStep(this.asAdmin()));
+ return this.asAdmin().addStep(scope.equals(Scope.global) ? new SumGlobalStep<>(this.asAdmin()) : new SumLocalStep(this.asAdmin()));
}
public default <E2 extends Number> GraphTraversal<S, E2> max() {
@@ -625,7 +625,7 @@ public interface GraphTraversal<S, E> extends Traversal<S, E> {
}
public default <E2 extends Number> GraphTraversal<S, E2> max(final Scope scope) {
- return this.asAdmin().addStep(scope.equals(Scope.global) ? new MaxGlobalStep<E2>(this.asAdmin()) : new MaxLocalStep(this.asAdmin()));
+ return this.asAdmin().addStep(scope.equals(Scope.global) ? new MaxGlobalStep<>(this.asAdmin()) : new MaxLocalStep(this.asAdmin()));
}
public default <E2 extends Number> GraphTraversal<S, E2> min() {
@@ -633,15 +633,15 @@ public interface GraphTraversal<S, E> extends Traversal<S, E> {
}
public default <E2 extends Number> GraphTraversal<S, E2> min(final Scope scope) {
- return this.asAdmin().addStep(scope.equals(Scope.global) ? new MinGlobalStep<E2>(this.asAdmin()) : new MinLocalStep(this.asAdmin()));
+ return this.asAdmin().addStep(scope.equals(Scope.global) ? new MinGlobalStep<>(this.asAdmin()) : new MinLocalStep(this.asAdmin()));
}
- public default GraphTraversal<S, Double> mean() {
+ public default <E2 extends Number> GraphTraversal<S, E2> mean() {
return this.mean(Scope.global);
}
- public default GraphTraversal<S, Double> mean(final Scope scope) {
- return this.asAdmin().addStep(scope.equals(Scope.global) ? new MeanGlobalStep<>(this.asAdmin()) : new MeanLocalStep<>(this.asAdmin()));
+ public default <E2 extends Number> GraphTraversal<S, E2> mean(final Scope scope) {
+ return this.asAdmin().addStep(scope.equals(Scope.global) ? new MeanGlobalStep<>(this.asAdmin()) : new MeanLocalStep(this.asAdmin()));
}
public default <K, V> GraphTraversal<S, Map<K, V>> group() {
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/af4f5dc1/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanGlobalStepTest.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanGlobalStepTest.java b/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanGlobalStepTest.java
index f0a5490..be7dd5c 100644
--- a/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanGlobalStepTest.java
+++ b/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanGlobalStepTest.java
@@ -21,10 +21,15 @@ package org.apache.tinkerpop.gremlin.process.traversal.step.map;
import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__;
import org.apache.tinkerpop.gremlin.process.traversal.step.StepTest;
+import org.junit.Test;
+import java.math.BigDecimal;
+import java.math.BigInteger;
import java.util.Collections;
import java.util.List;
+import static org.junit.Assert.assertEquals;
+
/**
* @author Daniel Kuppitz (http://gremlin.guru)
*/
@@ -34,4 +39,13 @@ public class MeanGlobalStepTest extends StepTest {
protected List<Traversal> getTraversals() {
return Collections.singletonList(__.mean());
}
+
+ @Test
+ public void testReturnTypes() {
+ assertEquals(3d, __.__(2, 4).mean().next());
+ assertEquals(3d, __.__((short) 2, (short) 4).mean().next()); // why? because the internal counter is a Long value
+ assertEquals(BigDecimal.ONE, __.__(BigInteger.ONE, BigInteger.ONE).mean().next());
+ assertEquals(BigDecimal.ONE, __.__((short) 1, BigInteger.ONE).mean().next());
+ assertEquals(BigDecimal.ONE, __.__(BigInteger.ONE, (short) 1).mean().next());
+ }
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/af4f5dc1/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanLocalStepTest.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanLocalStepTest.java b/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanLocalStepTest.java
index 5f317f6..cf267d3 100644
--- a/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanLocalStepTest.java
+++ b/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanLocalStepTest.java
@@ -22,10 +22,15 @@ import org.apache.tinkerpop.gremlin.process.traversal.Scope;
import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__;
import org.apache.tinkerpop.gremlin.process.traversal.step.StepTest;
+import org.junit.Test;
+import java.math.BigDecimal;
+import java.math.BigInteger;
import java.util.Collections;
import java.util.List;
+import static org.junit.Assert.assertEquals;
+
/**
* @author Daniel Kuppitz (http://gremlin.guru)
*/
@@ -35,4 +40,13 @@ public class MeanLocalStepTest extends StepTest {
protected List<Traversal> getTraversals() {
return Collections.singletonList(__.mean(Scope.local));
}
+
+ @Test
+ public void testReturnTypes() {
+ assertEquals(3d, __.__(2, 4).fold().mean(Scope.local).next());
+ assertEquals(3d, __.__((short) 2, (short) 4).fold().mean(Scope.local).next()); // why? because the internal counter is a Long value
+ assertEquals(BigDecimal.ONE, __.__(BigInteger.ONE, BigInteger.ONE).fold().mean(Scope.local).next());
+ assertEquals(BigDecimal.ONE, __.__((short) 1, BigInteger.ONE).fold().mean(Scope.local).next());
+ assertEquals(BigDecimal.ONE, __.__(BigInteger.ONE, (short) 1).fold().mean(Scope.local).next());
+ }
}
[21/50] [abbrv] incubator-tinkerpop git commit: updated CHANGELOG and
upgrade docs
Posted by sp...@apache.org.
updated CHANGELOG and upgrade docs
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/ccc3bfc3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/ccc3bfc3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/ccc3bfc3
Branch: refs/heads/TINKERPOP3-923
Commit: ccc3bfc31cdf84f7a25dc1f6da89b9dc42f5490a
Parents: eba7114
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Sun Nov 1 22:13:17 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Sun Nov 1 22:13:17 2015 +0100
----------------------------------------------------------------------
CHANGELOG.asciidoc | 1 +
docs/src/upgrade-release-3.1.x-incubating.asciidoc | 1 +
2 files changed, 2 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/ccc3bfc3/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index d773c02..6f7c50e 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -25,6 +25,7 @@ image::https://raw.githubusercontent.com/apache/incubator-tinkerpop/master/docs/
TinkerPop 3.1.0 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Integrated `NumberHelper` in `SumStep`, `MinStep`, `MaxStep` and `MeanStep` (local and global step variants).
* Renamed the `public static String` configuration variable names of TinkerGraph (deprecated old variables).
* Added `GraphComputer.config(key,value)` to allow engine-specific configurations.
* Added suppport for mid-traversal `V()`-steps (`GraphStep` semantics updated).
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/ccc3bfc3/docs/src/upgrade-release-3.1.x-incubating.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/upgrade-release-3.1.x-incubating.asciidoc b/docs/src/upgrade-release-3.1.x-incubating.asciidoc
index 1e52cb5..7043d74 100644
--- a/docs/src/upgrade-release-3.1.x-incubating.asciidoc
+++ b/docs/src/upgrade-release-3.1.x-incubating.asciidoc
@@ -35,6 +35,7 @@ Additional upgrade information can be found here:
Important Changes
~~~~~~~~~~~~~~~~~
+* `sum()`, `min()`, `max()` and `mean()` make now use of `NumberHelper`, leading to a more appropriate result type, instead of just `Double`.
* Traversals now support mid-traversal `V()`-steps. Graph system providers should ensure that a mid-traversal `V()` can leverage any suitable index.
* The data type of `Operator` enums will now always be the highest common data type of the two given numbers, rather than the data type of the first number, as it's been before.
* The Gephi Plugin has improved integration with Gephi, where manually inserting `store` steps to visualize a running traversal is no longer required.
[30/50] [abbrv] incubator-tinkerpop git commit: MatchStep is smart to
bias patterns towards the local star graph to reduce inter-machine
communication.
Posted by sp...@apache.org.
MatchStep is smart to bias patterns towards the local star graph to reduce inter-machine communication.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/956d7977
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/956d7977
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/956d7977
Branch: refs/heads/TINKERPOP3-923
Commit: 956d7977f4507db8d51b0468604f3b214e0681b3
Parents: aadd422
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Tue Nov 3 09:16:39 2015 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Tue Nov 3 09:16:39 2015 -0700
----------------------------------------------------------------------
.../process/traversal/step/map/MatchStep.java | 48 ++++++++----
.../traversal/step/map/MatchStepTest.java | 82 +++++++++++++++++++-
2 files changed, 115 insertions(+), 15 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/956d7977/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStep.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStep.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStep.java
index 724ab8a..1ed202f 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStep.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStep.java
@@ -22,6 +22,7 @@ import org.apache.tinkerpop.gremlin.process.traversal.Path;
import org.apache.tinkerpop.gremlin.process.traversal.Pop;
import org.apache.tinkerpop.gremlin.process.traversal.Step;
import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
+import org.apache.tinkerpop.gremlin.process.traversal.TraversalEngine;
import org.apache.tinkerpop.gremlin.process.traversal.Traverser;
import org.apache.tinkerpop.gremlin.process.traversal.step.Scoping;
import org.apache.tinkerpop.gremlin.process.traversal.step.TraversalParent;
@@ -64,8 +65,9 @@ import java.util.stream.Stream;
*/
public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>> implements TraversalParent, Scoping {
- public static enum TraversalType {WHERE_PREDICATE, WHERE_TRAVERSAL, MATCH_TRAVERSAL}
+ public enum TraversalType {WHERE_PREDICATE, WHERE_TRAVERSAL, MATCH_TRAVERSAL}
+ private TraversalEngine.Type traversalEngineType;
private List<Traversal.Admin<Object, Object>> matchTraversals = new ArrayList<>();
private boolean first = true;
private Set<String> matchStartLabels = new HashSet<>();
@@ -153,6 +155,12 @@ public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>>
}
}
+ @Override
+ public void onEngine(final TraversalEngine engine) {
+ super.onEngine(engine);
+ this.traversalEngineType = engine.getType();
+ }
+
public ConnectiveStep.Connective getConnective() {
return this.connective;
}
@@ -216,7 +224,7 @@ public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>>
clone.matchTraversals.add(clone.integrateChild(traversal.clone()));
}
if (this.dedups != null) clone.dedups = new HashSet<>();
- clone.initializeMatchAlgorithm();
+ clone.initializeMatchAlgorithm(this.traversalEngineType);
return clone;
}
@@ -281,13 +289,13 @@ public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>>
return bindings;
}
- private void initializeMatchAlgorithm() {
+ private void initializeMatchAlgorithm(final TraversalEngine.Type traversalEngineType) {
try {
this.matchAlgorithm = this.matchAlgorithmClass.getConstructor().newInstance();
} catch (final NoSuchMethodException | IllegalAccessException | InvocationTargetException | InstantiationException e) {
throw new IllegalStateException(e.getMessage(), e);
}
- this.matchAlgorithm.initialize(this.matchTraversals);
+ this.matchAlgorithm.initialize(traversalEngineType, this.matchTraversals);
}
@Override
@@ -296,7 +304,7 @@ public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>>
Traverser.Admin traverser = null;
if (this.first) {
this.first = false;
- this.initializeMatchAlgorithm();
+ this.initializeMatchAlgorithm(this.traversalEngineType);
} else {
for (final Traversal.Admin<?, ?> matchTraversal : this.matchTraversals) {
if (matchTraversal.hasNext()) {
@@ -420,7 +428,7 @@ public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>>
if (null != this.selectKey)
this.scopeKeys.add(this.selectKey);
if (this.getNextStep() instanceof WhereTraversalStep || this.getNextStep() instanceof WherePredicateStep)
- this.scopeKeys.addAll(((Scoping) this.getNextStep()).getScopeKeys());
+ this.scopeKeys.addAll(((Scoping) this.getNextStep()).getScopeKeys());
this.scopeKeys = Collections.unmodifiableSet(this.scopeKeys);
}
return this.scopeKeys;
@@ -555,7 +563,7 @@ public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>>
public static Function<List<Traversal.Admin<Object, Object>>, IllegalStateException> UNMATCHABLE_PATTERN = traversals -> new IllegalStateException("The provided match pattern is unsolvable: " + traversals);
- public void initialize(final List<Traversal.Admin<Object, Object>> traversals);
+ public void initialize(final TraversalEngine.Type traversalEngineType, final List<Traversal.Admin<Object, Object>> traversals);
public default void recordStart(final Traverser.Admin<Object> traverser, final Traversal.Admin<Object, Object> traversal) {
@@ -571,7 +579,7 @@ public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>>
private List<Traversal.Admin<Object, Object>> traversals;
@Override
- public void initialize(final List<Traversal.Admin<Object, Object>> traversals) {
+ public void initialize(final TraversalEngine.Type traversalEngineType, final List<Traversal.Admin<Object, Object>> traversals) {
this.traversals = traversals;
}
@@ -589,14 +597,26 @@ public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>>
protected List<Bundle> bundles;
protected int counter = 0;
+ protected boolean onComputer;
- @Override
- public void initialize(final List<Traversal.Admin<Object, Object>> traversals) {
+ public void initialize(final TraversalEngine.Type traversalEngineType, final List<Traversal.Admin<Object, Object>> traversals) {
+ this.onComputer = traversalEngineType.equals(TraversalEngine.Type.COMPUTER);
this.bundles = traversals.stream().map(Bundle::new).collect(Collectors.toList());
}
@Override
public Traversal.Admin<Object, Object> apply(final Traverser.Admin<Object> traverser) {
+ // optimization to favor processing StarGraph local objects first to limit message passing (GraphComputer only)
+ // TODO: generalize this for future MatchAlgorithms (given that 3.2.0 will focus on RealTimeStrategy, it will probably go there)
+ if (this.onComputer) {
+ final List<Set<String>> labels = traverser.path().labels();
+ final Set<String> lastLabels = labels.get(labels.size() - 1);
+ Collections.sort(this.bundles,
+ Comparator.<Bundle>comparingLong(b -> Helper.getStartLabels(b.traversal).stream().filter(startLabel -> !lastLabels.contains(startLabel)).count()).
+ thenComparingInt(b -> b.traversalType.ordinal()).
+ thenComparingDouble(b -> b.multiplicity));
+ }
+
Bundle startLabelsBundle = null;
for (final Bundle bundle : this.bundles) {
if (!Helper.hasExecutedTraversal(traverser, bundle.traversal) && Helper.hasStartLabels(traverser, bundle.traversal)) {
@@ -618,9 +638,11 @@ public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>>
@Override
public void recordEnd(final Traverser.Admin<Object> traverser, final Traversal.Admin<Object, Object> traversal) {
this.getBundle(traversal).incrementEndCount();
- if (this.counter < 200 || this.counter % 250 == 0) // aggressively sort for the first 200 results -- after that, sort every 250
- Collections.sort(this.bundles, Comparator.<Bundle>comparingInt(b -> b.traversalType.ordinal()).thenComparingDouble(b -> b.multiplicity));
- this.counter++;
+ if (!this.onComputer) { // if on computer, sort on a per traverser-basis with bias towards local star graph
+ if (this.counter < 200 || this.counter % 250 == 0) // aggressively sort for the first 200 results -- after that, sort every 250
+ Collections.sort(this.bundles, Comparator.<Bundle>comparingInt(b -> b.traversalType.ordinal()).thenComparingDouble(b -> b.multiplicity));
+ this.counter++;
+ }
}
protected Bundle getBundle(final Traversal.Admin<Object, Object> traversal) {
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/956d7977/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStepTest.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStepTest.java b/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStepTest.java
index 9c7cef1..529e717 100644
--- a/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStepTest.java
+++ b/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStepTest.java
@@ -20,19 +20,25 @@ package org.apache.tinkerpop.gremlin.process.traversal.step.map;
import org.apache.tinkerpop.gremlin.process.traversal.P;
import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
+import org.apache.tinkerpop.gremlin.process.traversal.TraversalEngine;
+import org.apache.tinkerpop.gremlin.process.traversal.Traverser;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__;
import org.apache.tinkerpop.gremlin.process.traversal.step.StepTest;
import org.apache.tinkerpop.gremlin.process.traversal.step.filter.CoinStep;
import org.apache.tinkerpop.gremlin.process.traversal.step.filter.ConnectiveStep;
import org.apache.tinkerpop.gremlin.process.traversal.step.filter.WherePredicateStep;
import org.apache.tinkerpop.gremlin.process.traversal.step.filter.WhereTraversalStep;
+import org.apache.tinkerpop.gremlin.process.traversal.step.util.EmptyStep;
+import org.apache.tinkerpop.gremlin.process.traversal.traverser.B_LP_O_P_S_SE_SL_TraverserGenerator;
import org.apache.tinkerpop.gremlin.process.traversal.traverser.util.EmptyTraverser;
import org.apache.tinkerpop.gremlin.structure.T;
import org.junit.Test;
import java.util.Arrays;
+import java.util.Collections;
import java.util.HashSet;
import java.util.List;
+import java.util.function.Consumer;
import static org.apache.tinkerpop.gremlin.process.traversal.P.eq;
import static org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__.*;
@@ -186,7 +192,7 @@ public class MatchStepTest extends StepTest {
// MAKE SURE THE SORT ORDER CHANGES AS MORE RESULTS ARE RETURNED BY ONE OR THE OTHER TRAVERSAL
Traversal.Admin<?, ?> traversal = __.match(as("a").out().as("b"), as("c").in().as("d")).asAdmin();
MatchStep.CountMatchAlgorithm countMatchAlgorithm = new MatchStep.CountMatchAlgorithm();
- countMatchAlgorithm.initialize(((MatchStep<?, ?>) traversal.getStartStep()).getGlobalChildren());
+ countMatchAlgorithm.initialize(TraversalEngine.Type.STANDARD, ((MatchStep<?, ?>) traversal.getStartStep()).getGlobalChildren());
Traversal.Admin<Object, Object> firstPattern = ((MatchStep<?, ?>) traversal.getStartStep()).getGlobalChildren().get(0);
Traversal.Admin<Object, Object> secondPattern = ((MatchStep<?, ?>) traversal.getStartStep()).getGlobalChildren().get(1);
//
@@ -232,7 +238,7 @@ public class MatchStepTest extends StepTest {
/////// MAKE SURE WHERE PREDICATE TRAVERSALS ARE ALWAYS FIRST AS THEY ARE SIMPLY .hasNext() CHECKS
traversal = __.match(as("a").out().as("b"), as("c").in().as("d"), where("a", P.eq("b"))).asAdmin();
countMatchAlgorithm = new MatchStep.CountMatchAlgorithm();
- countMatchAlgorithm.initialize(((MatchStep<?, ?>) traversal.getStartStep()).getGlobalChildren());
+ countMatchAlgorithm.initialize(TraversalEngine.Type.STANDARD, ((MatchStep<?, ?>) traversal.getStartStep()).getGlobalChildren());
assertEquals(3, countMatchAlgorithm.bundles.size());
firstPattern = ((MatchStep<?, ?>) traversal.getStartStep()).getGlobalChildren().get(0);
secondPattern = ((MatchStep<?, ?>) traversal.getStartStep()).getGlobalChildren().get(1);
@@ -307,6 +313,78 @@ public class MatchStepTest extends StepTest {
}
@Test
+ public void testComputerAwareCountMatchAlgorithm() {
+ // MAKE SURE THE SORT ORDER CHANGES AS MORE RESULTS ARE RETURNED BY ONE OR THE OTHER TRAVERSAL
+ final Consumer doNothing = s -> {
+ };
+ Traversal.Admin<?, ?> traversal = __.match(
+ as("a").sideEffect(doNothing).as("b"), // 1
+ as("b").sideEffect(doNothing).as("c"), // 2
+ as("a").sideEffect(doNothing).as("d"), // 5
+ as("c").sideEffect(doNothing).as("e"), // 4
+ as("c").sideEffect(doNothing).as("f")) // 3
+ .asAdmin();
+ traversal.applyStrategies(); // necessary to enure step ids are unique
+ MatchStep.CountMatchAlgorithm countMatchAlgorithm = new MatchStep.CountMatchAlgorithm();
+ countMatchAlgorithm.initialize(TraversalEngine.Type.COMPUTER, ((MatchStep<?, ?>) traversal.getStartStep()).getGlobalChildren());
+ Traversal.Admin<Object, Object> firstPattern = ((MatchStep<?, ?>) traversal.getStartStep()).getGlobalChildren().get(0);
+ Traversal.Admin<Object, Object> secondPattern = ((MatchStep<?, ?>) traversal.getStartStep()).getGlobalChildren().get(1);
+ Traversal.Admin<Object, Object> thirdPattern = ((MatchStep<?, ?>) traversal.getStartStep()).getGlobalChildren().get(2);
+ Traversal.Admin<Object, Object> forthPattern = ((MatchStep<?, ?>) traversal.getStartStep()).getGlobalChildren().get(3);
+ Traversal.Admin<Object, Object> fifthPattern = ((MatchStep<?, ?>) traversal.getStartStep()).getGlobalChildren().get(4);
+ countMatchAlgorithm.bundles.stream().forEach(bundle -> assertEquals(0.0d, bundle.multiplicity, 0.0d));
+ assertEquals(MatchStep.TraversalType.MATCH_TRAVERSAL, countMatchAlgorithm.getBundle(firstPattern).traversalType);
+ assertEquals(MatchStep.TraversalType.MATCH_TRAVERSAL, countMatchAlgorithm.getBundle(secondPattern).traversalType);
+ assertEquals(MatchStep.TraversalType.MATCH_TRAVERSAL, countMatchAlgorithm.getBundle(thirdPattern).traversalType);
+ assertEquals(MatchStep.TraversalType.MATCH_TRAVERSAL, countMatchAlgorithm.getBundle(forthPattern).traversalType);
+ assertEquals(MatchStep.TraversalType.MATCH_TRAVERSAL, countMatchAlgorithm.getBundle(fifthPattern).traversalType);
+ assertEquals(firstPattern, countMatchAlgorithm.bundles.get(0).traversal);
+ assertEquals(secondPattern, countMatchAlgorithm.bundles.get(1).traversal);
+ assertEquals(thirdPattern, countMatchAlgorithm.bundles.get(2).traversal);
+ assertEquals(forthPattern, countMatchAlgorithm.bundles.get(3).traversal);
+ assertEquals(fifthPattern, countMatchAlgorithm.bundles.get(4).traversal);
+ // MAKE THE SECOND PATTERN EXPENSIVE
+ countMatchAlgorithm.recordStart(EmptyTraverser.instance(), secondPattern);
+ countMatchAlgorithm.recordEnd(EmptyTraverser.instance(), secondPattern);
+ countMatchAlgorithm.recordEnd(EmptyTraverser.instance(), secondPattern);
+ countMatchAlgorithm.recordEnd(EmptyTraverser.instance(), secondPattern);
+ countMatchAlgorithm.recordEnd(EmptyTraverser.instance(), secondPattern);
+ countMatchAlgorithm.recordEnd(EmptyTraverser.instance(), secondPattern);
+ countMatchAlgorithm.recordEnd(EmptyTraverser.instance(), secondPattern);
+ // MAKE THE THIRD PATTERN MORE EXPENSIVE THAN FORTH
+ countMatchAlgorithm.recordStart(EmptyTraverser.instance(), thirdPattern);
+ countMatchAlgorithm.recordEnd(EmptyTraverser.instance(), thirdPattern);
+ countMatchAlgorithm.recordEnd(EmptyTraverser.instance(), thirdPattern);
+ countMatchAlgorithm.recordEnd(EmptyTraverser.instance(), thirdPattern);
+ // MAKE THE FORTH PATTERN EXPENSIVE
+ countMatchAlgorithm.recordStart(EmptyTraverser.instance(), forthPattern);
+ countMatchAlgorithm.recordEnd(EmptyTraverser.instance(), forthPattern);
+ countMatchAlgorithm.recordEnd(EmptyTraverser.instance(), forthPattern);
+ //
+ Traverser.Admin traverser = B_LP_O_P_S_SE_SL_TraverserGenerator.instance().generate(1, EmptyStep.instance(), 1l);
+ traverser.addLabels(Collections.singleton("a"));
+ assertEquals(firstPattern, countMatchAlgorithm.apply(traverser));
+ traverser = traverser.split(1, EmptyStep.instance());
+ traverser.addLabels(new HashSet<>(Arrays.asList("b", firstPattern.getStartStep().getId())));
+ //
+ assertEquals(secondPattern, countMatchAlgorithm.apply(traverser));
+ traverser = traverser.split(1, EmptyStep.instance());
+ traverser.addLabels(new HashSet<>(Arrays.asList("c", secondPattern.getStartStep().getId())));
+ //
+ assertEquals(fifthPattern, countMatchAlgorithm.apply(traverser));
+ traverser = traverser.split(1, EmptyStep.instance());
+ traverser.addLabels(new HashSet<>(Arrays.asList("f", fifthPattern.getStartStep().getId())));
+ //
+ assertEquals(forthPattern, countMatchAlgorithm.apply(traverser));
+ traverser = traverser.split(1, EmptyStep.instance());
+ traverser.addLabels(new HashSet<>(Arrays.asList("e", forthPattern.getStartStep().getId())));
+ //
+ assertEquals(thirdPattern, countMatchAlgorithm.apply(traverser));
+ traverser = traverser.split(1, EmptyStep.instance());
+ traverser.addLabels(new HashSet<>(Arrays.asList("d", thirdPattern.getStartStep().getId())));
+ }
+
+ @Test
public void shouldCalculateStartLabelCorrectly() {
Traversal.Admin<?, ?> traversal = match(
where(and(
[15/50] [abbrv] incubator-tinkerpop git commit: integrated
NumberHelper in SumGlobalStep and SumLocalStep
Posted by sp...@apache.org.
integrated NumberHelper in SumGlobalStep and SumLocalStep
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/d49deafb
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/d49deafb
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/d49deafb
Branch: refs/heads/TINKERPOP3-923
Commit: d49deafb0d0c157bb59c26d6148993a8cccd64fb
Parents: 70a12e6
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Sat Oct 31 12:07:50 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Sat Oct 31 12:07:50 2015 +0100
----------------------------------------------------------------------
.../traversal/dsl/graph/GraphTraversal.java | 6 ++--
.../traversal/step/map/SumGlobalStep.java | 25 +++++++++--------
.../traversal/step/map/SumLocalStep.java | 5 ++--
.../step/sideEffect/GroovyGroupTest.groovy | 2 +-
.../process/computer/GraphComputerTest.java | 29 +++++++++++---------
.../process/traversal/step/map/SumTest.java | 2 +-
.../traversal/step/sideEffect/GroupTest.java | 22 +++++++++------
...ComputerVerificationStrategyProcessTest.java | 3 +-
.../groovy/plugin/HadoopGremlinPluginTest.java | 2 +-
.../spark/process/computer/io/InputRDDTest.java | 2 +-
.../structure/TinkerGraphPlayTest.java | 20 ++++----------
11 files changed, 62 insertions(+), 56 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d49deafb/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversal.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversal.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversal.java
index b3f3c2b..f1028a2 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversal.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversal.java
@@ -612,12 +612,12 @@ public interface GraphTraversal<S, E> extends Traversal<S, E> {
*
* @return the traversal with an appended {@link SumGlobalStep}.
*/
- public default GraphTraversal<S, Double> sum() {
+ public default <E2 extends Number> GraphTraversal<S, E2> sum() {
return this.sum(Scope.global);
}
- public default GraphTraversal<S, Double> sum(final Scope scope) {
- return this.asAdmin().addStep(scope.equals(Scope.global) ? new SumGlobalStep(this.asAdmin()) : new SumLocalStep<>(this.asAdmin()));
+ public default <E2 extends Number> GraphTraversal<S, E2> sum(final Scope scope) {
+ return this.asAdmin().addStep(scope.equals(Scope.global) ? new SumGlobalStep(this.asAdmin()) : new SumLocalStep(this.asAdmin()));
}
public default <E2 extends Number> GraphTraversal<S, E2> max() {
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d49deafb/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumGlobalStep.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumGlobalStep.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumGlobalStep.java
index 721eda7..8816390 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumGlobalStep.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumGlobalStep.java
@@ -37,10 +37,13 @@ import java.util.Iterator;
import java.util.Set;
import java.util.function.BiFunction;
+import static org.apache.tinkerpop.gremlin.process.traversal.NumberHelper.add;
+import static org.apache.tinkerpop.gremlin.process.traversal.NumberHelper.mul;
+
/**
* @author Marko A. Rodriguez (http://markorodriguez.com)
*/
-public final class SumGlobalStep extends ReducingBarrierStep<Number, Double> implements MapReducer {
+public final class SumGlobalStep extends ReducingBarrierStep<Number, Number> implements MapReducer {
private static final Set<TraverserRequirement> REQUIREMENTS = EnumSet.of(
TraverserRequirement.BULK,
@@ -49,7 +52,7 @@ public final class SumGlobalStep extends ReducingBarrierStep<Number, Double> imp
public SumGlobalStep(final Traversal.Admin traversal) {
super(traversal);
- this.setSeedSupplier(new ConstantSupplier<>(0.0d));
+ this.setSeedSupplier(new ConstantSupplier<>(0));
this.setBiFunction(SumGlobalBiFunction.instance());
}
@@ -66,20 +69,20 @@ public final class SumGlobalStep extends ReducingBarrierStep<Number, Double> imp
/////
- private static class SumGlobalBiFunction<S extends Number> implements BiFunction<Double, Traverser<S>, Double>, Serializable {
+ private static class SumGlobalBiFunction<S extends Number> implements BiFunction<Number, Traverser<S>, Number>, Serializable {
- private static final SumGlobalBiFunction INSTANCE = new SumGlobalBiFunction();
+ private static final SumGlobalBiFunction INSTANCE = new SumGlobalBiFunction<>();
private SumGlobalBiFunction() {
}
@Override
- public Double apply(final Double mutatingSeed, final Traverser<S> traverser) {
- return mutatingSeed + (traverser.get().doubleValue() * traverser.bulk());
+ public Number apply(final Number mutatingSeed, final Traverser<S> traverser) {
+ return add(mutatingSeed, mul(traverser.get(), traverser.bulk()));
}
- public final static <S extends Number> SumGlobalBiFunction<S> instance() {
+ public static <S extends Number> SumGlobalBiFunction<S> instance() {
return INSTANCE;
}
}
@@ -101,7 +104,7 @@ public final class SumGlobalStep extends ReducingBarrierStep<Number, Double> imp
@Override
public void map(final Vertex vertex, final MapEmitter<NullObject, Number> emitter) {
- vertex.<TraverserSet<Number>>property(TraversalVertexProgram.HALTED_TRAVERSERS).ifPresent(traverserSet -> traverserSet.forEach(traverser -> emitter.emit(traverser.get().doubleValue() * traverser.bulk())));
+ vertex.<TraverserSet<Number>>property(TraversalVertexProgram.HALTED_TRAVERSERS).ifPresent(traverserSet -> traverserSet.forEach(traverser -> emitter.emit(mul(traverser.get(), traverser.bulk()))));
}
@Override
@@ -114,7 +117,7 @@ public final class SumGlobalStep extends ReducingBarrierStep<Number, Double> imp
if (values.hasNext()) {
Number sum = values.next();
while (values.hasNext()) {
- sum = sum.doubleValue() + values.next().doubleValue();
+ sum = add(sum, values.next());
}
emitter.emit(sum);
}
@@ -127,10 +130,10 @@ public final class SumGlobalStep extends ReducingBarrierStep<Number, Double> imp
@Override
public Number generateFinalResult(final Iterator<KeyValue<NullObject, Number>> keyValues) {
- return keyValues.hasNext() ? keyValues.next().getValue() : 0.0d;
+ return keyValues.hasNext() ? keyValues.next().getValue() : 0;
}
- public static final SumGlobalMapReduce instance() {
+ public static SumGlobalMapReduce instance() {
return INSTANCE;
}
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d49deafb/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumLocalStep.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumLocalStep.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumLocalStep.java
index 4ca2e5a..c383b25 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumLocalStep.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumLocalStep.java
@@ -18,6 +18,7 @@
*/
package org.apache.tinkerpop.gremlin.process.traversal.step.map;
+import org.apache.tinkerpop.gremlin.process.traversal.NumberHelper;
import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
import org.apache.tinkerpop.gremlin.process.traversal.Traverser;
import org.apache.tinkerpop.gremlin.process.traversal.traverser.TraverserRequirement;
@@ -43,10 +44,10 @@ public final class SumLocalStep<E extends Number, S extends Iterable<E>> extends
if (iterator.hasNext()) {
result = iterator.next();
while (iterator.hasNext()) {
- result = result.doubleValue() + iterator.next().doubleValue();
+ result = NumberHelper.add(result, iterator.next());
}
} else {
- result = 0.0d;
+ result = 0;
}
return (E) result;
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d49deafb/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroovyGroupTest.groovy
----------------------------------------------------------------------
diff --git a/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroovyGroupTest.groovy b/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroovyGroupTest.groovy
index 7c41666..5a8a09e 100644
--- a/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroovyGroupTest.groovy
+++ b/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroovyGroupTest.groovy
@@ -67,7 +67,7 @@ public abstract class GroovyGroupTest {
}
@Override
- public Traversal<Vertex, Map<String, Double>> get_g_V_groupXaX_byXlabelX_byXoutE_weight_sumX_capXaX() {
+ public Traversal<Vertex, Map<String, Number>> get_g_V_groupXaX_byXlabelX_byXoutE_weight_sumX_capXaX() {
TraversalScriptHelper.compute("g.V.group('a').by(label).by(outE().weight.sum).cap('a')", g);
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d49deafb/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputerTest.java
----------------------------------------------------------------------
diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputerTest.java b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputerTest.java
index 0026296..0f5f3e9 100644
--- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputerTest.java
+++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputerTest.java
@@ -45,7 +45,10 @@ import java.util.concurrent.Future;
import static org.apache.tinkerpop.gremlin.LoadGraphWith.GraphData.GRATEFUL;
import static org.apache.tinkerpop.gremlin.LoadGraphWith.GraphData.MODERN;
-import static org.junit.Assert.*;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+import static org.junit.Assert.fail;
/**
* @author Marko A. Rodriguez (http://markorodriguez.com)
@@ -1223,13 +1226,13 @@ public class GraphComputerTest extends AbstractGremlinProcessTest {
assertEquals(Long.valueOf(0l), result.graph().traversal().E().count().next());
assertEquals(Long.valueOf(0l), result.graph().traversal().V().values().count().next());
assertEquals(Long.valueOf(0l), result.graph().traversal().E().values().count().next());
- assertEquals(Double.valueOf(0.0d), result.graph().traversal().V().values("money").sum().next());
+ assertEquals(0, result.graph().traversal().V().values("money").sum().next());
///
assertEquals(Long.valueOf(6l), graph.traversal().V().count().next());
assertEquals(Long.valueOf(6l), graph.traversal().E().count().next());
assertEquals(Long.valueOf(12l), graph.traversal().V().values().count().next());
assertEquals(Long.valueOf(6l), graph.traversal().E().values().count().next());
- assertEquals(Double.valueOf(0.0d), graph.traversal().V().values("money").sum().next());
+ assertEquals(0, graph.traversal().V().values("money").sum().next());
}
}
@@ -1243,13 +1246,13 @@ public class GraphComputerTest extends AbstractGremlinProcessTest {
assertEquals(Long.valueOf(0l), result.graph().traversal().E().count().next());
assertEquals(Long.valueOf(18l), result.graph().traversal().V().values().count().next());
assertEquals(Long.valueOf(0l), result.graph().traversal().E().values().count().next());
- assertEquals(Double.valueOf(28.0d), result.graph().traversal().V().values("money").sum().next());
+ assertEquals(28l, result.graph().traversal().V().values("money").sum().next());
///
assertEquals(Long.valueOf(6l), graph.traversal().V().count().next());
assertEquals(Long.valueOf(6l), graph.traversal().E().count().next());
assertEquals(Long.valueOf(12l), graph.traversal().V().values().count().next());
assertEquals(Long.valueOf(6l), graph.traversal().E().values().count().next());
- assertEquals(Double.valueOf(0.0d), graph.traversal().V().values("money").sum().next());
+ assertEquals(0, graph.traversal().V().values("money").sum().next());
}
}
@@ -1263,13 +1266,13 @@ public class GraphComputerTest extends AbstractGremlinProcessTest {
assertEquals(Long.valueOf(6l), result.graph().traversal().E().count().next());
assertEquals(Long.valueOf(18l), result.graph().traversal().V().values().count().next());
assertEquals(Long.valueOf(6l), result.graph().traversal().E().values().count().next());
- assertEquals(Double.valueOf(28.0d), result.graph().traversal().V().values("money").sum().next());
+ assertEquals(28l, result.graph().traversal().V().values("money").sum().next());
///
assertEquals(Long.valueOf(6l), graph.traversal().V().count().next());
assertEquals(Long.valueOf(6l), graph.traversal().E().count().next());
assertEquals(Long.valueOf(12l), graph.traversal().V().values().count().next());
assertEquals(Long.valueOf(6l), graph.traversal().E().values().count().next());
- assertEquals(Double.valueOf(0.0d), graph.traversal().V().values("money").sum().next());
+ assertEquals(0, graph.traversal().V().values("money").sum().next());
}
}
@@ -1283,13 +1286,13 @@ public class GraphComputerTest extends AbstractGremlinProcessTest {
assertEquals(Long.valueOf(6l), result.graph().traversal().E().count().next());
assertEquals(Long.valueOf(12l), result.graph().traversal().V().values().count().next());
assertEquals(Long.valueOf(6l), result.graph().traversal().E().values().count().next());
- assertEquals(Double.valueOf(0.0d), result.graph().traversal().V().values("money").sum().next());
+ assertEquals(0, result.graph().traversal().V().values("money").sum().next());
///
assertEquals(Long.valueOf(6l), graph.traversal().V().count().next());
assertEquals(Long.valueOf(6l), graph.traversal().E().count().next());
assertEquals(Long.valueOf(12l), graph.traversal().V().values().count().next());
assertEquals(Long.valueOf(6l), graph.traversal().E().values().count().next());
- assertEquals(Double.valueOf(0.0d), graph.traversal().V().values("money").sum().next());
+ assertEquals(0, graph.traversal().V().values("money").sum().next());
}
}
@@ -1303,13 +1306,13 @@ public class GraphComputerTest extends AbstractGremlinProcessTest {
assertEquals(Long.valueOf(6l), result.graph().traversal().E().count().next());
assertEquals(Long.valueOf(18l), result.graph().traversal().V().values().count().next());
assertEquals(Long.valueOf(6l), result.graph().traversal().E().values().count().next());
- assertEquals(Double.valueOf(28.0d), result.graph().traversal().V().values("money").sum().next());
+ assertEquals(28l, result.graph().traversal().V().values("money").sum().next());
///
assertEquals(Long.valueOf(6l), graph.traversal().V().count().next());
assertEquals(Long.valueOf(6l), graph.traversal().E().count().next());
assertEquals(Long.valueOf(18l), graph.traversal().V().values().count().next());
assertEquals(Long.valueOf(6l), graph.traversal().E().values().count().next());
- assertEquals(Double.valueOf(28.0d), graph.traversal().V().values("money").sum().next());
+ assertEquals(28l, graph.traversal().V().values("money").sum().next());
}
}
@@ -1323,13 +1326,13 @@ public class GraphComputerTest extends AbstractGremlinProcessTest {
assertEquals(Long.valueOf(6l), result.graph().traversal().E().count().next());
assertEquals(Long.valueOf(18l), result.graph().traversal().V().values().count().next());
assertEquals(Long.valueOf(6l), result.graph().traversal().E().values().count().next());
- assertEquals(Double.valueOf(28.0d), result.graph().traversal().V().values("money").sum().next());
+ assertEquals(28l, result.graph().traversal().V().values("money").sum().next());
///
assertEquals(Long.valueOf(6l), graph.traversal().V().count().next());
assertEquals(Long.valueOf(6l), graph.traversal().E().count().next());
assertEquals(Long.valueOf(18l), graph.traversal().V().values().count().next());
assertEquals(Long.valueOf(6l), graph.traversal().E().values().count().next());
- assertEquals(Double.valueOf(28.0d), graph.traversal().V().values("money").sum().next());
+ assertEquals(28l, graph.traversal().V().values("money").sum().next());
}
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d49deafb/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumTest.java
----------------------------------------------------------------------
diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumTest.java b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumTest.java
index 6e2eabb..1183863 100644
--- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumTest.java
+++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/SumTest.java
@@ -49,7 +49,7 @@ public abstract class SumTest extends AbstractGremlinProcessTest {
final Traversal<Vertex, Double> traversal = get_g_V_valuesXageX_sum();
printTraversalForm(traversal);
final Number sum = traversal.next();
- assertEquals(123.0, sum);
+ assertEquals(123L, sum);
assertFalse(traversal.hasNext());
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d49deafb/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroupTest.java
----------------------------------------------------------------------
diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroupTest.java b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroupTest.java
index 9bfdf8e..b2298be 100644
--- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroupTest.java
+++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroupTest.java
@@ -33,8 +33,14 @@ import java.util.Map;
import static org.apache.tinkerpop.gremlin.LoadGraphWith.GraphData.GRATEFUL;
import static org.apache.tinkerpop.gremlin.LoadGraphWith.GraphData.MODERN;
-import static org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__.*;
-import static org.junit.Assert.*;
+import static org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__.both;
+import static org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__.constant;
+import static org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__.count;
+import static org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__.out;
+import static org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__.outE;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
/**
* @author Marko A. Rodriguez (http://markorodriguez.com)
@@ -57,7 +63,7 @@ public abstract class GroupTest extends AbstractGremlinProcessTest {
public abstract Traversal<Vertex, Map<Long, Collection<String>>> get_g_V_group_byXoutE_countX_byXnameX();
- public abstract Traversal<Vertex, Map<String, Double>> get_g_V_groupXaX_byXlabelX_byXoutE_weight_sumX_capXaX();
+ public abstract Traversal<Vertex, Map<String, Number>> get_g_V_groupXaX_byXlabelX_byXoutE_weight_sumX_capXaX();
public abstract Traversal<Vertex, Map<String, Long>> get_g_V_repeatXbothXfollowedByXX_timesX2X_group_byXsongTypeX_byXcountX();
@@ -174,14 +180,14 @@ public abstract class GroupTest extends AbstractGremlinProcessTest {
@Test
@LoadGraphWith(MODERN)
public void g_V_groupXaX_byXlabelX_byXoutE_weight_sumX_capXaX() {
- final Traversal<Vertex, Map<String, Double>> traversal = get_g_V_groupXaX_byXlabelX_byXoutE_weight_sumX_capXaX();
+ final Traversal<Vertex, Map<String, Number>> traversal = get_g_V_groupXaX_byXlabelX_byXoutE_weight_sumX_capXaX();
printTraversalForm(traversal);
assertTrue(traversal.hasNext());
- final Map<String, Double> map = traversal.next();
+ final Map<String, Number> map = traversal.next();
assertFalse(traversal.hasNext());
assertEquals(2, map.size());
- assertEquals(0.0d, map.get("software"), 0.01d);
- assertEquals(3.5d, map.get("person"), 0.01d);
+ assertEquals(0, map.get("software"));
+ assertEquals(3.5d, (double) map.get("person"), 0.01d);
}
@Test
@@ -277,7 +283,7 @@ public abstract class GroupTest extends AbstractGremlinProcessTest {
}
@Override
- public Traversal<Vertex, Map<String, Double>> get_g_V_groupXaX_byXlabelX_byXoutE_weight_sumX_capXaX() {
+ public Traversal<Vertex, Map<String, Number>> get_g_V_groupXaX_byXlabelX_byXoutE_weight_sumX_capXaX() {
return g.V().<String, Double>group("a").by(T.label).by(outE().values("weight").sum()).cap("a");
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d49deafb/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/verification/ComputerVerificationStrategyProcessTest.java
----------------------------------------------------------------------
diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/verification/ComputerVerificationStrategyProcessTest.java b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/verification/ComputerVerificationStrategyProcessTest.java
index 1402ee2..f6ded69 100644
--- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/verification/ComputerVerificationStrategyProcessTest.java
+++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/verification/ComputerVerificationStrategyProcessTest.java
@@ -25,6 +25,7 @@ import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversal;
import org.junit.Test;
import static org.apache.tinkerpop.gremlin.LoadGraphWith.GraphData.MODERN;
+import static org.apache.tinkerpop.gremlin.process.traversal.NumberHelper.mul;
import static org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__.*;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
@@ -90,7 +91,7 @@ public abstract class ComputerVerificationStrategyProcessTest extends AbstractGr
}
try {
- final GraphTraversal t = g.V().count().sum().map(x -> x.get() * 19).iterate();
+ final GraphTraversal t = g.V().count().sum().map(x -> mul(x.get(), 19)).iterate();
fail("Mid-traversal barrier steps are not allowed (COMPUTER): " + t);
} catch (IllegalStateException e) {
assertTrue(true);
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d49deafb/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPluginTest.java
----------------------------------------------------------------------
diff --git a/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPluginTest.java b/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPluginTest.java
index 83a206d..0d7da18 100644
--- a/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPluginTest.java
+++ b/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPluginTest.java
@@ -88,7 +88,7 @@ public class HadoopGremlinPluginTest extends AbstractGremlinTest {
this.remote.configure(Arrays.asList("useSugar", "true"));
this.remote.connect(Arrays.asList("graph", "g"));
Traversal<?, ?> traversal = (Traversal<?, ?>) this.remote.submit(Arrays.asList("g.V.name.map{it.length()}.sum"));
- assertEquals(28.0d, traversal.next());
+ assertEquals(28l, traversal.next());
assertFalse(traversal.hasNext());
assertNotNull(this.console.getBindings().get(RemoteAcceptor.RESULT));
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d49deafb/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputRDDTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputRDDTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputRDDTest.java
index 3070fea..46d735f 100644
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputRDDTest.java
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputRDDTest.java
@@ -48,7 +48,7 @@ public class InputRDDTest {
configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
////////
Graph graph = GraphFactory.open(configuration);
- assertEquals(Double.valueOf(123.0d), graph.traversal(GraphTraversalSource.computer(SparkGraphComputer.class)).V().values("age").sum().next());
+ assertEquals(123l, graph.traversal(GraphTraversalSource.computer(SparkGraphComputer.class)).V().values("age").sum().next());
assertEquals(Long.valueOf(4l), graph.traversal(GraphTraversalSource.computer(SparkGraphComputer.class)).V().count().next());
}
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/d49deafb/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java
----------------------------------------------------------------------
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java
index 06994e2..efb4a5b 100644
--- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java
+++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java
@@ -192,20 +192,12 @@ public class TinkerGraphPlayTest {
@Test
@Ignore
public void testPlayDK() throws Exception {
- final Graph graph = TinkerFactory.createModern();
- final GraphTraversalSource g = graph.traversal();
- Traversal traversal = g.V().where(out().and().in()).profile().cap(TraversalMetrics.METRICS_KEY);
- //traversal.forEachRemaining(System.out::println);
- System.out.println(traversal.toString());
- traversal.asAdmin().applyStrategies();
- System.out.println(traversal.toString());
- traversal.forEachRemaining(System.out::println);
- traversal = g.V().where(and(out(), in())).profile().cap(TraversalMetrics.METRICS_KEY);
- //traversal.forEachRemaining(System.out::println);
- System.out.println(traversal.toString());
- traversal.asAdmin().applyStrategies();
- System.out.println(traversal.toString());
- //System.out.println(traversal.toString());
+ final Number n = __.inject(1, 2L, 3f).sum().next();
+ System.out.println(n.toString());
+ System.out.println(n.getClass().toString());
+ final Double d = __.inject(1, 2L, 3f).<Double>sum().next();
+ System.out.println(d.toString());
+ System.out.println(d.getClass().toString());
}
@Test
[38/50] [abbrv] incubator-tinkerpop git commit: MatchStep
CountMatchAlgorithm is smart to bias patterns that don't reqiure message
passing in OLAP.
Posted by sp...@apache.org.
MatchStep CountMatchAlgorithm is smart to bias patterns that don't reqiure message passing in OLAP.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/b338f962
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/b338f962
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/b338f962
Branch: refs/heads/TINKERPOP3-923
Commit: b338f9625fdadcb991ac8d5414641f190add24c6
Parents: 558c04e f644fb4
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Wed Nov 4 06:42:40 2015 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Wed Nov 4 06:42:40 2015 -0700
----------------------------------------------------------------------
CHANGELOG.asciidoc | 2 +
.../process/traversal/step/map/MatchStep.java | 46 +++++++----
.../traversal/step/map/MatchStepTest.java | 82 +++++++++++++++++++-
3 files changed, 115 insertions(+), 15 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/b338f962/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --cc CHANGELOG.asciidoc
index 5ad89e5,e4eb554..31671c0
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@@ -25,7 -25,8 +25,9 @@@ image::https://raw.githubusercontent.co
TinkerPop 3.1.0 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Integrated `NumberHelper` in `SumStep`, `MinStep`, `MaxStep` and `MeanStep` (local and global step variants).
+ * `CountMatchAlgorithm`, in OLAP, now biases traversal selection towards those traversals that start at the current traverser location to reduce message passing.
+ * Fixed a file stream bug in Hadoop OLTP that showed up if the streamed file was more than 2G of data.
* Bumped to Neo4j 2.3.0.
* Added `PersistedInputRDD` and `PersistedOutputRDD` which enables `SparkGraphComputer` to store the graph RDD in the context between jobs (no HDFS serialization required).
* Renamed the `public static String` configuration variable names of TinkerGraph (deprecated old variables).
[25/50] [abbrv] incubator-tinkerpop git commit: TINKERPOP-911: Lets
SparkGraphComputer Set ThreadLocal Properties
Posted by sp...@apache.org.
TINKERPOP-911: Lets SparkGraphComputer Set ThreadLocal Properties
When running in persistent context mode, the configuration for the Spark
Context is always inherited from the origin configuration. This can be
overridden for select properties using setLocalProperty. By enabling
this for SparkGraphComputer a user will be able to take advantage of the
FairScheduler system with spark or add custom Spark JobGroupIDs to jobs in the
same JVM.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/82229d0b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/82229d0b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/82229d0b
Branch: refs/heads/TINKERPOP3-923
Commit: 82229d0bef3da7bb125e9b31fc5ef559a8868af0
Parents: aadd422
Author: Russell Spitzer <Ru...@gmail.com>
Authored: Mon Nov 2 15:22:45 2015 -0800
Committer: Russell Spitzer <Ru...@gmail.com>
Committed: Mon Nov 2 18:55:36 2015 -0800
----------------------------------------------------------------------
.../process/computer/SparkGraphComputer.java | 32 ++++++
.../process/computer/LocalPropertyTest.java | 102 +++++++++++++++++++
2 files changed, 134 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/82229d0b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
index b25073b..6425aab 100644
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
@@ -135,6 +135,7 @@ public final class SparkGraphComputer extends AbstractHadoopGraphComputer {
JavaSparkContext sparkContext = null;
try {
sparkContext = new JavaSparkContext(SparkContext.getOrCreate(sparkConfiguration));
+ updateLocalConfiguration(sparkContext, sparkConfiguration);
// add the project jars to the cluster
this.loadJars(sparkContext, hadoopConfiguration);
// create a message-passing friendly rdd from the input rdd
@@ -246,6 +247,37 @@ public final class SparkGraphComputer extends AbstractHadoopGraphComputer {
}
}
+ /**
+ * When using a persistent context the running Context's configuration will override a passed
+ * in configuration. Spark allows us to override these inherited properties via
+ * SparkContext.setLocalProperty
+ */
+ private void updateLocalConfiguration(final JavaSparkContext sparkContext, final SparkConf sparkConfiguration) {
+ /*
+ * While we could enumerate over the entire SparkConfiguration and copy into the Thread
+ * Local properties of the Spark Context this could cause adverse effects with future
+ * versions of Spark. Since the api for setting multiple local properties at once is
+ * restricted as private, we will only set those properties we know can effect SparkGraphComputer
+ * Execution rather than applying the entire configuration.
+ */
+ final String[] validPropertyNames = {
+ "spark.job.description",
+ "spark.jobGroup.id",
+ "spark.job.interruptOnCancel",
+ "spark.scheduler.pool"
+ };
+
+ for (String propertyName: validPropertyNames){
+ if (sparkConfiguration.contains(propertyName)){
+ String propertyValue = sparkConfiguration.get(propertyName);
+ this.logger.info("Setting Thread Local SparkContext Property - "
+ + propertyName + " : " + propertyValue);
+
+ sparkContext.setLocalProperty(propertyName, sparkConfiguration.get(propertyName));
+ }
+ }
+ }
+
public static void main(final String[] args) throws Exception {
final FileConfiguration configuration = new PropertiesConfiguration(args[0]);
new SparkGraphComputer(HadoopGraph.open(configuration)).program(VertexProgram.createVertexProgram(HadoopGraph.open(configuration), configuration)).submit().get();
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/82229d0b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/LocalPropertyTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/LocalPropertyTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/LocalPropertyTest.java
new file mode 100644
index 0000000..57c3526
--- /dev/null
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/LocalPropertyTest.java
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tinkerpop.gremlin.spark.process.computer;
+
+import org.apache.commons.configuration.BaseConfiguration;
+import org.apache.commons.configuration.Configuration;
+import org.apache.spark.SparkConf;
+import org.apache.spark.SparkContext;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.api.java.JavaSparkStatusTracker;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat;
+import org.apache.tinkerpop.gremlin.hadoop.structure.util.ConfUtil;
+import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
+import org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram;
+import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
+import org.apache.tinkerpop.gremlin.process.traversal.engine.ComputerTraversalEngine;
+import org.apache.tinkerpop.gremlin.spark.structure.io.PersistedInputRDD;
+import org.apache.tinkerpop.gremlin.spark.structure.io.PersistedOutputRDD;
+import org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer;
+import org.apache.tinkerpop.gremlin.structure.Graph;
+import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
+import org.junit.Test;
+
+import java.util.UUID;
+
+import static org.junit.Assert.assertTrue;
+
+public class LocalPropertyTest
+{
+
+
+ @Test
+ public void shouldSetThreadLocalProperties() throws Exception
+ {
+ final String testName = "ThreadLocalProperties";
+ final String rddName = "target/test-output/" + UUID.randomUUID();
+ final Configuration configuration = new BaseConfiguration();
+ configuration.setProperty("spark.master", "local[4]");
+ configuration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
+ configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, PersistedOutputRDD.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, rddName);
+ configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
+ configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, true);
+ configuration.setProperty("spark.jobGroup.id", "22");
+ Graph graph = GraphFactory.open(configuration);
+ graph.compute(SparkGraphComputer.class)
+ .result(GraphComputer.ResultGraph.NEW)
+ .persist(GraphComputer.Persist.EDGES)
+ .program(TraversalVertexProgram.build()
+ .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
+ "gremlin-groovy",
+ "g.V()").create(graph)).submit().get();
+ ////////
+ SparkConf sparkConfiguration = new SparkConf();
+ sparkConfiguration.setAppName(testName);
+ ConfUtil.makeHadoopConfiguration(configuration).forEach(entry -> sparkConfiguration.set(entry.getKey(), entry.getValue()));
+ JavaSparkContext sparkContext = new JavaSparkContext(SparkContext.getOrCreate(sparkConfiguration));
+ JavaSparkStatusTracker statusTracker = sparkContext.statusTracker();
+ assertTrue(statusTracker.getJobIdsForGroup("22").length >= 1);
+ assertTrue(PersistedInputRDD.getPersistedRDD(sparkContext, rddName).isPresent());
+ ///////
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, PersistedInputRDD.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, rddName);
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, null);
+ configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, null);
+ configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, false);
+ configuration.setProperty("spark.jobGroup.id", "44");
+ graph = GraphFactory.open(configuration);
+ graph.compute(SparkGraphComputer.class)
+ .result(GraphComputer.ResultGraph.NEW)
+ .persist(GraphComputer.Persist.NOTHING)
+ .program(TraversalVertexProgram.build()
+ .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
+ "gremlin-groovy",
+ "g.V()").create(graph)).submit().get();
+ ///////
+ assertTrue(statusTracker.getJobIdsForGroup("44").length >= 1);
+ }
+}
+
[03/50] [abbrv] incubator-tinkerpop git commit: found a wacky named
test in AddVertexTest. Fixed it. CTR.
Posted by sp...@apache.org.
found a wacky named test in AddVertexTest. Fixed it. CTR.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/ac0924d3
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/ac0924d3
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/ac0924d3
Branch: refs/heads/TINKERPOP3-923
Commit: ac0924d3271157b4c1e3ac8984593d88a0b75288
Parents: 522c256
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Thu Oct 29 16:22:57 2015 -0600
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Thu Oct 29 16:22:57 2015 -0600
----------------------------------------------------------------------
.../gremlin/process/traversal/step/map/AddVertexTest.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/ac0924d3/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/AddVertexTest.java
----------------------------------------------------------------------
diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/AddVertexTest.java b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/AddVertexTest.java
index 3b2f8b5..5a616f8 100644
--- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/AddVertexTest.java
+++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/AddVertexTest.java
@@ -209,7 +209,7 @@ public abstract class AddVertexTest extends AbstractGremlinTest {
@FeatureRequirement(featureClass = Graph.Features.VertexFeatures.class, feature = Graph.Features.VertexFeatures.FEATURE_ADD_VERTICES)
@FeatureRequirement(featureClass = Graph.Features.VertexFeatures.class, feature = Graph.Features.VertexFeatures.FEATURE_ADD_PROPERTY)
@FeatureRequirement(featureClass = Graph.Features.VertexFeatures.class, feature = Graph.Features.VertexFeatures.FEATURE_MULTI_PROPERTIES)
- public void xxx() {
+ public void g_V_addVXanimalX_propertyXname_valuesXnameXX_propertyXname_an_animalX_propertyXvaluesXnameX_labelX() {
final Traversal<Vertex, Vertex> traversal = get_g_V_addVXanimalX_propertyXname_valuesXnameXX_propertyXname_an_animalX_propertyXvaluesXnameX_labelX();
printTraversalForm(traversal);
while (traversal.hasNext()) {
[24/50] [abbrv] incubator-tinkerpop git commit: tweaks to upgrade
docs.
Posted by sp...@apache.org.
tweaks to upgrade docs.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/aadd4221
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/aadd4221
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/aadd4221
Branch: refs/heads/TINKERPOP3-923
Commit: aadd4221200be353da0cd8369c135ab9396c55cc
Parents: 0dc2dbc
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Mon Nov 2 08:51:49 2015 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Mon Nov 2 08:51:49 2015 -0700
----------------------------------------------------------------------
.../src/upgrade-release-3.1.x-incubating.asciidoc | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/aadd4221/docs/src/upgrade-release-3.1.x-incubating.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/upgrade-release-3.1.x-incubating.asciidoc b/docs/src/upgrade-release-3.1.x-incubating.asciidoc
index 77a448f..9e5b50d 100644
--- a/docs/src/upgrade-release-3.1.x-incubating.asciidoc
+++ b/docs/src/upgrade-release-3.1.x-incubating.asciidoc
@@ -40,7 +40,7 @@ Important Changes
* The data type of `Operator` enums will now always be the highest common data type of the two given numbers, rather than the data type of the first number, as it's been before.
* The Gephi Plugin has improved integration with Gephi, where manually inserting `store` steps to visualize a running traversal is no longer required.
* Entire TinkerGraph instances can be serialized over Gryo.
-* Hadoop1 support has been dropped. Hadoop2 is now supported. Giraph 1.1.0 is now supported and Spark of Hadoop2 YARN.
+* Hadoop1 support has been dropped. Hadoop2 is now supported. Giraph and Spark can work over Hadoop2 YARN.
* The implementation and semantics of `GraphTraversal.group()` has changed. The previous model is deprecated and renamed to `groupV3d0()`.
* `PartitionStrategy` now supports `VertexProperty` such those properties can be added to partitions.
* The Jackson library is now "shaded" and included in the `gremlin-shaded` module.
@@ -69,14 +69,22 @@ Hadoop-Gremlin Updates
++++++++++++++++++++++
* Hadoop1 is no longer supported. Hadoop2 is now the only supported Hadoop version in TinkerPop.
+* Spark and Giraph have been split out of Hadoop-Gremlin into their own respective packages (Spark-Gremlin and Giraph-Gremlin).
* The directory where application jars are stored in HDFS is now `hadoop-gremlin-x.y.z-libs`.
** This versioning is important so that cross-version TinkerPop use does not cause jar conflicts.
+Spark-Gremlin Updates
++++++++++++++++++++++
+
+* Providers that wish to reuse a graphRDD can leverage the new `PersistedInputRDD` and `PersistedOutputRDD`.
+** This allows the graphRDD to avoid serialization into HDFS for reuse. Be sure to enabled persisted `SparkContext` (see documentation).
+
TinkerGraph-Gremlin Updates
+++++++++++++++++++++++++++
* The `public static String` configurations have been renamed. The old `public static` variables have been deprecated.
+
Gremlin Process
^^^^^^^^^^^^^^^
@@ -168,7 +176,7 @@ See link:https://issues.apache.org/jira/browse/TINKERPOP3-876[TINKERPOP3-876] fo
GraphComputer Updates
+++++++++++++++++++++
-`GraphComputer.configure(String key, Object value)` is now a method. This allows the user to specify engine-specific
-parameters to the underlying OLAP system. These parameters are not intended to be cross engine supported. Moreover, if
-there are not parameters that can be altered (beyond the standard `GraphComputer` methods), then the provider's `GraphComputer`
-implementation should simply return and do nothing.
\ No newline at end of file
+`GraphComputer.configure(String key, Object value)` is now a method (with default implementation).
+This allows the user to specify engine-specific parameters to the underlying OLAP system. These parameters are not intended
+to be cross engine supported. Moreover, if there are not parameters that can be altered (beyond the standard `GraphComputer`
+methods), then the provider's `GraphComputer` implementation should simply return and do nothing.
\ No newline at end of file
[48/50] [abbrv] incubator-tinkerpop git commit: Added notes about the
deprecation label to the dev docs.
Posted by sp...@apache.org.
Added notes about the deprecation label to the dev docs.
This was done per discussion here: https://pony-poc.apache.org/thread.html/9dbdba0ad7970b2671d0a9b36ffa3d389a0125747bc84eea109ce1e7@1446038620@%3Cdev.tinkerpop.apache.org%3E
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/151f64b4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/151f64b4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/151f64b4
Branch: refs/heads/TINKERPOP3-923
Commit: 151f64b4ee28f7ad5d097d347f27565409e9b6fd
Parents: b2daf09
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Wed Nov 4 15:41:27 2015 -0500
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Wed Nov 4 15:41:27 2015 -0500
----------------------------------------------------------------------
docs/src/developer-contributing.asciidoc | 17 +++++++++--------
1 file changed, 9 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/151f64b4/docs/src/developer-contributing.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/developer-contributing.asciidoc b/docs/src/developer-contributing.asciidoc
index fcdade0..dd4578f 100644
--- a/docs/src/developer-contributing.asciidoc
+++ b/docs/src/developer-contributing.asciidoc
@@ -165,8 +165,8 @@ Developers should remove their own branches when they are no longer needed.
Tags
~~~~
-Tags are used for milestones, release candidates, and approved releases. Please
-refrain from creating arbitrary tags, as they produce permanent clutter.
+Tags are used for milestones, release candidates, and approved releases. Please refrain from creating arbitrary
+tags, as they produce permanent clutter.
Issue Tracker Conventions
~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -187,9 +187,10 @@ reasonable workaround. Given that definition, a `bug` should generally have ver
** Everything else is an `improvement` in the sense that any other work is an enhancement to the current codebase.
* The "component" should be representative of the primary area of code that it applies to and all issues should have
this property set.
-* Issues are not assigned "labels" with one exception: the "breaking" label. The "breaking" label marks an issue
-as one that is representative of a change in the API that might affect users or vendors. This label is important when
-organizing release notes.
+* Issues are not assigned "labels" with two exceptions:
+** The "breaking" label which marks an issue as one that is representative of a change in the API that might
+affect users or vendors. This label is important when organizing release notes.
+** The "deprecation" label which is assigned to an issue that is about removing a deprecated portion of the API.
* The "affects/fix version(s)" fields should be appropriately set, where the "fix version" implies the version on
which that particular issue will completed.
* The "priority" field can be arbitrarily applied with one exception. The "trivial" option should be reserved for
@@ -217,14 +218,14 @@ replaced then the comment can simply read "not replaced". Additional comments t
encouraged.
** `@see <a href="https://issues.apache.org/jira/browse/TINKERPOP3-XXX">TINKERPOP3-XXX</a>` - supply a link to the
JIRA issue for reference.
-* All deprecation should be tied to a JIRA issue with a "breaking" label - the issue itself does not need to
+* All deprecation should typically be tied to a JIRA issue with a "breaking" label - the issue itself does not need to
specifically or solely be about "deprecation" but it should be documented very clearly in the comments what was
deprecated and what the path forward should be.
* Be sure that deprecated methods are still under test - consider using javadoc/comments in the tests themselves to
call out this fact.
* Create a new JIRA issue to track removal of the deprecation for future evaluation - this issue should have the
-"breaking" label.
-* Provide a post to the developers and/or users mailing lists as the case requires to alert the community to the change.
+"breaking" label as well as a "deprecation" label.
+* Update the "upgrade documentation" to reflect the API change and how the reader should resolve it.
The JIRA issues that track removal of deprecated methods should be periodically evaluated to determine if it is
prudent to schedule them into a release.
[19/50] [abbrv] incubator-tinkerpop git commit: Integrated
NumberHelper in Min-, Max- and MeanStep
Posted by sp...@apache.org.
Integrated NumberHelper in Min-, Max- and MeanStep
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/eba71143
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/eba71143
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/eba71143
Branch: refs/heads/TINKERPOP3-923
Commit: eba7114341b1257106db2111ed8b90b2012c0860
Parents: eda16b5
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Sun Nov 1 12:55:26 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Sun Nov 1 12:55:26 2015 +0100
----------------------------------------------------------------------
.../gremlin/process/traversal/NumberHelper.java | 12 ++++-
.../traversal/step/map/MaxGlobalStep.java | 14 ++---
.../traversal/step/map/MaxLocalStep.java | 5 +-
.../traversal/step/map/MeanGlobalStep.java | 57 +++++++++++---------
.../traversal/step/map/MinGlobalStep.java | 14 ++---
.../traversal/step/map/MinLocalStep.java | 5 +-
.../structure/TinkerGraphPlayTest.java | 9 ++--
7 files changed, 67 insertions(+), 49 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/eba71143/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/NumberHelper.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/NumberHelper.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/NumberHelper.java
index 8dfd113..b094f29 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/NumberHelper.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/NumberHelper.java
@@ -162,8 +162,12 @@ public class NumberHelper {
}
public static Class<? extends Number> getHighestCommonNumberClass(final Number... numbers) {
+ return getHighestCommonNumberClass(false, numbers);
+ }
+
+ public static Class<? extends Number> getHighestCommonNumberClass(final boolean forceFloatingPoint, final Number... numbers) {
int bits = 8;
- boolean fp = false;
+ boolean fp = forceFloatingPoint;
for (final Number number : numbers) {
if (number == null) continue;
final Class<? extends Number> clazz = number.getClass();
@@ -207,7 +211,11 @@ public class NumberHelper {
}
public static Number div(final Number a, final Number b) {
- final Class<? extends Number> clazz = getHighestCommonNumberClass(a, b);
+ return div(a, b, false);
+ }
+
+ public static Number div(final Number a, final Number b, final boolean forceFloatingPoint) {
+ final Class<? extends Number> clazz = getHighestCommonNumberClass(forceFloatingPoint, a, b);
return getHelper(clazz).div.apply(a, b);
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/eba71143/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MaxGlobalStep.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MaxGlobalStep.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MaxGlobalStep.java
index d7dc34d..d876d2d 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MaxGlobalStep.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MaxGlobalStep.java
@@ -37,6 +37,8 @@ import java.util.Iterator;
import java.util.Set;
import java.util.function.BiFunction;
+import static org.apache.tinkerpop.gremlin.process.traversal.NumberHelper.max;
+
/**
* @author Marko A. Rodriguez (http://markorodriguez.com)
*/
@@ -70,10 +72,11 @@ public final class MaxGlobalStep<S extends Number> extends ReducingBarrierStep<S
@Override
public S apply(final S mutatingSeed, final Traverser<S> traverser) {
- return mutatingSeed != null && mutatingSeed.doubleValue() >= traverser.get().doubleValue() ? mutatingSeed : traverser.get();
+ final S value = traverser.get();
+ return mutatingSeed != null ? (S) max(mutatingSeed, traverser.get()) : value;
}
- public final static <S extends Number> MaxGlobalBiFunction<S> instance() {
+ public static <S extends Number> MaxGlobalBiFunction<S> instance() {
return INSTANCE;
}
}
@@ -106,11 +109,10 @@ public final class MaxGlobalStep<S extends Number> extends ReducingBarrierStep<S
@Override
public void reduce(final NullObject key, final Iterator<Number> values, final ReduceEmitter<NullObject, Number> emitter) {
if (values.hasNext()) {
- Number max = -Double.MAX_VALUE;
+ Number max = null;
while (values.hasNext()) {
final Number value = values.next();
- if (value.doubleValue() > max.doubleValue())
- max = value;
+ max = max != null ? max(value, max) : value;
}
emitter.emit(max);
}
@@ -127,7 +129,7 @@ public final class MaxGlobalStep<S extends Number> extends ReducingBarrierStep<S
}
- public static final MaxGlobalMapReduce instance() {
+ public static MaxGlobalMapReduce instance() {
return INSTANCE;
}
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/eba71143/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MaxLocalStep.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MaxLocalStep.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MaxLocalStep.java
index 6d2a228..dc4c5ce 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MaxLocalStep.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MaxLocalStep.java
@@ -26,6 +26,8 @@ import java.util.Collections;
import java.util.Iterator;
import java.util.Set;
+import static org.apache.tinkerpop.gremlin.process.traversal.NumberHelper.max;
+
/**
* @author Marko A. Rodriguez (http://markorodriguez.com)
* @author Daniel Kuppitz (http://gremlin.guru)
@@ -43,8 +45,7 @@ public final class MaxLocalStep<E extends Number, S extends Iterable<E>> extends
if (iterator.hasNext()) {
result = iterator.next();
while (iterator.hasNext()) {
- final Number curr = iterator.next();
- if (result.doubleValue() < curr.doubleValue()) result = curr;
+ result = max(iterator.next(), result);
}
} else {
result = Double.NaN;
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/eba71143/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanGlobalStep.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanGlobalStep.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanGlobalStep.java
index 66092a5..f658271 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanGlobalStep.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanGlobalStep.java
@@ -18,17 +18,18 @@
*/
package org.apache.tinkerpop.gremlin.process.traversal.step.map;
-import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
-import org.apache.tinkerpop.gremlin.process.traversal.Traverser;
import org.apache.tinkerpop.gremlin.process.computer.KeyValue;
import org.apache.tinkerpop.gremlin.process.computer.MapReduce;
import org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram;
import org.apache.tinkerpop.gremlin.process.computer.util.StaticMapReduce;
-import org.apache.tinkerpop.gremlin.process.traversal.step.util.ReducingBarrierStep;
+import org.apache.tinkerpop.gremlin.process.traversal.NumberHelper;
+import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
+import org.apache.tinkerpop.gremlin.process.traversal.Traverser;
import org.apache.tinkerpop.gremlin.process.traversal.step.MapReducer;
+import org.apache.tinkerpop.gremlin.process.traversal.step.util.FinalGet;
+import org.apache.tinkerpop.gremlin.process.traversal.step.util.ReducingBarrierStep;
import org.apache.tinkerpop.gremlin.process.traversal.traverser.TraverserRequirement;
import org.apache.tinkerpop.gremlin.process.traversal.traverser.util.TraverserSet;
-import org.apache.tinkerpop.gremlin.process.traversal.step.util.FinalGet;
import org.apache.tinkerpop.gremlin.structure.Vertex;
import org.apache.tinkerpop.gremlin.util.function.MeanNumberSupplier;
@@ -39,6 +40,10 @@ import java.util.Set;
import java.util.function.BiFunction;
import java.util.function.Supplier;
+import static org.apache.tinkerpop.gremlin.process.traversal.NumberHelper.add;
+import static org.apache.tinkerpop.gremlin.process.traversal.NumberHelper.div;
+import static org.apache.tinkerpop.gremlin.process.traversal.NumberHelper.mul;
+
/**
* @author Marko A. Rodriguez (http://markorodriguez.com)
* @author Daniel Kuppitz (http://gremlin.guru)
@@ -59,7 +64,7 @@ public final class MeanGlobalStep<S extends Number, E extends Number> extends Re
}
@Override
- public MapReduce<Number, Long, Number, Long, Double> getMapReduce() {
+ public MapReduce<Number, Long, Number, Long, Number> getMapReduce() {
return MeanGlobalMapReduce.instance();
}
@@ -78,14 +83,14 @@ public final class MeanGlobalStep<S extends Number, E extends Number> extends Re
return (S) ((MeanNumber) mutatingSeed).add(traverser.get(), traverser.bulk());
}
- public final static <S extends Number> MeanGlobalBiFunction<S> instance() {
+ public static <S extends Number> MeanGlobalBiFunction<S> instance() {
return INSTANCE;
}
}
///////////
- private static final class MeanGlobalMapReduce extends StaticMapReduce<Number, Long, Number, Long, Double> {
+ private static final class MeanGlobalMapReduce extends StaticMapReduce<Number, Long, Number, Long, Number> {
private static final MeanGlobalMapReduce INSTANCE = new MeanGlobalMapReduce();
@@ -123,81 +128,83 @@ public final class MeanGlobalStep<S extends Number, E extends Number> extends Re
}
@Override
- public Double generateFinalResult(final Iterator<KeyValue<Number, Long>> keyValues) {
+ public Number generateFinalResult(final Iterator<KeyValue<Number, Long>> keyValues) {
if (keyValues.hasNext()) {
KeyValue<Number, Long> pair = keyValues.next();
- double result = pair.getKey().doubleValue() * pair.getValue();
long counter = pair.getValue();
+ Number result = mul(pair.getKey(), counter);
while (keyValues.hasNext()) {
+ long incr = pair.getValue();
pair = keyValues.next();
- result += pair.getKey().doubleValue() * pair.getValue();
- counter += pair.getValue();
+ result = add(result, mul(pair.getKey(), incr));
+ counter += incr;
}
- return result / counter;
+ return div(result, counter, true);
}
return Double.NaN;
}
- public static final MeanGlobalMapReduce instance() {
+ public static MeanGlobalMapReduce instance() {
return INSTANCE;
}
}
///
- public static final class MeanNumber extends Number implements Comparable<Number>, FinalGet<Double> {
+ public static final class MeanNumber extends Number implements Comparable<Number>, FinalGet<Number> {
private long count;
- private double sum;
+ private Number sum;
public MeanNumber() {
- this(0.0d, 0l);
+ this(0, 0);
}
- public MeanNumber(final double number, final long count) {
+ public MeanNumber(final Number number, final long count) {
this.count = count;
- this.sum = number * count;
+ this.sum = mul(number, count);
}
public MeanNumber add(final Number amount, final long count) {
this.count += count;
- this.sum += amount.doubleValue() * count;
+ this.sum = NumberHelper.add(sum, mul(amount, count));
return this;
}
public MeanNumber add(final MeanNumber other) {
this.count += other.count;
- this.sum += other.sum;
+ this.sum = NumberHelper.add(sum, other.sum);
return this;
}
@Override
public int intValue() {
- return (int) (this.sum / this.count);
+ return div(this.sum, this.count).intValue();
}
@Override
public long longValue() {
- return (long) (this.sum / this.count);
+ return div(this.sum, this.count).longValue();
}
@Override
public float floatValue() {
- return (float) (this.sum / this.count);
+ return div(this.sum, this.count, true).floatValue();
}
@Override
public double doubleValue() {
- return this.sum / this.count;
+ return div(this.sum, this.count, true).doubleValue();
}
@Override
public String toString() {
- return Double.toString(this.doubleValue());
+ return div(this.sum, this.count, true).toString();
}
@Override
public int compareTo(final Number number) {
+ // TODO: NumberHelper should provide a compareTo() implementation
return Double.valueOf(this.doubleValue()).compareTo(number.doubleValue());
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/eba71143/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MinGlobalStep.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MinGlobalStep.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MinGlobalStep.java
index 6a3ba10..750e36c 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MinGlobalStep.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MinGlobalStep.java
@@ -37,6 +37,8 @@ import java.util.Iterator;
import java.util.Set;
import java.util.function.BiFunction;
+import static org.apache.tinkerpop.gremlin.process.traversal.NumberHelper.min;
+
/**
* @author Marko A. Rodriguez (http://markorodriguez.com)
*/
@@ -70,10 +72,11 @@ public final class MinGlobalStep<S extends Number> extends ReducingBarrierStep<S
@Override
public S apply(final S mutatingSeed, final Traverser<S> traverser) {
- return mutatingSeed != null && mutatingSeed.doubleValue() <= traverser.get().doubleValue() ? mutatingSeed : traverser.get();
+ final S value = traverser.get();
+ return mutatingSeed != null ? (S) min(mutatingSeed, traverser.get()) : value;
}
- public final static <S extends Number> MinGlobalBiFunction<S> instance() {
+ public static <S extends Number> MinGlobalBiFunction<S> instance() {
return INSTANCE;
}
}
@@ -106,11 +109,10 @@ public final class MinGlobalStep<S extends Number> extends ReducingBarrierStep<S
@Override
public void reduce(final NullObject key, final Iterator<Number> values, final ReduceEmitter<NullObject, Number> emitter) {
if (values.hasNext()) {
- Number min = Double.MAX_VALUE;
+ Number min = null;
while (values.hasNext()) {
final Number value = values.next();
- if (value.doubleValue() < min.doubleValue())
- min = value;
+ min = min != null ? min(value, min) : value;
}
emitter.emit(min);
}
@@ -126,7 +128,7 @@ public final class MinGlobalStep<S extends Number> extends ReducingBarrierStep<S
return keyValues.hasNext() ? keyValues.next().getValue() : Double.NaN;
}
- public static final MinGlobalMapReduce instance() {
+ public static MinGlobalMapReduce instance() {
return INSTANCE;
}
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/eba71143/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MinLocalStep.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MinLocalStep.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MinLocalStep.java
index 2959285..116a5e1 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MinLocalStep.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MinLocalStep.java
@@ -26,6 +26,8 @@ import java.util.Collections;
import java.util.Iterator;
import java.util.Set;
+import static org.apache.tinkerpop.gremlin.process.traversal.NumberHelper.min;
+
/**
* @author Marko A. Rodriguez (http://markorodriguez.com)
* @author Daniel Kuppitz (http://gremlin.guru)
@@ -43,8 +45,7 @@ public final class MinLocalStep<E extends Number, S extends Iterable<E>> extends
if (iterator.hasNext()) {
result = iterator.next();
while (iterator.hasNext()) {
- final Number curr = iterator.next();
- if (result.doubleValue() > curr.doubleValue()) result = curr;
+ result = min(iterator.next(), result);
}
} else {
result = Double.NaN;
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/eba71143/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java
----------------------------------------------------------------------
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java
index efb4a5b..d5a7d1d 100644
--- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java
+++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraphPlayTest.java
@@ -192,12 +192,9 @@ public class TinkerGraphPlayTest {
@Test
@Ignore
public void testPlayDK() throws Exception {
- final Number n = __.inject(1, 2L, 3f).sum().next();
- System.out.println(n.toString());
- System.out.println(n.getClass().toString());
- final Double d = __.inject(1, 2L, 3f).<Double>sum().next();
- System.out.println(d.toString());
- System.out.println(d.getClass().toString());
+ TinkerGraph graph = TinkerFactory.createModern();
+ GraphTraversalSource g = graph.traversal(GraphTraversalSource.standard());
+ final Number m = g.V().values("age").mean().next();
}
@Test
[34/50] [abbrv] incubator-tinkerpop git commit: updated CHANGELOG
Posted by sp...@apache.org.
updated CHANGELOG
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/63c8cb87
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/63c8cb87
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/63c8cb87
Branch: refs/heads/TINKERPOP3-923
Commit: 63c8cb87572ab2b6b3b3e386c5326edca017421e
Parents: 2a4b4ac
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Tue Nov 3 17:52:31 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Tue Nov 3 17:52:31 2015 +0100
----------------------------------------------------------------------
CHANGELOG.asciidoc | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/63c8cb87/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index e79a0a0..ef0d032 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -25,6 +25,7 @@ image::https://raw.githubusercontent.com/apache/incubator-tinkerpop/master/docs/
TinkerPop 3.1.0 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Optimized `BulkLoaderVertexProgram`. It now uses `EventStrategy` to monitor what the underlying `BulkLoader` implementation does (e.g. whether it creates a new vertex or returns an existing).
* Bumped to Neo4j 2.3.0.
* Added `PersistedInputRDD` and `PersistedOutputRDD` which enables `SparkGraphComputer` to store the graph RDD in the context between jobs (no HDFS serialization required).
* Renamed the `public static String` configuration variable names of TinkerGraph (deprecated old variables).
[43/50] [abbrv] incubator-tinkerpop git commit: added a note for
BulkLoader implementers
Posted by sp...@apache.org.
added a note for BulkLoader implementers
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/77fac190
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/77fac190
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/77fac190
Branch: refs/heads/TINKERPOP3-923
Commit: 77fac190dfedd107a38086e201334298e0bd3829
Parents: f2b9083
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Wed Nov 4 20:07:27 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Wed Nov 4 20:07:27 2015 +0100
----------------------------------------------------------------------
docs/src/the-graphcomputer.asciidoc | 4 ++++
1 file changed, 4 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/77fac190/docs/src/the-graphcomputer.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/the-graphcomputer.asciidoc b/docs/src/the-graphcomputer.asciidoc
index 4fd0e8c..af65f07 100644
--- a/docs/src/the-graphcomputer.asciidoc
+++ b/docs/src/the-graphcomputer.asciidoc
@@ -368,6 +368,10 @@ will work for the most use-cases, but has one limitation though: It doesn't supp
`IncrementalBulkLoader` will handle every property as a single-valued property. A custom `BulkLoader` implementation
has to be used if the default behavior is not sufficient.
+NOTE: A custom `BulkLoader` implementation for incremental loading should use `GraphTraversal` methods to create/update
+elements (e.g. `g.addV()` instead of `graph.addVertex()`). This way the `BulkLoaderVertexProgram` is able to efficiently
+track changes in the underlying graph and can apply several optimization techniques.
+
[[traversalvertexprogram]]
TraversalVertexProgram
~~~~~~~~~~~~~~~~~~~~~~
[22/50] [abbrv] incubator-tinkerpop git commit: Merge branch 'master'
into TINKERPOP3-909
Posted by sp...@apache.org.
Merge branch 'master' into TINKERPOP3-909
Resolved Conflicts:
CHANGELOG.asciidoc
docs/src/upgrade-release-3.1.x-incubating.asciidoc
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/e29a9362
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/e29a9362
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/e29a9362
Branch: refs/heads/TINKERPOP3-923
Commit: e29a9362beb2cb2784a547b7f9e1e5d589c83d47
Parents: ccc3bfc 19b7ae0
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Sun Nov 1 22:15:04 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Sun Nov 1 22:15:04 2015 +0100
----------------------------------------------------------------------
CHANGELOG.asciidoc | 4 +-
docs/src/developer-meetings.asciidoc | 64 ++++++++++++++++++++
docs/src/developer-release.asciidoc | 13 ++--
docs/src/developer.asciidoc | 4 +-
docs/src/gremlin-applications.asciidoc | 7 +++
.../upgrade-release-3.1.x-incubating.asciidoc | 5 ++
.../traversal/step/map/AddVertexTest.java | 2 +-
neo4j-gremlin/pom.xml | 28 ++++++++-
.../gremlin/neo4j/structure/Neo4jGraph.java | 2 +-
.../gremlin/neo4j/structure/Neo4jHelper.java | 16 ++---
.../gremlin/neo4j/Neo4jStrategySuite.java | 44 --------------
.../gremlin/neo4j/Neo4jStrategyTest.java | 31 ----------
.../traversal/strategy/Neo4jStrategySuite.java | 44 ++++++++++++++
.../traversal/strategy/Neo4jStrategyTest.java | 32 ++++++++++
.../Neo4jGraphStepStrategyTest.java | 13 ++++
.../tinkergraph/structure/TinkerGraph.java | 2 +-
.../tinkergraph/TinkerGraphStrategySuite.java | 44 --------------
.../tinkergraph/TinkerGraphStrategyTest.java | 31 ----------
.../strategy/TinkerGraphStrategySuite.java | 44 ++++++++++++++
.../strategy/TinkerGraphStrategyTest.java | 32 ++++++++++
.../TinkerGraphStepStrategyTest.java | 13 ++++
21 files changed, 305 insertions(+), 170 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/e29a9362/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --cc CHANGELOG.asciidoc
index 6f7c50e,06d0db4..84cb9f5
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@@ -25,10 -25,11 +25,12 @@@ image::https://raw.githubusercontent.co
TinkerPop 3.1.0 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Integrated `NumberHelper` in `SumStep`, `MinStep`, `MaxStep` and `MeanStep` (local and global step variants).
+ * Bumped to Neo4j 2.3.0.
* Renamed the `public static String` configuration variable names of TinkerGraph (deprecated old variables).
* Added `GraphComputer.config(key,value)` to allow engine-specific configurations.
- * Added suppport for mid-traversal `V()`-steps (`GraphStep` semantics updated).
+ * `GraphStep` is no longer in the `sideEffect`-package and is now in `map`-package (breaking change).
+ * Added support for mid-traversal `V()`-steps (`GraphStep` semantics updated).
* Fixed `Number` handling in `Operator` enums. Prior this change a lot of operations on mixed `Number` types returned a wrong result (wrong data type).
* Fixed a bug in Gremlin Server/Driver serializer where empty buffers were getting returned in certain cases.
* Renamed `ConjunctionX` to `ConnectiveX` because "conjunction" is assumed "and" (disjunction "or"), where "connective" is the parent concept.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/e29a9362/docs/src/upgrade-release-3.1.x-incubating.asciidoc
----------------------------------------------------------------------
diff --cc docs/src/upgrade-release-3.1.x-incubating.asciidoc
index 7043d74,b50dd7e..00be151
--- a/docs/src/upgrade-release-3.1.x-incubating.asciidoc
+++ b/docs/src/upgrade-release-3.1.x-incubating.asciidoc
@@@ -35,7 -35,7 +35,8 @@@ Additional upgrade information can be f
Important Changes
~~~~~~~~~~~~~~~~~
+* `sum()`, `min()`, `max()` and `mean()` make now use of `NumberHelper`, leading to a more appropriate result type, instead of just `Double`.
+ * `GraphStep` is no longer in `sideEffect`-package, but now in `map`-package as traversals support mid-traversal `V()`.
* Traversals now support mid-traversal `V()`-steps. Graph system providers should ensure that a mid-traversal `V()` can leverage any suitable index.
* The data type of `Operator` enums will now always be the highest common data type of the two given numbers, rather than the data type of the first number, as it's been before.
* The Gephi Plugin has improved integration with Gephi, where manually inserting `store` steps to visualize a running traversal is no longer required.
[10/50] [abbrv] incubator-tinkerpop git commit: @RussellSpitzer has
schooled me in cache()/unpersist(). I now am smart to unpersist() RDDs that
are just dangling around (especailly since we now have persistent contexts).
Finally, I did some refactoring o
Posted by sp...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputOutputRDDTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputOutputRDDTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputOutputRDDTest.java
deleted file mode 100644
index b83314a..0000000
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputOutputRDDTest.java
+++ /dev/null
@@ -1,59 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.spark.process.computer.io;
-
-import org.apache.commons.configuration.BaseConfiguration;
-import org.apache.commons.configuration.Configuration;
-import org.apache.tinkerpop.gremlin.hadoop.Constants;
-import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
-import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
-import org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram;
-import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
-import org.apache.tinkerpop.gremlin.process.traversal.engine.ComputerTraversalEngine;
-import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;
-import org.apache.tinkerpop.gremlin.structure.Graph;
-import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
-import org.junit.Test;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public class InputOutputRDDTest {
-
- @Test
- public void shouldReadFromWriteToArbitraryRDD() throws Exception {
- final Configuration configuration = new BaseConfiguration();
- configuration.setProperty("spark.master", "local[4]");
- configuration.setProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
- configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
- configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, ExampleInputRDD.class.getCanonicalName());
- configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, ExampleOutputRDD.class.getCanonicalName());
- configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, "target/test-output");
- configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
- ////////
- Graph graph = GraphFactory.open(configuration);
- graph.compute(SparkGraphComputer.class)
- .result(GraphComputer.ResultGraph.NEW)
- .persist(GraphComputer.Persist.EDGES)
- .program(TraversalVertexProgram.build()
- .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
- "gremlin-groovy",
- "g.V()").create(graph)).submit().get();
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputRDDTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputRDDTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputRDDTest.java
deleted file mode 100644
index 3070fea..0000000
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputRDDTest.java
+++ /dev/null
@@ -1,54 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.spark.process.computer.io;
-
-import org.apache.commons.configuration.BaseConfiguration;
-import org.apache.commons.configuration.Configuration;
-import org.apache.tinkerpop.gremlin.hadoop.Constants;
-import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat;
-import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
-import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;
-import org.apache.tinkerpop.gremlin.structure.Graph;
-import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
-import org.junit.Test;
-
-import static org.junit.Assert.assertEquals;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public class InputRDDTest {
-
- @Test
- public void shouldReadFromArbitraryRDD() {
- final Configuration configuration = new BaseConfiguration();
- configuration.setProperty("spark.master", "local[4]");
- configuration.setProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
- configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
- configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, ExampleInputRDD.class.getCanonicalName());
- configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT, GryoOutputFormat.class.getCanonicalName());
- configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, "target/test-output");
- configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
- ////////
- Graph graph = GraphFactory.open(configuration);
- assertEquals(Double.valueOf(123.0d), graph.traversal(GraphTraversalSource.computer(SparkGraphComputer.class)).V().values("age").sum().next());
- assertEquals(Long.valueOf(4l), graph.traversal(GraphTraversalSource.computer(SparkGraphComputer.class)).V().count().next());
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputRDDTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputRDDTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputRDDTest.java
deleted file mode 100644
index febece6..0000000
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputRDDTest.java
+++ /dev/null
@@ -1,62 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.spark.process.computer.io;
-
-import org.apache.commons.configuration.BaseConfiguration;
-import org.apache.commons.configuration.Configuration;
-import org.apache.tinkerpop.gremlin.hadoop.Constants;
-import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat;
-import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
-import org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram;
-import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
-import org.apache.tinkerpop.gremlin.process.traversal.engine.ComputerTraversalEngine;
-import org.apache.tinkerpop.gremlin.spark.process.computer.SparkHadoopGraphProvider;
-import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;
-import org.apache.tinkerpop.gremlin.structure.Graph;
-import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
-import org.junit.Test;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public class OutputRDDTest {
-
- @Test
- public void shouldWriteToArbitraryRDD() throws Exception {
- final Configuration configuration = new BaseConfiguration();
- configuration.setProperty("spark.master", "local[4]");
- configuration.setProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
- configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
- configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
- configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
- configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, ExampleOutputRDD.class.getCanonicalName());
- configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, "target/test-output");
- configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
- ////////
- Graph graph = GraphFactory.open(configuration);
- graph.compute(SparkGraphComputer.class)
- .result(GraphComputer.ResultGraph.NEW)
- .persist(GraphComputer.Persist.EDGES)
- .program(TraversalVertexProgram.build()
- .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
- "gremlin-groovy",
- "g.V()").create(graph)).submit().get();
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputOutputRDDTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputOutputRDDTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputOutputRDDTest.java
deleted file mode 100644
index a7d1ac0..0000000
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputOutputRDDTest.java
+++ /dev/null
@@ -1,172 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.tinkerpop.gremlin.spark.process.computer.io;
-
-import org.apache.commons.configuration.BaseConfiguration;
-import org.apache.commons.configuration.Configuration;
-import org.apache.spark.SparkConf;
-import org.apache.spark.SparkContext;
-import org.apache.spark.api.java.JavaSparkContext;
-import org.apache.tinkerpop.gremlin.hadoop.Constants;
-import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat;
-import org.apache.tinkerpop.gremlin.hadoop.structure.util.ConfUtil;
-import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
-import org.apache.tinkerpop.gremlin.process.computer.bulkloading.BulkLoaderVertexProgram;
-import org.apache.tinkerpop.gremlin.process.computer.ranking.pagerank.PageRankVertexProgram;
-import org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram;
-import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
-import org.apache.tinkerpop.gremlin.process.traversal.engine.ComputerTraversalEngine;
-import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;
-import org.apache.tinkerpop.gremlin.spark.process.computer.SparkHadoopGraphProvider;
-import org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer;
-import org.apache.tinkerpop.gremlin.structure.Graph;
-import org.apache.tinkerpop.gremlin.structure.io.IoCore;
-import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
-import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
-import org.junit.Test;
-
-import java.util.UUID;
-
-import static org.junit.Assert.*;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public class PersistedInputOutputRDDTest {
-
- @Test
- public void shouldNotPersistRDDAcrossJobs() throws Exception {
- final String rddName = "target/test-output/" + UUID.randomUUID();
- final Configuration configuration = new BaseConfiguration();
- configuration.setProperty("spark.master", "local[4]");
- configuration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
- configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
- configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
- configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
- configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, PersistedOutputRDD.class.getCanonicalName());
- configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
- configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, rddName);
- configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, false); // because the spark context is NOT persisted, neither is the RDD
- Graph graph = GraphFactory.open(configuration);
- graph.compute(SparkGraphComputer.class)
- .result(GraphComputer.ResultGraph.NEW)
- .persist(GraphComputer.Persist.EDGES)
- .program(TraversalVertexProgram.build()
- .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
- "gremlin-groovy",
- "g.V()").create(graph)).submit().get();
- ////////
- SparkConf sparkConfiguration = new SparkConf();
- sparkConfiguration.setAppName("shouldNotPersistRDDAcrossJobs");
- ConfUtil.makeHadoopConfiguration(configuration).forEach(entry -> sparkConfiguration.set(entry.getKey(), entry.getValue()));
- JavaSparkContext sparkContext = new JavaSparkContext(SparkContext.getOrCreate(sparkConfiguration));
- assertFalse(PersistedInputRDD.getPersistedRDD(sparkContext, rddName).isPresent());
- }
-
- @Test
- public void shouldPersistRDDAcrossJobs() throws Exception {
- final String rddName = "target/test-output/" + UUID.randomUUID();
- final Configuration configuration = new BaseConfiguration();
- configuration.setProperty("spark.master", "local[4]");
- configuration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
- configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
- configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
- configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
- configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, PersistedOutputRDD.class.getCanonicalName());
- configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, rddName);
- configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
- configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, true);
- Graph graph = GraphFactory.open(configuration);
- graph.compute(SparkGraphComputer.class)
- .result(GraphComputer.ResultGraph.NEW)
- .persist(GraphComputer.Persist.EDGES)
- .program(TraversalVertexProgram.build()
- .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
- "gremlin-groovy",
- "g.V()").create(graph)).submit().get();
- ////////
- SparkConf sparkConfiguration = new SparkConf();
- sparkConfiguration.setAppName("shouldPersistRDDAcrossJobs");
- ConfUtil.makeHadoopConfiguration(configuration).forEach(entry -> sparkConfiguration.set(entry.getKey(), entry.getValue()));
- JavaSparkContext sparkContext = new JavaSparkContext(SparkContext.getOrCreate(sparkConfiguration));
- assertTrue(PersistedInputRDD.getPersistedRDD(sparkContext, rddName).isPresent());
- ///////
- configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, PersistedInputRDD.class.getCanonicalName());
- configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, rddName);
- configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, null);
- configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, null);
- graph = GraphFactory.open(configuration);
- graph.compute(SparkGraphComputer.class)
- .result(GraphComputer.ResultGraph.NEW)
- .persist(GraphComputer.Persist.NOTHING)
- .program(TraversalVertexProgram.build()
- .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
- "gremlin-groovy",
- "g.V()").create(graph)).submit().get();
- }
-
- @Test
- public void testBulkLoaderVertexProgramChain() throws Exception {
- final String rddName = "target/test-output/" + UUID.randomUUID().toString();
- final Configuration readConfiguration = new BaseConfiguration();
- readConfiguration.setProperty("spark.master", "local[4]");
- readConfiguration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
- readConfiguration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
- readConfiguration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
- readConfiguration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
- readConfiguration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, PersistedOutputRDD.class.getCanonicalName());
- readConfiguration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, rddName);
- readConfiguration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
- readConfiguration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, true);
- Graph pageRankGraph = GraphFactory.open(readConfiguration);
- ///////////////
- final Configuration writeConfiguration = new BaseConfiguration();
- writeConfiguration.setProperty(Graph.GRAPH, TinkerGraph.class.getCanonicalName());
- writeConfiguration.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo");
- writeConfiguration.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, "target/test-output/tinkergraph.kryo");
- final Graph bulkLoaderGraph = pageRankGraph.compute(SparkGraphComputer.class).persist(GraphComputer.Persist.VERTEX_PROPERTIES).program(PageRankVertexProgram.build().create(pageRankGraph)).submit().get().graph();
- bulkLoaderGraph.compute(SparkGraphComputer.class)
- .persist(GraphComputer.Persist.NOTHING)
- .workers(1)
- .configure(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, PersistedInputRDD.class.getCanonicalName())
- .configure(Constants.GREMLIN_HADOOP_INPUT_LOCATION, rddName)
- .configure(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, null)
- .configure(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, null)
- .program(BulkLoaderVertexProgram.build().userSuppliedIds(true).writeGraph(writeConfiguration).create(bulkLoaderGraph))
- .submit().get();
- ////
- SparkConf sparkConfiguration = new SparkConf();
- sparkConfiguration.setAppName("testBulkLoaderVertexProgramChain");
- JavaSparkContext sparkContext = new JavaSparkContext(SparkContext.getOrCreate(sparkConfiguration));
- assertFalse(PersistedInputRDD.getPersistedRDD(sparkContext, rddName).isPresent());
- ////
- final Graph graph = TinkerGraph.open();
- final GraphTraversalSource g = graph.traversal();
- graph.io(IoCore.gryo()).readGraph("target/test-output/tinkergraph.kryo");
- assertEquals(6l, g.V().count().next().longValue());
- assertEquals(6l, g.E().count().next().longValue());
- assertEquals("marko", g.V().has("name", "marko").values("name").next());
- assertEquals(6l, g.V().values(PageRankVertexProgram.PAGE_RANK).count().next().longValue());
- assertEquals(0l, g.E().values(PageRankVertexProgram.PAGE_RANK).count().next().longValue());
- assertEquals(6l, g.V().values(PageRankVertexProgram.EDGE_COUNT).count().next().longValue());
- assertEquals(0l, g.E().values(PageRankVertexProgram.EDGE_COUNT).count().next().longValue());
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleInputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleInputRDD.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleInputRDD.java
new file mode 100644
index 0000000..8417d1c
--- /dev/null
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleInputRDD.java
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.spark.structure.io;
+
+import org.apache.commons.configuration.Configuration;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+import org.apache.tinkerpop.gremlin.spark.structure.io.InputRDD;
+import org.apache.tinkerpop.gremlin.structure.T;
+import org.apache.tinkerpop.gremlin.structure.Vertex;
+import org.apache.tinkerpop.gremlin.structure.util.star.StarGraph;
+import scala.Tuple2;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public final class ExampleInputRDD implements InputRDD {
+
+ @Override
+ public JavaPairRDD<Object, VertexWritable> readGraphRDD(final Configuration configuration, final JavaSparkContext sparkContext) {
+ final List<Vertex> list = new ArrayList<>();
+ list.add(StarGraph.open().addVertex(T.id, 1l, T.label,"person","age", 29));
+ list.add(StarGraph.open().addVertex(T.id, 2l, T.label,"person","age", 27));
+ list.add(StarGraph.open().addVertex(T.id, 4l, T.label,"person","age", 32));
+ list.add(StarGraph.open().addVertex(T.id, 6l, T.label,"person","age", 35));
+ return sparkContext.parallelize(list).mapToPair(vertex -> new Tuple2<>(vertex.id(), new VertexWritable(vertex)));
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleOutputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleOutputRDD.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleOutputRDD.java
new file mode 100644
index 0000000..9a9e5ef
--- /dev/null
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/ExampleOutputRDD.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.spark.structure.io;
+
+import org.apache.commons.configuration.Configuration;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+import org.apache.tinkerpop.gremlin.spark.structure.io.OutputRDD;
+import org.apache.tinkerpop.gremlin.structure.Vertex;
+
+import java.util.Iterator;
+
+import static org.junit.Assert.assertEquals;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public final class ExampleOutputRDD implements OutputRDD {
+ @Override
+ public void writeGraphRDD(final Configuration configuration, final JavaPairRDD<Object, VertexWritable> graphRDD) {
+ int totalAge = 0;
+ final Iterator<VertexWritable> iterator = graphRDD.values().toLocalIterator();
+ while (iterator.hasNext()) {
+ final Vertex vertex = iterator.next().get();
+ if (vertex.label().equals("person"))
+ totalAge = totalAge + vertex.<Integer>value("age");
+ }
+ assertEquals(123, totalAge);
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputRDDTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputRDDTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputRDDTest.java
new file mode 100644
index 0000000..1feeb61
--- /dev/null
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputRDDTest.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.spark.structure.io;
+
+import org.apache.commons.configuration.BaseConfiguration;
+import org.apache.commons.configuration.Configuration;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
+import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
+import org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram;
+import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
+import org.apache.tinkerpop.gremlin.process.traversal.engine.ComputerTraversalEngine;
+import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;
+import org.apache.tinkerpop.gremlin.structure.Graph;
+import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
+import org.junit.Test;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public class InputOutputRDDTest {
+
+ @Test
+ public void shouldReadFromWriteToArbitraryRDD() throws Exception {
+ final Configuration configuration = new BaseConfiguration();
+ configuration.setProperty("spark.master", "local[4]");
+ configuration.setProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
+ configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, ExampleInputRDD.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, ExampleOutputRDD.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, "target/test-output");
+ configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
+ ////////
+ Graph graph = GraphFactory.open(configuration);
+ graph.compute(SparkGraphComputer.class)
+ .result(GraphComputer.ResultGraph.NEW)
+ .persist(GraphComputer.Persist.EDGES)
+ .program(TraversalVertexProgram.build()
+ .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
+ "gremlin-groovy",
+ "g.V()").create(graph)).submit().get();
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDDTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDDTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDDTest.java
new file mode 100644
index 0000000..8d6c06e
--- /dev/null
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDDTest.java
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.spark.structure.io;
+
+import org.apache.commons.configuration.BaseConfiguration;
+import org.apache.commons.configuration.Configuration;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat;
+import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
+import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;
+import org.apache.tinkerpop.gremlin.structure.Graph;
+import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public class InputRDDTest {
+
+ @Test
+ public void shouldReadFromArbitraryRDD() {
+ final Configuration configuration = new BaseConfiguration();
+ configuration.setProperty("spark.master", "local[4]");
+ configuration.setProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
+ configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, ExampleInputRDD.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT, GryoOutputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, "target/test-output");
+ configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
+ ////////
+ Graph graph = GraphFactory.open(configuration);
+ assertEquals(Double.valueOf(123.0d), graph.traversal(GraphTraversalSource.computer(SparkGraphComputer.class)).V().values("age").sum().next());
+ assertEquals(Long.valueOf(4l), graph.traversal(GraphTraversalSource.computer(SparkGraphComputer.class)).V().count().next());
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputRDDTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputRDDTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputRDDTest.java
new file mode 100644
index 0000000..43dcdbd
--- /dev/null
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputRDDTest.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.spark.structure.io;
+
+import org.apache.commons.configuration.BaseConfiguration;
+import org.apache.commons.configuration.Configuration;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat;
+import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
+import org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram;
+import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
+import org.apache.tinkerpop.gremlin.process.traversal.engine.ComputerTraversalEngine;
+import org.apache.tinkerpop.gremlin.spark.process.computer.SparkHadoopGraphProvider;
+import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;
+import org.apache.tinkerpop.gremlin.structure.Graph;
+import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
+import org.junit.Test;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public class OutputRDDTest {
+
+ @Test
+ public void shouldWriteToArbitraryRDD() throws Exception {
+ final Configuration configuration = new BaseConfiguration();
+ configuration.setProperty("spark.master", "local[4]");
+ configuration.setProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
+ configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, ExampleOutputRDD.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, "target/test-output");
+ configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
+ ////////
+ Graph graph = GraphFactory.open(configuration);
+ graph.compute(SparkGraphComputer.class)
+ .result(GraphComputer.ResultGraph.NEW)
+ .persist(GraphComputer.Persist.EDGES)
+ .program(TraversalVertexProgram.build()
+ .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
+ "gremlin-groovy",
+ "g.V()").create(graph)).submit().get();
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedInputOutputRDDTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedInputOutputRDDTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedInputOutputRDDTest.java
new file mode 100644
index 0000000..6ee7aaa
--- /dev/null
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedInputOutputRDDTest.java
@@ -0,0 +1,217 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tinkerpop.gremlin.spark.structure.io;
+
+import org.apache.commons.configuration.BaseConfiguration;
+import org.apache.commons.configuration.Configuration;
+import org.apache.spark.SparkConf;
+import org.apache.spark.SparkContext;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat;
+import org.apache.tinkerpop.gremlin.hadoop.structure.util.ConfUtil;
+import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
+import org.apache.tinkerpop.gremlin.process.computer.bulkloading.BulkLoaderVertexProgram;
+import org.apache.tinkerpop.gremlin.process.computer.ranking.pagerank.PageRankVertexProgram;
+import org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram;
+import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
+import org.apache.tinkerpop.gremlin.process.traversal.engine.ComputerTraversalEngine;
+import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;
+import org.apache.tinkerpop.gremlin.spark.process.computer.SparkHadoopGraphProvider;
+import org.apache.tinkerpop.gremlin.spark.structure.io.PersistedInputRDD;
+import org.apache.tinkerpop.gremlin.spark.structure.io.PersistedOutputRDD;
+import org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer;
+import org.apache.tinkerpop.gremlin.structure.Graph;
+import org.apache.tinkerpop.gremlin.structure.io.IoCore;
+import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
+import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
+import org.junit.Test;
+
+import java.util.UUID;
+
+import static org.junit.Assert.*;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public class PersistedInputOutputRDDTest {
+
+ @Test
+ public void shouldNotPersistRDDAcrossJobs() throws Exception {
+ final String rddName = "target/test-output/" + UUID.randomUUID();
+ final Configuration configuration = new BaseConfiguration();
+ configuration.setProperty("spark.master", "local[4]");
+ configuration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
+ configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, PersistedOutputRDD.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
+ configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, rddName);
+ configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, false); // because the spark context is NOT persisted, neither is the RDD
+ Graph graph = GraphFactory.open(configuration);
+ graph.compute(SparkGraphComputer.class)
+ .result(GraphComputer.ResultGraph.NEW)
+ .persist(GraphComputer.Persist.EDGES)
+ .program(TraversalVertexProgram.build()
+ .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
+ "gremlin-groovy",
+ "g.V()").create(graph)).submit().get();
+ ////////
+ SparkConf sparkConfiguration = new SparkConf();
+ sparkConfiguration.setAppName("shouldNotPersistRDDAcrossJobs");
+ ConfUtil.makeHadoopConfiguration(configuration).forEach(entry -> sparkConfiguration.set(entry.getKey(), entry.getValue()));
+ JavaSparkContext sparkContext = new JavaSparkContext(SparkContext.getOrCreate(sparkConfiguration));
+ assertFalse(PersistedInputRDD.getPersistedRDD(sparkContext, rddName).isPresent());
+ }
+
+ @Test
+ public void shouldPersistRDDAcrossJobs() throws Exception {
+ final String rddName = "target/test-output/" + UUID.randomUUID();
+ final Configuration configuration = new BaseConfiguration();
+ configuration.setProperty("spark.master", "local[4]");
+ configuration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
+ configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, PersistedOutputRDD.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, rddName);
+ configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
+ configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, true);
+ Graph graph = GraphFactory.open(configuration);
+ graph.compute(SparkGraphComputer.class)
+ .result(GraphComputer.ResultGraph.NEW)
+ .persist(GraphComputer.Persist.EDGES)
+ .program(TraversalVertexProgram.build()
+ .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
+ "gremlin-groovy",
+ "g.V()").create(graph)).submit().get();
+ ////////
+ SparkConf sparkConfiguration = new SparkConf();
+ sparkConfiguration.setAppName("shouldPersistRDDAcrossJobs");
+ ConfUtil.makeHadoopConfiguration(configuration).forEach(entry -> sparkConfiguration.set(entry.getKey(), entry.getValue()));
+ JavaSparkContext sparkContext = new JavaSparkContext(SparkContext.getOrCreate(sparkConfiguration));
+ assertTrue(PersistedInputRDD.getPersistedRDD(sparkContext, rddName).isPresent());
+ ///////
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, PersistedInputRDD.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, rddName);
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, null);
+ configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, null);
+ graph = GraphFactory.open(configuration);
+ graph.compute(SparkGraphComputer.class)
+ .result(GraphComputer.ResultGraph.NEW)
+ .persist(GraphComputer.Persist.NOTHING)
+ .program(TraversalVertexProgram.build()
+ .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
+ "gremlin-groovy",
+ "g.V()").create(graph)).submit().get();
+ }
+
+ @Test
+ public void testBulkLoaderVertexProgramChain() throws Exception {
+ final String rddName = "target/test-output/" + UUID.randomUUID().toString();
+ final Configuration readConfiguration = new BaseConfiguration();
+ readConfiguration.setProperty("spark.master", "local[4]");
+ readConfiguration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
+ readConfiguration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
+ readConfiguration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, PersistedOutputRDD.class.getCanonicalName());
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, rddName);
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
+ readConfiguration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, true);
+ Graph pageRankGraph = GraphFactory.open(readConfiguration);
+ ///////////////
+ final Configuration writeConfiguration = new BaseConfiguration();
+ writeConfiguration.setProperty(Graph.GRAPH, TinkerGraph.class.getCanonicalName());
+ writeConfiguration.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo");
+ writeConfiguration.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, "target/test-output/tinkergraph.kryo");
+ final Graph bulkLoaderGraph = pageRankGraph.compute(SparkGraphComputer.class).persist(GraphComputer.Persist.VERTEX_PROPERTIES).program(PageRankVertexProgram.build().create(pageRankGraph)).submit().get().graph();
+ bulkLoaderGraph.compute(SparkGraphComputer.class)
+ .persist(GraphComputer.Persist.NOTHING)
+ .workers(1)
+ .configure(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, PersistedInputRDD.class.getCanonicalName())
+ .configure(Constants.GREMLIN_HADOOP_INPUT_LOCATION, rddName)
+ .configure(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, null)
+ .configure(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, null)
+ .program(BulkLoaderVertexProgram.build().userSuppliedIds(true).writeGraph(writeConfiguration).create(bulkLoaderGraph))
+ .submit().get();
+ ////
+ SparkConf sparkConfiguration = new SparkConf();
+ sparkConfiguration.setAppName("testBulkLoaderVertexProgramChain");
+ JavaSparkContext sparkContext = new JavaSparkContext(SparkContext.getOrCreate(sparkConfiguration));
+ assertFalse(PersistedInputRDD.getPersistedRDD(sparkContext, rddName).isPresent());
+ ////
+ final Graph graph = TinkerGraph.open();
+ final GraphTraversalSource g = graph.traversal();
+ graph.io(IoCore.gryo()).readGraph("target/test-output/tinkergraph.kryo");
+ assertEquals(6l, g.V().count().next().longValue());
+ assertEquals(6l, g.E().count().next().longValue());
+ assertEquals("marko", g.V().has("name", "marko").values("name").next());
+ assertEquals(6l, g.V().values(PageRankVertexProgram.PAGE_RANK).count().next().longValue());
+ assertEquals(0l, g.E().values(PageRankVertexProgram.PAGE_RANK).count().next().longValue());
+ assertEquals(6l, g.V().values(PageRankVertexProgram.EDGE_COUNT).count().next().longValue());
+ assertEquals(0l, g.E().values(PageRankVertexProgram.EDGE_COUNT).count().next().longValue());
+ }
+
+ @Test
+ public void testBulkLoaderVertexProgramChainWithInputOutputHelperMapping() throws Exception {
+ final String rddName = "target/test-output/" + UUID.randomUUID().toString();
+ final Configuration readConfiguration = new BaseConfiguration();
+ readConfiguration.setProperty("spark.master", "local[4]");
+ readConfiguration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
+ readConfiguration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
+ readConfiguration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, PersistedOutputRDD.class.getCanonicalName());
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, rddName);
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
+ readConfiguration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, true);
+ Graph pageRankGraph = GraphFactory.open(readConfiguration);
+ ///////////////
+ final Configuration writeConfiguration = new BaseConfiguration();
+ writeConfiguration.setProperty(Graph.GRAPH, TinkerGraph.class.getCanonicalName());
+ writeConfiguration.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo");
+ writeConfiguration.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, "target/test-output/tinkergraph.kryo");
+ final Graph bulkLoaderGraph = pageRankGraph.compute(SparkGraphComputer.class).persist(GraphComputer.Persist.VERTEX_PROPERTIES).program(PageRankVertexProgram.build().create(pageRankGraph)).submit().get().graph();
+ bulkLoaderGraph.compute(SparkGraphComputer.class)
+ .persist(GraphComputer.Persist.NOTHING)
+ .workers(1)
+ .program(BulkLoaderVertexProgram.build().userSuppliedIds(true).writeGraph(writeConfiguration).create(bulkLoaderGraph))
+ .submit().get();
+ ////
+ SparkConf sparkConfiguration = new SparkConf();
+ sparkConfiguration.setAppName("testBulkLoaderVertexProgramChain");
+ JavaSparkContext sparkContext = new JavaSparkContext(SparkContext.getOrCreate(sparkConfiguration));
+ assertFalse(PersistedInputRDD.getPersistedRDD(sparkContext, rddName).isPresent());
+ ////
+ final Graph graph = TinkerGraph.open();
+ final GraphTraversalSource g = graph.traversal();
+ graph.io(IoCore.gryo()).readGraph("target/test-output/tinkergraph.kryo");
+ assertEquals(6l, g.V().count().next().longValue());
+ assertEquals(6l, g.E().count().next().longValue());
+ assertEquals("marko", g.V().has("name", "marko").values("name").next());
+ assertEquals(6l, g.V().values(PageRankVertexProgram.PAGE_RANK).count().next().longValue());
+ assertEquals(0l, g.E().values(PageRankVertexProgram.PAGE_RANK).count().next().longValue());
+ assertEquals(6l, g.V().values(PageRankVertexProgram.EDGE_COUNT).count().next().longValue());
+ assertEquals(0l, g.E().values(PageRankVertexProgram.EDGE_COUNT).count().next().longValue());
+ }
+}
[45/50] [abbrv] incubator-tinkerpop git commit: Added a bit more to
the "getting started" contributor docs.
Posted by sp...@apache.org.
Added a bit more to the "getting started" contributor docs.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/88abff69
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/88abff69
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/88abff69
Branch: refs/heads/TINKERPOP3-923
Commit: 88abff69f7cbecc04fa4db5ca06f52e942bd109b
Parents: bfb99f3
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Wed Nov 4 14:36:02 2015 -0500
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Wed Nov 4 14:36:02 2015 -0500
----------------------------------------------------------------------
docs/src/developer-contributing.asciidoc | 16 ++++++++++++++++
1 file changed, 16 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/88abff69/docs/src/developer-contributing.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/developer-contributing.asciidoc b/docs/src/developer-contributing.asciidoc
index acf5dc8..fceedb2 100644
--- a/docs/src/developer-contributing.asciidoc
+++ b/docs/src/developer-contributing.asciidoc
@@ -38,6 +38,21 @@ After making changes, submit a link:https://help.github.com/articles/using-pull-
GitHub, where the name of the pull request is prefixed with the JIRA issue number. In this way, the pull request
and its comments get tied back to the JIRA issue it references.
+Before issuing your pull request, please be sure of the following:
+
+. `mvn clean install` works successfully.
+. If the change requires modification to the documentation, which can be found in `docs/src`, please be sure to try to
+generate the docs. If the changes are minimal and do not include code examples, it might be sufficient to test
+generate the docs to validate formatting by just doing `bin/process-docs.sh --dryRun`. If there are code examples,
+please be sure to have Zookeeper and Hadoop running when doing a `bin/process-docs.sh`. The documentation is
+generated to `/target/docs/htmlsingle`.
+. If necessary, run the integration tests. For example, if the changes affect serialization or Gremlin Server/Driver
+operations then running the integration tests assures in addition to unit tests will definitely be necessary. After
+a successful `mvn clean install`, do `mvn verify -DskipIntegrationTests=false -pl gremlin-server`.
+
+Once a pull request is submitted it must go through <<rtc,review>> and will be merged once three TinkerPop committers
+offer positive vote and achieve Apache consensus.
+
Building and Testing
--------------------
@@ -232,6 +247,7 @@ When writing a test case for a Gremlin step, be sure to use the following conven
** `checkResults(Arrays.asList("marko","josh"), traversal)`
** `checkMap(new HashMap<String,Long>() {{ put("marko",1l); }}, traversal.next())`
+[rtc]
Review then Commit
~~~~~~~~~~~~~~~~~~
[09/50] [abbrv] incubator-tinkerpop git commit: forgot to remove this
when figuring out how unpersist() works. Sorry for the slop.
Posted by sp...@apache.org.
forgot to remove this when figuring out how unpersist() works. Sorry for the slop.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/eeb0d9fa
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/eeb0d9fa
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/eeb0d9fa
Branch: refs/heads/TINKERPOP3-923
Commit: eeb0d9fa7c975f0d7d3d820bba21f75211f3a129
Parents: 1f6c574
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Fri Oct 30 12:37:39 2015 -0600
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Fri Oct 30 12:37:39 2015 -0600
----------------------------------------------------------------------
.../gremlin/spark/process/computer/io/OutputFormatRDD.java | 1 -
1 file changed, 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/eeb0d9fa/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputFormatRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputFormatRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputFormatRDD.java
index 661dc51..56a1297 100644
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputFormatRDD.java
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputFormatRDD.java
@@ -45,6 +45,5 @@ public final class OutputFormatRDD implements OutputRDD {
VertexWritable.class,
(Class<OutputFormat<NullWritable, VertexWritable>>) hadoopConfiguration.getClass(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT, OutputFormat.class), hadoopConfiguration);
}
- graphRDD.unpersist();
}
}
\ No newline at end of file
[02/50] [abbrv] incubator-tinkerpop git commit: TINKERPOP3-933
Distributions named properly for initial vote.
Posted by sp...@apache.org.
TINKERPOP3-933 Distributions named properly for initial vote.
Invoking CTR for this small change.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/522c256a
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/522c256a
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/522c256a
Branch: refs/heads/TINKERPOP3-923
Commit: 522c256a17b0f7b1650f20fde9fe57b6568e25da
Parents: 553f91f
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Thu Oct 29 16:40:25 2015 -0400
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Thu Oct 29 16:40:25 2015 -0400
----------------------------------------------------------------------
docs/src/developer-release.asciidoc | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/522c256a/docs/src/developer-release.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/developer-release.asciidoc b/docs/src/developer-release.asciidoc
index ef7349c..ade3fa8 100644
--- a/docs/src/developer-release.asciidoc
+++ b/docs/src/developer-release.asciidoc
@@ -102,7 +102,8 @@ A positive vote for a particular release from the TinkerPop PMC is required to m
.. `cp ~/.m2/repository/org/apache/tinkerpop/gremlin-console/x.y.z/gremlin-console-x.y.z-distribution.zip* dev/x.y.z`
.. `cp ~/.m2/repository/org/apache/tinkerpop/gremlin-server/x.y.z/gremlin-server-x.y.z-distribution.zip* dev/x.y.z`
.. `cp ~/.m2/repository/org/apache/tinkerpop/tinkerpop/x.y.z/tinkerpop-x.y.z-source-release.zip* dev/x.y.z`
-.. `cd dev/x.y.z` and `for f in \*.zip*; do mv "$f" "apache-$f"; done`
+.. `cd dev/x.y.z`
+.. pass:[<code>ls * | xargs -n1 -I {} echo "mv apache-{} {}" | sed -e 's/distribution/bin/' -e 's/source-release/src/' -e s'/^\(.*\) \(.*\) \(.*\)$/\1 \3 \2/' | /bin/bash</code>]
.. `cd ..; svn add x.y.z/; svn ci -m "TinkerPop x.y.z release"`
. Execute `bin/validate-distribution.sh` and any other relevant testing.
. Perform JIRA administration tasks:
@@ -127,7 +128,6 @@ Release & Promote
. Close the staging repository at link:https://repository.apache.org/[Apache Nexus]) and then release.
. `svn co --depth empty https://dist.apache.org/repos/dist/dev/incubator/tinkerpop dev; svn up dev/x.y.z`
. `svn co --depth empty https://dist.apache.org/repos/dist/release/incubator/tinkerpop release; mkdir release/x.y.z`
-. `ls dev/x.y.z/ | grep '\-\(distribution\|source\-release\)\.zip' | sed -e 's/\(^[^ ]\*\)-distribution\([^ ]*\)/cp dev\/x.y.z\/\0 release\/x.y.z\/\1-bin\2/' -e 's/\(^[^ ]\*\)-source-release\([^ ]*\)/cp dev\/x.y.z\/\0 release\/x.y.z\/\1-src\2/' | /bin/sh`
. `cd release; svn add x.y.z/; svn ci -m "TinkerPop x.y.z release"`
. Update homepage with references to latest distribution and to other internal links elsewhere on the page.
. Wait for Apache Central to sync the jars and src (link:http://repo1.maven.org/maven2/org/apache/tinkerpop/tinkerpop/[http://repo1.maven.org/maven2/org/apache/tinkerpop/tinkerpop/]).
[50/50] [abbrv] incubator-tinkerpop git commit: Add the start of the
10 minute tutorial.
Posted by sp...@apache.org.
Add the start of the 10 minute tutorial.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/7e9aef4f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/7e9aef4f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/7e9aef4f
Branch: refs/heads/TINKERPOP3-923
Commit: 7e9aef4f5047a229f2c2c565afd4d537d82616da
Parents: 2b7dc77
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Wed Nov 4 20:55:40 2015 -0500
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Wed Nov 4 20:55:40 2015 -0500
----------------------------------------------------------------------
docs/src/tutorials-getting-started.asciidoc | 32 +++++++++++++++++++++++
docs/static/images/modern-edge-1-to-3-1.png | Bin 0 -> 3210 bytes
docs/static/images/modern-edge-1-to-3-2.png | Bin 0 -> 4420 bytes
docs/static/images/modern-edge-1-to-3-3.png | Bin 0 -> 7372 bytes
4 files changed, 32 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/7e9aef4f/docs/src/tutorials-getting-started.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/tutorials-getting-started.asciidoc b/docs/src/tutorials-getting-started.asciidoc
index 95de047..6cd99a5 100644
--- a/docs/src/tutorials-getting-started.asciidoc
+++ b/docs/src/tutorials-getting-started.asciidoc
@@ -130,3 +130,35 @@ In this first five minutes with Gremlin, you've gotten the Gremlin Console insta
some traversals and hopefully learned something about TinkerPop in general. You've only scratched the surface of
what there is to know, but those accomplishments will help enable understanding of the more detailed tutorials to
come.
+
+In Ten More Minutes
+-------------------
+
+In the first five minutes of getting started with TinkerPop, you learned some basics for telling Gremlin how to
+traverse a graph. Of course, there wasn't much discussion about what a graph is. A graph is a collection of
+vertices (i.e. nodes, dots, etc.) and edges (i.e. relationships, lines, etc.), where a vertex is an entity which
+represents some domain object (e.g. a person, a place, etc.) and an edge represents the relationship between two
+vertices.
+
+image:modern-edge-1-to-3-1.png[width=300,align=center]
+
+The above example shows a graph with two vertices, one with a unique identifier of "1" and another with a unique
+identifier of "3". There is a edge connecting the two with a unique identifier of "9". It is important to consider
+that the edge has a direction which goes out from vertex "1" and in to vertex "3'.
+
+IMPORTANT: Most TinkerPop implementations do not allow for identifier assignment. They will rather assign identifiers
+and ignore assigned identifiers you attempt to assign to them.
+
+A graph with elements that just have identifiers does not make for much of a database. To give some meaning to
+this basic structure, vertices and edges can each be given labels to categorize them.
+
+image:modern-edge-1-to-3-2.png[width=300,align=center]
+
+You can now see that a vertex "1" is a "person" and vertex "3" is a "software" vertex. They are joined by a "created"
+edge which allows you to see that a "person created software". The "label" and the "id" are reserved attributes of
+vertices and edges, but you can add your own arbitrary properties as well:
+
+image:modern-edge-1-to-3-3.png[width=300,align=center]
+
+This model is referred to as a property graph and it provides a flexible and intuitive way in which to model your data.
+
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/7e9aef4f/docs/static/images/modern-edge-1-to-3-1.png
----------------------------------------------------------------------
diff --git a/docs/static/images/modern-edge-1-to-3-1.png b/docs/static/images/modern-edge-1-to-3-1.png
new file mode 100755
index 0000000..d60f7b6
Binary files /dev/null and b/docs/static/images/modern-edge-1-to-3-1.png differ
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/7e9aef4f/docs/static/images/modern-edge-1-to-3-2.png
----------------------------------------------------------------------
diff --git a/docs/static/images/modern-edge-1-to-3-2.png b/docs/static/images/modern-edge-1-to-3-2.png
new file mode 100755
index 0000000..afe4269
Binary files /dev/null and b/docs/static/images/modern-edge-1-to-3-2.png differ
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/7e9aef4f/docs/static/images/modern-edge-1-to-3-3.png
----------------------------------------------------------------------
diff --git a/docs/static/images/modern-edge-1-to-3-3.png b/docs/static/images/modern-edge-1-to-3-3.png
new file mode 100755
index 0000000..154ca6b
Binary files /dev/null and b/docs/static/images/modern-edge-1-to-3-3.png differ
[16/50] [abbrv] incubator-tinkerpop git commit: bumped to support
Neo4j 2.3.0
Posted by sp...@apache.org.
bumped to support Neo4j 2.3.0
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/4c6ba90f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/4c6ba90f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/4c6ba90f
Branch: refs/heads/TINKERPOP3-923
Commit: 4c6ba90fc34d64379af8e0f9e0574167e57f78ab
Parents: 9300485
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Sat Oct 31 10:54:52 2015 -0600
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Sat Oct 31 10:54:52 2015 -0600
----------------------------------------------------------------------
CHANGELOG.asciidoc | 3 ++-
neo4j-gremlin/pom.xml | 28 +++++++++++++++++++++++++++-
2 files changed, 29 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/4c6ba90f/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index 8bd6e51..06d0db4 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -25,10 +25,11 @@ image::https://raw.githubusercontent.com/apache/incubator-tinkerpop/master/docs/
TinkerPop 3.1.0 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Bumped to Neo4j 2.3.0.
* Renamed the `public static String` configuration variable names of TinkerGraph (deprecated old variables).
* Added `GraphComputer.config(key,value)` to allow engine-specific configurations.
* `GraphStep` is no longer in the `sideEffect`-package and is now in `map`-package (breaking change).
-* Added suppport for mid-traversal `V()`-steps (`GraphStep` semantics updated).
+* Added support for mid-traversal `V()`-steps (`GraphStep` semantics updated).
* Fixed `Number` handling in `Operator` enums. Prior this change a lot of operations on mixed `Number` types returned a wrong result (wrong data type).
* Fixed a bug in Gremlin Server/Driver serializer where empty buffers were getting returned in certain cases.
* Renamed `ConjunctionX` to `ConnectiveX` because "conjunction" is assumed "and" (disjunction "or"), where "connective" is the parent concept.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/4c6ba90f/neo4j-gremlin/pom.xml
----------------------------------------------------------------------
diff --git a/neo4j-gremlin/pom.xml b/neo4j-gremlin/pom.xml
index 8661064..eca07d1 100644
--- a/neo4j-gremlin/pom.xml
+++ b/neo4j-gremlin/pom.xml
@@ -136,7 +136,33 @@ limitations under the License.
<dependency>
<groupId>org.neo4j</groupId>
<artifactId>neo4j-tinkerpop-api-impl</artifactId>
- <version>0.1-2.2</version>
+ <version>0.3-2.3.0</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-lang3</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.googlecode.concurrentlinkedhashmap</groupId>
+ <artifactId>concurrentlinkedhashmap-lru</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.scala-lang</groupId>
+ <artifactId>scala-library</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.scala-lang</groupId>
+ <artifactId>scala-library</artifactId>
+ <version>2.11.6</version>
+ <scope>test</scope>
+ </dependency>
+ <dependency>
+ <groupId>com.googlecode.concurrentlinkedhashmap</groupId>
+ <artifactId>concurrentlinkedhashmap-lru</artifactId>
+ <version>1.4.2</version>
<scope>test</scope>
</dependency>
<!-- *** WARNING *** -->
[32/50] [abbrv] incubator-tinkerpop git commit: Merge branch 'master'
into TINKERPOP3-909
Posted by sp...@apache.org.
Merge branch 'master' into TINKERPOP3-909
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/558c04e7
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/558c04e7
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/558c04e7
Branch: refs/heads/TINKERPOP3-923
Commit: 558c04e7f3d51b12ddd06cd11fe4bbea5298843d
Parents: af4f5dc 06d747f
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Tue Nov 3 17:47:37 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Tue Nov 3 17:47:37 2015 +0100
----------------------------------------------------------------------
CHANGELOG.asciidoc | 3 +-
docs/src/implementations.asciidoc | 36 +--
docs/src/the-traversal.asciidoc | 11 +-
.../upgrade-release-3.1.x-incubating.asciidoc | 18 +-
.../process/computer/GiraphGraphComputer.java | 11 +-
.../computer/io/GiraphVertexInputFormat.java | 70 ------
.../computer/io/GiraphVertexOutputFormat.java | 65 ------
.../process/computer/io/GiraphVertexReader.java | 67 ------
.../process/computer/io/GiraphVertexWriter.java | 57 -----
.../structure/io/GiraphVertexInputFormat.java | 70 ++++++
.../structure/io/GiraphVertexOutputFormat.java | 65 ++++++
.../giraph/structure/io/GiraphVertexReader.java | 67 ++++++
.../giraph/structure/io/GiraphVertexWriter.java | 57 +++++
.../gremlin/process/computer/GraphComputer.java | 5 +-
.../process/computer/GraphComputerTest.java | 2 +-
.../tinkerpop/gremlin/hadoop/Constants.java | 1 -
.../structure/hdfs/HadoopElementIterator.java | 4 +-
.../hadoop/structure/io/InputOutputHelper.java | 22 ++
.../hadoop/structure/util/HadoopHelper.java | 50 -----
.../process/computer/SparkGraphComputer.java | 75 ++++---
.../process/computer/io/InputFormatRDD.java | 47 ----
.../spark/process/computer/io/InputRDD.java | 41 ----
.../process/computer/io/OutputFormatRDD.java | 49 -----
.../spark/process/computer/io/OutputRDD.java | 31 ---
.../spark/structure/io/InputFormatRDD.java | 47 ++++
.../spark/structure/io/InputOutputHelper.java | 81 +++++++
.../gremlin/spark/structure/io/InputRDD.java | 41 ++++
.../spark/structure/io/OutputFormatRDD.java | 49 +++++
.../gremlin/spark/structure/io/OutputRDD.java | 31 +++
.../spark/structure/io/PersistedInputRDD.java | 60 +++++
.../spark/structure/io/PersistedOutputRDD.java | 41 ++++
.../process/computer/io/ExampleInputRDD.java | 47 ----
.../process/computer/io/ExampleOutputRDD.java | 45 ----
.../process/computer/io/InputOutputRDDTest.java | 59 -----
.../spark/process/computer/io/InputRDDTest.java | 54 -----
.../process/computer/io/OutputRDDTest.java | 62 ------
.../spark/structure/io/ExampleInputRDD.java | 51 +++++
.../spark/structure/io/ExampleOutputRDD.java | 50 +++++
.../spark/structure/io/InputOutputRDDTest.java | 59 +++++
.../spark/structure/io/InputRDDTest.java | 54 +++++
.../spark/structure/io/OutputRDDTest.java | 62 ++++++
.../io/PersistedInputOutputRDDTest.java | 217 +++++++++++++++++++
.../process/computer/TinkerGraphComputer.java | 5 -
43 files changed, 1228 insertions(+), 811 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/558c04e7/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --cc CHANGELOG.asciidoc
index 84cb9f5,e79a0a0..5ad89e5
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@@ -25,10 -25,10 +25,11 @@@ image::https://raw.githubusercontent.co
TinkerPop 3.1.0 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Integrated `NumberHelper` in `SumStep`, `MinStep`, `MaxStep` and `MeanStep` (local and global step variants).
* Bumped to Neo4j 2.3.0.
+ * Added `PersistedInputRDD` and `PersistedOutputRDD` which enables `SparkGraphComputer` to store the graph RDD in the context between jobs (no HDFS serialization required).
* Renamed the `public static String` configuration variable names of TinkerGraph (deprecated old variables).
- * Added `GraphComputer.config(key,value)` to allow engine-specific configurations.
+ * Added `GraphComputer.configure(key,value)` to allow engine-specific configurations.
* `GraphStep` is no longer in the `sideEffect`-package and is now in `map`-package (breaking change).
* Added support for mid-traversal `V()`-steps (`GraphStep` semantics updated).
* Fixed `Number` handling in `Operator` enums. Prior this change a lot of operations on mixed `Number` types returned a wrong result (wrong data type).
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/558c04e7/docs/src/upgrade-release-3.1.x-incubating.asciidoc
----------------------------------------------------------------------
diff --cc docs/src/upgrade-release-3.1.x-incubating.asciidoc
index 00be151,9e5b50d..ada54f9
--- a/docs/src/upgrade-release-3.1.x-incubating.asciidoc
+++ b/docs/src/upgrade-release-3.1.x-incubating.asciidoc
@@@ -165,3 -172,11 +173,11 @@@ The `VendorOptimizationStrategy` has be
with revised terminology for what were formerly referred to as "vendors".
See link:https://issues.apache.org/jira/browse/TINKERPOP3-876[TINKERPOP3-876] for more information.
+
+ GraphComputer Updates
+ +++++++++++++++++++++
+
+ `GraphComputer.configure(String key, Object value)` is now a method (with default implementation).
+ This allows the user to specify engine-specific parameters to the underlying OLAP system. These parameters are not intended
+ to be cross engine supported. Moreover, if there are not parameters that can be altered (beyond the standard `GraphComputer`
-methods), then the provider's `GraphComputer` implementation should simply return and do nothing.
++methods), then the provider's `GraphComputer` implementation should simply return and do nothing.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/558c04e7/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/computer/GraphComputerTest.java
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/558c04e7/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDDTest.java
----------------------------------------------------------------------
diff --cc spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDDTest.java
index 0000000,8d6c06e..e8bd44a
mode 000000,100644..100644
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDDTest.java
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDDTest.java
@@@ -1,0 -1,54 +1,54 @@@
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+ package org.apache.tinkerpop.gremlin.spark.structure.io;
+
+ import org.apache.commons.configuration.BaseConfiguration;
+ import org.apache.commons.configuration.Configuration;
+ import org.apache.tinkerpop.gremlin.hadoop.Constants;
+ import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
+ import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat;
+ import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
+ import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;
+ import org.apache.tinkerpop.gremlin.structure.Graph;
+ import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
+ import org.junit.Test;
+
+ import static org.junit.Assert.assertEquals;
+
+ /**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+ public class InputRDDTest {
+
+ @Test
+ public void shouldReadFromArbitraryRDD() {
+ final Configuration configuration = new BaseConfiguration();
+ configuration.setProperty("spark.master", "local[4]");
+ configuration.setProperty("spark.serializer", "org.apache.spark.serializer.KryoSerializer");
+ configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, ExampleInputRDD.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT, GryoOutputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, "target/test-output");
+ configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
+ ////////
+ Graph graph = GraphFactory.open(configuration);
- assertEquals(Double.valueOf(123.0d), graph.traversal(GraphTraversalSource.computer(SparkGraphComputer.class)).V().values("age").sum().next());
++ assertEquals(123l, graph.traversal(GraphTraversalSource.computer(SparkGraphComputer.class)).V().values("age").sum().next());
+ assertEquals(Long.valueOf(4l), graph.traversal(GraphTraversalSource.computer(SparkGraphComputer.class)).V().count().next());
+ }
+ }
[23/50] [abbrv] incubator-tinkerpop git commit: merged TINKERPOP3-925
Posted by sp...@apache.org.
merged TINKERPOP3-925
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/0dc2dbce
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/0dc2dbce
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/0dc2dbce
Branch: refs/heads/TINKERPOP3-923
Commit: 0dc2dbce055b827a675fd4535e4760a0f5539cee
Parents: 19b7ae0 f9d1fa7
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Mon Nov 2 08:40:50 2015 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Mon Nov 2 08:40:50 2015 -0700
----------------------------------------------------------------------
CHANGELOG.asciidoc | 3 +-
docs/src/implementations.asciidoc | 36 +--
docs/src/the-traversal.asciidoc | 11 +-
.../upgrade-release-3.1.x-incubating.asciidoc | 8 +
.../process/computer/GiraphGraphComputer.java | 11 +-
.../computer/io/GiraphVertexInputFormat.java | 70 ------
.../computer/io/GiraphVertexOutputFormat.java | 65 ------
.../process/computer/io/GiraphVertexReader.java | 67 ------
.../process/computer/io/GiraphVertexWriter.java | 57 -----
.../structure/io/GiraphVertexInputFormat.java | 70 ++++++
.../structure/io/GiraphVertexOutputFormat.java | 65 ++++++
.../giraph/structure/io/GiraphVertexReader.java | 67 ++++++
.../giraph/structure/io/GiraphVertexWriter.java | 57 +++++
.../gremlin/process/computer/GraphComputer.java | 5 +-
.../process/computer/GraphComputerTest.java | 2 +-
.../tinkerpop/gremlin/hadoop/Constants.java | 1 -
.../hadoop/structure/io/InputOutputHelper.java | 22 ++
.../hadoop/structure/util/HadoopHelper.java | 50 -----
.../process/computer/SparkGraphComputer.java | 75 ++++---
.../process/computer/io/InputFormatRDD.java | 47 ----
.../spark/process/computer/io/InputRDD.java | 41 ----
.../process/computer/io/OutputFormatRDD.java | 49 -----
.../spark/process/computer/io/OutputRDD.java | 31 ---
.../spark/structure/io/InputFormatRDD.java | 47 ++++
.../spark/structure/io/InputOutputHelper.java | 81 +++++++
.../gremlin/spark/structure/io/InputRDD.java | 41 ++++
.../spark/structure/io/OutputFormatRDD.java | 49 +++++
.../gremlin/spark/structure/io/OutputRDD.java | 31 +++
.../spark/structure/io/PersistedInputRDD.java | 60 +++++
.../spark/structure/io/PersistedOutputRDD.java | 41 ++++
.../process/computer/io/ExampleInputRDD.java | 47 ----
.../process/computer/io/ExampleOutputRDD.java | 45 ----
.../process/computer/io/InputOutputRDDTest.java | 59 -----
.../spark/process/computer/io/InputRDDTest.java | 54 -----
.../process/computer/io/OutputRDDTest.java | 62 ------
.../spark/structure/io/ExampleInputRDD.java | 51 +++++
.../spark/structure/io/ExampleOutputRDD.java | 50 +++++
.../spark/structure/io/InputOutputRDDTest.java | 59 +++++
.../spark/structure/io/InputRDDTest.java | 54 +++++
.../spark/structure/io/OutputRDDTest.java | 62 ++++++
.../io/PersistedInputOutputRDDTest.java | 217 +++++++++++++++++++
.../process/computer/TinkerGraphComputer.java | 5 -
42 files changed, 1217 insertions(+), 808 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/0dc2dbce/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --cc CHANGELOG.asciidoc
index 06d0db4,7ee21ad..e79a0a0
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@@ -25,11 -25,11 +25,12 @@@ image::https://raw.githubusercontent.co
TinkerPop 3.1.0 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-* Added `PersistedInputRDD` and `PersistedOutputRDD` which enables `SparkGraphComputer` to store the graph RDD in the context between jobs.
+* Bumped to Neo4j 2.3.0.
++* Added `PersistedInputRDD` and `PersistedOutputRDD` which enables `SparkGraphComputer` to store the graph RDD in the context between jobs (no HDFS serialization required).
* Renamed the `public static String` configuration variable names of TinkerGraph (deprecated old variables).
- * Added `GraphComputer.config(key,value)` to allow engine-specific configurations.
+ * Added `GraphComputer.configure(key,value)` to allow engine-specific configurations.
* `GraphStep` is no longer in the `sideEffect`-package and is now in `map`-package (breaking change).
-* Added suppport for mid-traversal `V()`-steps (`GraphStep` semantics updated).
+* Added support for mid-traversal `V()`-steps (`GraphStep` semantics updated).
* Fixed `Number` handling in `Operator` enums. Prior this change a lot of operations on mixed `Number` types returned a wrong result (wrong data type).
* Fixed a bug in Gremlin Server/Driver serializer where empty buffers were getting returned in certain cases.
* Renamed `ConjunctionX` to `ConnectiveX` because "conjunction" is assumed "and" (disjunction "or"), where "connective" is the parent concept.
[06/50] [abbrv] incubator-tinkerpop git commit: merged master/.
Posted by sp...@apache.org.
merged master/.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/16b50052
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/16b50052
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/16b50052
Branch: refs/heads/TINKERPOP3-923
Commit: 16b50052eb27ebb365341dc3b6e90608b07a71e6
Parents: 3d2d6a6 9300485
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Thu Oct 29 17:29:02 2015 -0600
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Thu Oct 29 17:29:02 2015 -0600
----------------------------------------------------------------------
CHANGELOG.asciidoc | 4 +
CONTRIBUTING.asciidoc | 246 +----------------
RELEASE.asciidoc | 172 ------------
docs/src/developer-contributing.asciidoc | 272 +++++++++++++++++++
docs/src/developer-meetings.asciidoc | 64 +++++
docs/src/developer-release.asciidoc | 171 ++++++++++++
docs/src/developer.asciidoc | 31 +++
docs/src/gremlin-applications.asciidoc | 3 +-
docs/src/the-traversal.asciidoc | 25 ++
.../upgrade-release-3.1.x-incubating.asciidoc | 65 ++++-
docs/static/images/gremlin-standing-strong.png | Bin 0 -> 30985 bytes
.../process/computer/GiraphGraphComputer.java | 7 +
.../GephiTraversalVisualizationStrategy.groovy | 2 +-
.../gremlin/process/computer/GraphComputer.java | 16 +-
.../traversal/TraversalVertexProgram.java | 4 +-
.../traversal/dsl/graph/GraphTraversal.java | 7 +-
.../dsl/graph/GraphTraversalSource.java | 10 +-
.../gremlin/process/traversal/dsl/graph/__.java | 9 +-
.../process/traversal/step/map/GraphStep.java | 149 ++++++++++
.../traversal/step/sideEffect/GraphStep.java | 101 -------
.../step/util/TraversalComparator.java | 5 +-
.../strategy/decoration/ConnectiveStrategy.java | 3 +-
.../strategy/decoration/ElementIdStrategy.java | 2 +-
.../strategy/decoration/PartitionStrategy.java | 2 +-
.../strategy/decoration/SubgraphStrategy.java | 2 +-
.../finalization/LazyBarrierStrategy.java | 2 +-
.../ComputerVerificationStrategy.java | 39 ++-
.../process/traversal/util/TraversalHelper.java | 3 +
.../traversal/step/map/GraphStepTest.java | 41 +++
.../ElementIdStrategyTraverseTest.java | 4 +-
.../PartitionStrategyTraverseTest.java | 4 +-
.../SubgraphStrategyTraverseTest.java | 5 +-
.../traversal/step/map/GroovyGraphTest.groovy | 50 ++++
.../traversal/step/map/GroovyOrderTest.groovy | 10 +
.../process/GroovyProcessComputerSuite.java | 2 +
.../process/GroovyProcessStandardSuite.java | 2 +
.../handler/NioGremlinBinaryRequestDecoder.java | 5 +
.../handler/WsGremlinBinaryRequestDecoder.java | 5 +
.../handler/WsGremlinCloseRequestDecoder.java | 5 +
.../handler/WsGremlinTextRequestDecoder.java | 5 +
.../server/GremlinDriverIntegrateTest.java | 72 ++++-
.../server/GremlinServerIntegrateTest.java | 100 +++++++
.../gremlin/process/ProcessComputerSuite.java | 2 +
.../gremlin/process/ProcessStandardSuite.java | 2 +
.../process/computer/GraphComputerTest.java | 5 +
.../traversal/step/map/AddVertexTest.java | 2 +-
.../process/traversal/step/map/GraphTest.java | 109 ++++++++
.../process/traversal/step/map/OrderTest.java | 58 +++-
.../step/sideEffect/Neo4jGraphStep.java | 10 +-
.../optimization/Neo4jGraphStepStrategy.java | 13 +-
.../gremlin/neo4j/structure/Neo4jGraph.java | 1 +
.../traversal/strategy/Neo4jStrategySuite.java | 44 +++
.../traversal/strategy/Neo4jStrategyTest.java | 32 +++
.../Neo4jGraphStepStrategyTest.java | 76 ++++++
pom.xml | 34 ++-
.../process/computer/SparkGraphComputer.java | 16 +-
.../io/SparkContextPersistenceTest.java | 14 +-
.../process/computer/TinkerGraphComputer.java | 5 +
.../step/sideEffect/TinkerGraphStep.java | 10 +-
.../optimization/TinkerGraphStepStrategy.java | 13 +-
.../tinkergraph/structure/TinkerFactory.java | 8 +-
.../tinkergraph/structure/TinkerGraph.java | 59 +++-
.../tinkergraph/TinkerGraphProvider.java | 14 +-
.../strategy/TinkerGraphStrategySuite.java | 44 +++
.../strategy/TinkerGraphStrategyTest.java | 32 +++
.../TinkerGraphStepStrategyTest.java | 33 ++-
.../structure/TinkerGraphIdManagerTest.java | 18 +-
.../structure/TinkerGraphPlayTest.java | 11 +
.../tinkergraph/structure/TinkerGraphTest.java | 16 +-
69 files changed, 1751 insertions(+), 656 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/16b50052/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
----------------------------------------------------------------------
diff --cc spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
index 7134f26,c3593c1..ef2ae6f
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
@@@ -80,20 -88,17 +90,20 @@@ public final class SparkGraphComputer e
public Future<ComputerResult> submit() {
super.validateStatePriorToExecution();
// apache and hadoop configurations that are used throughout the graph computer computation
- final org.apache.commons.configuration.Configuration apacheConfiguration = new HadoopConfiguration(this.hadoopGraph.configuration());
- this.sparkConfiguration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT_HAS_EDGES, this.persist.equals(GraphComputer.Persist.EDGES));
- final Configuration hadoopConfiguration = ConfUtil.makeHadoopConfiguration(this.sparkConfiguration);
- if (FileInputFormat.class.isAssignableFrom(hadoopConfiguration.getClass(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, InputFormat.class))) {
- try {
- final String inputLocation = FileSystem.get(hadoopConfiguration).getFileStatus(new Path(hadoopConfiguration.get(Constants.GREMLIN_HADOOP_INPUT_LOCATION))).getPath().toString();
- this.sparkConfiguration.setProperty(Constants.MAPREDUCE_INPUT_FILEINPUTFORMAT_INPUTDIR, inputLocation);
- hadoopConfiguration.set(Constants.MAPREDUCE_INPUT_FILEINPUTFORMAT_INPUTDIR, inputLocation);
- } catch (final IOException e) {
- throw new IllegalStateException(e.getMessage(), e);
++ final org.apache.commons.configuration.Configuration apacheConfiguration = new HadoopConfiguration(this.sparkConfiguration);
+ apacheConfiguration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT_HAS_EDGES, this.persist.equals(GraphComputer.Persist.EDGES));
+ final Configuration hadoopConfiguration = ConfUtil.makeHadoopConfiguration(apacheConfiguration);
+ if (null == hadoopConfiguration.get(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD_NAME, null)) {
+ if (FileInputFormat.class.isAssignableFrom(hadoopConfiguration.getClass(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, InputFormat.class))) {
+ try {
+ final String inputLocation = FileSystem.get(hadoopConfiguration).getFileStatus(new Path(hadoopConfiguration.get(Constants.GREMLIN_HADOOP_INPUT_LOCATION))).getPath().toString();
+ apacheConfiguration.setProperty(Constants.MAPREDUCE_INPUT_FILEINPUTFORMAT_INPUTDIR, inputLocation);
+ hadoopConfiguration.set(Constants.MAPREDUCE_INPUT_FILEINPUTFORMAT_INPUTDIR, inputLocation);
+ } catch (final IOException e) {
+ throw new IllegalStateException(e.getMessage(), e);
+ }
}
- }
+ } // else WARN that both an INPUT_FORMAT and INPUT_RDD_NAME were provided?
// create the completable future
return CompletableFuture.<ComputerResult>supplyAsync(() -> {
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/16b50052/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/SparkContextPersistenceTest.java
----------------------------------------------------------------------
diff --cc spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/SparkContextPersistenceTest.java
index e19d89f,0000000..cc0957f
mode 100644,000000..100644
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/SparkContextPersistenceTest.java
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/SparkContextPersistenceTest.java
@@@ -1,126 -1,0 +1,126 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tinkerpop.gremlin.spark.process.computer.io;
+
+import org.apache.commons.configuration.BaseConfiguration;
+import org.apache.commons.configuration.Configuration;
+import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat;
+import org.apache.spark.SparkConf;
+import org.apache.spark.SparkContext;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat;
+import org.apache.tinkerpop.gremlin.hadoop.structure.util.ConfUtil;
+import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
+import org.apache.tinkerpop.gremlin.process.computer.bulkloading.BulkLoaderVertexProgram;
+import org.apache.tinkerpop.gremlin.process.computer.ranking.pagerank.PageRankVertexProgram;
+import org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram;
+import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
+import org.apache.tinkerpop.gremlin.process.traversal.engine.ComputerTraversalEngine;
+import org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer;
+import org.apache.tinkerpop.gremlin.spark.process.computer.SparkHadoopGraphProvider;
+import org.apache.tinkerpop.gremlin.spark.process.computer.util.SparkHelper;
+import org.apache.tinkerpop.gremlin.structure.Graph;
+import org.apache.tinkerpop.gremlin.structure.io.IoCore;
- import org.apache.tinkerpop.gremlin.structure.io.gryo.GryoIo;
+import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
+import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
+import org.junit.Test;
+
- import static org.junit.Assert.*;
++import static org.junit.Assert.assertEquals;
++import static org.junit.Assert.assertTrue;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public class SparkContextPersistenceTest {
+
+ @Test
+ public void shouldPersistRDDAcrossJobs() throws Exception {
+ final Configuration configuration = new BaseConfiguration();
+ configuration.setProperty("spark.master", "local[4]");
+ configuration.setProperty("spark.serializer", "org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer");
+ configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, NullOutputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD_NAME, "a-random-name-for-testing");
+ configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, true);
+ Graph graph = GraphFactory.open(configuration);
+ graph.compute(SparkGraphComputer.class)
+ .result(GraphComputer.ResultGraph.NEW)
+ .persist(GraphComputer.Persist.NOTHING)
+ .program(TraversalVertexProgram.build()
+ .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
+ "gremlin-groovy",
+ "g.V()").create(graph)).submit().get();
+ ////////
+ SparkConf sparkConfiguration = new SparkConf();
+ sparkConfiguration.setAppName("shouldPersistRDDAcrossJobs");
+ ConfUtil.makeHadoopConfiguration(configuration).forEach(entry -> sparkConfiguration.set(entry.getKey(), entry.getValue()));
+ JavaSparkContext sparkContext = new JavaSparkContext(SparkContext.getOrCreate(sparkConfiguration));
+ assertTrue(SparkHelper.getPersistedRDD(sparkContext, "a-random-name-for-testing").isPresent());
+ ///////
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD_NAME, "a-random-name-for-testing");
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD_NAME, null);
+ graph = GraphFactory.open(configuration);
+ graph.compute(SparkGraphComputer.class)
+ .result(GraphComputer.ResultGraph.NEW)
+ .persist(GraphComputer.Persist.NOTHING)
+ .program(TraversalVertexProgram.build()
+ .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
+ "gremlin-groovy",
+ "g.V()").create(graph)).submit().get();
+ }
+
+ @Test
+ public void testBulkLoaderVertexProgramChain() throws Exception {
+ final Configuration readConfiguration = new BaseConfiguration();
+ readConfiguration.setProperty("spark.master", "local[4]");
+ readConfiguration.setProperty("spark.serializer", "org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer");
+ readConfiguration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
+ readConfiguration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, NullOutputFormat.class.getCanonicalName());
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, "target/test-output");
+ readConfiguration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
+ readConfiguration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD_NAME, "a-random-name-for-testing");
+ readConfiguration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, true);
+ Graph graph = GraphFactory.open(readConfiguration);
+
+ ///////////////
+ final Configuration writeConfiguration = new BaseConfiguration();
+ writeConfiguration.setProperty(Graph.GRAPH, "org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph");
- writeConfiguration.setProperty(TinkerGraph.CONFIG_GRAPH_FORMAT, "gryo");
- writeConfiguration.setProperty(TinkerGraph.CONFIG_GRAPH_LOCATION, "target/test-output/tinkergraph.kryo");
++ writeConfiguration.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo");
++ writeConfiguration.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, "target/test-output/tinkergraph.kryo");
+ final Graph secondGraph = graph.compute(SparkGraphComputer.class).persist(GraphComputer.Persist.NOTHING).program(PageRankVertexProgram.build().create(graph)).submit().get().graph();
- secondGraph.configuration().setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD_NAME, "a-random-name-for-testing");
- secondGraph.configuration().setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD_NAME, null);
+ secondGraph.compute(SparkGraphComputer.class)
+ .persist(GraphComputer.Persist.NOTHING)
+ .workers(1)
++ .config(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD_NAME, "a-random-name-for-testing")
++ .config(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD_NAME, null)
+ .program(BulkLoaderVertexProgram.build().userSuppliedIds(true).writeGraph(writeConfiguration).create(secondGraph))
+ .submit().get();
+ final Graph finalGraph = TinkerGraph.open();
+ finalGraph.io(IoCore.gryo()).readGraph("target/test-output/tinkergraph.kryo");
- assertEquals(6l,finalGraph.traversal().V().count().next().longValue());
++ assertEquals(6l, finalGraph.traversal().V().count().next().longValue());
+ }
+}
[33/50] [abbrv] incubator-tinkerpop git commit: Merge branch 'master'
into TINKERPOP3-904
Posted by sp...@apache.org.
Merge branch 'master' into TINKERPOP3-904
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/2a4b4ac6
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/2a4b4ac6
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/2a4b4ac6
Branch: refs/heads/TINKERPOP3-923
Commit: 2a4b4ac67f4b1770a2ad69566fb819e42f312b07
Parents: 8b8222d 06d747f
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Tue Nov 3 17:48:56 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Tue Nov 3 17:48:56 2015 +0100
----------------------------------------------------------------------
CHANGELOG.asciidoc | 3 +-
docs/src/implementations.asciidoc | 36 +--
docs/src/the-traversal.asciidoc | 11 +-
.../upgrade-release-3.1.x-incubating.asciidoc | 18 +-
.../process/computer/GiraphGraphComputer.java | 11 +-
.../computer/io/GiraphVertexInputFormat.java | 70 ------
.../computer/io/GiraphVertexOutputFormat.java | 65 ------
.../process/computer/io/GiraphVertexReader.java | 67 ------
.../process/computer/io/GiraphVertexWriter.java | 57 -----
.../structure/io/GiraphVertexInputFormat.java | 70 ++++++
.../structure/io/GiraphVertexOutputFormat.java | 65 ++++++
.../giraph/structure/io/GiraphVertexReader.java | 67 ++++++
.../giraph/structure/io/GiraphVertexWriter.java | 57 +++++
.../gremlin/process/computer/GraphComputer.java | 5 +-
.../process/computer/GraphComputerTest.java | 2 +-
.../tinkerpop/gremlin/hadoop/Constants.java | 1 -
.../structure/hdfs/HadoopElementIterator.java | 4 +-
.../hadoop/structure/io/InputOutputHelper.java | 22 ++
.../hadoop/structure/util/HadoopHelper.java | 50 -----
.../process/computer/SparkGraphComputer.java | 75 ++++---
.../process/computer/io/InputFormatRDD.java | 47 ----
.../spark/process/computer/io/InputRDD.java | 41 ----
.../process/computer/io/OutputFormatRDD.java | 49 -----
.../spark/process/computer/io/OutputRDD.java | 31 ---
.../spark/structure/io/InputFormatRDD.java | 47 ++++
.../spark/structure/io/InputOutputHelper.java | 81 +++++++
.../gremlin/spark/structure/io/InputRDD.java | 41 ++++
.../spark/structure/io/OutputFormatRDD.java | 49 +++++
.../gremlin/spark/structure/io/OutputRDD.java | 31 +++
.../spark/structure/io/PersistedInputRDD.java | 60 +++++
.../spark/structure/io/PersistedOutputRDD.java | 41 ++++
.../process/computer/io/ExampleInputRDD.java | 47 ----
.../process/computer/io/ExampleOutputRDD.java | 45 ----
.../process/computer/io/InputOutputRDDTest.java | 59 -----
.../spark/process/computer/io/InputRDDTest.java | 54 -----
.../process/computer/io/OutputRDDTest.java | 62 ------
.../spark/structure/io/ExampleInputRDD.java | 51 +++++
.../spark/structure/io/ExampleOutputRDD.java | 50 +++++
.../spark/structure/io/InputOutputRDDTest.java | 59 +++++
.../spark/structure/io/InputRDDTest.java | 54 +++++
.../spark/structure/io/OutputRDDTest.java | 62 ++++++
.../io/PersistedInputOutputRDDTest.java | 217 +++++++++++++++++++
.../process/computer/TinkerGraphComputer.java | 5 -
43 files changed, 1228 insertions(+), 811 deletions(-)
----------------------------------------------------------------------
[41/50] [abbrv] incubator-tinkerpop git commit: fixed
LocalPropertyTest to use KryoSerializer instead of GryoSerializer.
Posted by sp...@apache.org.
fixed LocalPropertyTest to use KryoSerializer instead of GryoSerializer.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/2edefa60
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/2edefa60
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/2edefa60
Branch: refs/heads/TINKERPOP3-923
Commit: 2edefa604c30c5f93d51f0f624241e4e80ac0725
Parents: ebec095 caf0e2a
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Wed Nov 4 11:39:13 2015 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Wed Nov 4 11:39:13 2015 -0700
----------------------------------------------------------------------
CHANGELOG.asciidoc | 1 +
docs/src/implementations.asciidoc | 9 ++
.../process/computer/SparkGraphComputer.java | 32 ++++++
.../process/computer/LocalPropertyTest.java | 100 +++++++++++++++++++
4 files changed, 142 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/2edefa60/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --cc CHANGELOG.asciidoc
index 31671c0,331ad93..085bed7
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@@ -25,9 -25,7 +25,10 @@@ image::https://raw.githubusercontent.co
TinkerPop 3.1.0 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-* Added the ability to set thread local properties in `SparkGraphComputer` when using a persistent context
+* Integrated `NumberHelper` in `SumStep`, `MinStep`, `MaxStep` and `MeanStep` (local and global step variants).
+* `CountMatchAlgorithm`, in OLAP, now biases traversal selection towards those traversals that start at the current traverser location to reduce message passing.
+* Fixed a file stream bug in Hadoop OLTP that showed up if the streamed file was more than 2G of data.
++* Added the ability to set thread local properties in `SparkGraphComputer` when using a persistent context.
* Bumped to Neo4j 2.3.0.
* Added `PersistedInputRDD` and `PersistedOutputRDD` which enables `SparkGraphComputer` to store the graph RDD in the context between jobs (no HDFS serialization required).
* Renamed the `public static String` configuration variable names of TinkerGraph (deprecated old variables).
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/2edefa60/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/LocalPropertyTest.java
----------------------------------------------------------------------
diff --cc spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/LocalPropertyTest.java
index 0000000,57c3526..e0fe796
mode 000000,100644..100644
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/LocalPropertyTest.java
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/LocalPropertyTest.java
@@@ -1,0 -1,102 +1,100 @@@
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+ package org.apache.tinkerpop.gremlin.spark.process.computer;
+
+ import org.apache.commons.configuration.BaseConfiguration;
+ import org.apache.commons.configuration.Configuration;
+ import org.apache.spark.SparkConf;
+ import org.apache.spark.SparkContext;
+ import org.apache.spark.api.java.JavaSparkContext;
+ import org.apache.spark.api.java.JavaSparkStatusTracker;
++import org.apache.spark.serializer.KryoSerializer;
+ import org.apache.tinkerpop.gremlin.hadoop.Constants;
+ import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
+ import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat;
+ import org.apache.tinkerpop.gremlin.hadoop.structure.util.ConfUtil;
+ import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
+ import org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram;
+ import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
+ import org.apache.tinkerpop.gremlin.process.traversal.engine.ComputerTraversalEngine;
+ import org.apache.tinkerpop.gremlin.spark.structure.io.PersistedInputRDD;
+ import org.apache.tinkerpop.gremlin.spark.structure.io.PersistedOutputRDD;
-import org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerializer;
+ import org.apache.tinkerpop.gremlin.structure.Graph;
+ import org.apache.tinkerpop.gremlin.structure.util.GraphFactory;
+ import org.junit.Test;
+
+ import java.util.UUID;
+
+ import static org.junit.Assert.assertTrue;
+
-public class LocalPropertyTest
-{
++public class LocalPropertyTest {
+
+
+ @Test
- public void shouldSetThreadLocalProperties() throws Exception
- {
++ public void shouldSetThreadLocalProperties() throws Exception {
+ final String testName = "ThreadLocalProperties";
+ final String rddName = "target/test-output/" + UUID.randomUUID();
+ final Configuration configuration = new BaseConfiguration();
+ configuration.setProperty("spark.master", "local[4]");
- configuration.setProperty("spark.serializer", GryoSerializer.class.getCanonicalName());
++ configuration.setProperty("spark.serializer", KryoSerializer.class.getCanonicalName());
+ configuration.setProperty(Graph.GRAPH, HadoopGraph.class.getName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, SparkHadoopGraphProvider.PATHS.get("tinkerpop-modern.kryo"));
+ configuration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, GryoInputFormat.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, PersistedOutputRDD.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, rddName);
+ configuration.setProperty(Constants.GREMLIN_HADOOP_JARS_IN_DISTRIBUTED_CACHE, false);
+ configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, true);
+ configuration.setProperty("spark.jobGroup.id", "22");
+ Graph graph = GraphFactory.open(configuration);
+ graph.compute(SparkGraphComputer.class)
+ .result(GraphComputer.ResultGraph.NEW)
+ .persist(GraphComputer.Persist.EDGES)
+ .program(TraversalVertexProgram.build()
+ .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
+ "gremlin-groovy",
+ "g.V()").create(graph)).submit().get();
+ ////////
+ SparkConf sparkConfiguration = new SparkConf();
+ sparkConfiguration.setAppName(testName);
+ ConfUtil.makeHadoopConfiguration(configuration).forEach(entry -> sparkConfiguration.set(entry.getKey(), entry.getValue()));
+ JavaSparkContext sparkContext = new JavaSparkContext(SparkContext.getOrCreate(sparkConfiguration));
+ JavaSparkStatusTracker statusTracker = sparkContext.statusTracker();
+ assertTrue(statusTracker.getJobIdsForGroup("22").length >= 1);
+ assertTrue(PersistedInputRDD.getPersistedRDD(sparkContext, rddName).isPresent());
+ ///////
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, PersistedInputRDD.class.getCanonicalName());
+ configuration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, rddName);
+ configuration.setProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, null);
+ configuration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, null);
+ configuration.setProperty(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, false);
+ configuration.setProperty("spark.jobGroup.id", "44");
+ graph = GraphFactory.open(configuration);
+ graph.compute(SparkGraphComputer.class)
+ .result(GraphComputer.ResultGraph.NEW)
+ .persist(GraphComputer.Persist.NOTHING)
+ .program(TraversalVertexProgram.build()
+ .traversal(GraphTraversalSource.build().engine(ComputerTraversalEngine.build().computer(SparkGraphComputer.class)),
+ "gremlin-groovy",
+ "g.V()").create(graph)).submit().get();
+ ///////
+ assertTrue(statusTracker.getJobIdsForGroup("44").length >= 1);
+ }
+ }
+
[36/50] [abbrv] incubator-tinkerpop git commit: TINKERPOP-911:
Documentation for Thread Local Properties for Spark
Posted by sp...@apache.org.
TINKERPOP-911: Documentation for Thread Local Properties for Spark
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/caf0e2a4
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/caf0e2a4
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/caf0e2a4
Branch: refs/heads/TINKERPOP3-923
Commit: caf0e2a434126efdb882680e8fa0bbd752cde989
Parents: 82229d0
Author: Russell Spitzer <Ru...@gmail.com>
Authored: Tue Nov 3 09:18:14 2015 -0800
Committer: Russell Spitzer <Ru...@gmail.com>
Committed: Tue Nov 3 09:18:14 2015 -0800
----------------------------------------------------------------------
CHANGELOG.asciidoc | 1 +
docs/src/implementations.asciidoc | 9 +++++++++
2 files changed, 10 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/caf0e2a4/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index e79a0a0..331ad93 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -25,6 +25,7 @@ image::https://raw.githubusercontent.com/apache/incubator-tinkerpop/master/docs/
TinkerPop 3.1.0 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Added the ability to set thread local properties in `SparkGraphComputer` when using a persistent context
* Bumped to Neo4j 2.3.0.
* Added `PersistedInputRDD` and `PersistedOutputRDD` which enables `SparkGraphComputer` to store the graph RDD in the context between jobs (no HDFS serialization required).
* Renamed the `public static String` configuration variable names of TinkerGraph (deprecated old variables).
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/caf0e2a4/docs/src/implementations.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/implementations.asciidoc b/docs/src/implementations.asciidoc
index 7792665..b5c2c4d 100644
--- a/docs/src/implementations.asciidoc
+++ b/docs/src/implementations.asciidoc
@@ -1206,6 +1206,15 @@ Note that `gremlin.spark.persistContext` should be set to `true` or else the per
The persisted RDD is named by the `gremlin.hadoop.outputLocation` configuration (i.e. named in `SparkContext.getPersistedRDDs()`).
Finally, `PersistedInputRDD` is used with respective `gremlin.hadoop.inputLocation` to retrieve the persisted RDD from the `SparkContext`.
+When using a persistent `Spark Context` the configuration used by the original Spark Configuration will be inherited by all threaded
+references to that Spark Context. The exception to this rule are those properties which have a specific thread local effect.
+
+.Thread Local Properties
+. spark.jobGroup.id
+. spark.job.description
+. spark.job.interruptOnCancel
+. spark.scheduler.pool
+
Loading with BulkLoaderVertexProgram
++++++++++++++++++++++++++++++++++++
[17/50] [abbrv] incubator-tinkerpop git commit: Tweaks to Neo4jHelper
as NotFoundException should be checked as the reason for isNotFound() catch.
Also, static String construction for cost savings.
Posted by sp...@apache.org.
Tweaks to Neo4jHelper as NotFoundException should be checked as the reason for isNotFound() catch. Also, static String construction for cost savings.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/1a31b78c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/1a31b78c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/1a31b78c
Branch: refs/heads/TINKERPOP3-923
Commit: 1a31b78c4a158f7fcdbbf01f09d9d2801a12ab11
Parents: 4c6ba90
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Sat Oct 31 11:25:00 2015 -0600
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Sat Oct 31 11:25:00 2015 -0600
----------------------------------------------------------------------
.../gremlin/neo4j/structure/Neo4jHelper.java | 16 +++++++++-------
1 file changed, 9 insertions(+), 7 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/1a31b78c/neo4j-gremlin/src/main/java/org/apache/tinkerpop/gremlin/neo4j/structure/Neo4jHelper.java
----------------------------------------------------------------------
diff --git a/neo4j-gremlin/src/main/java/org/apache/tinkerpop/gremlin/neo4j/structure/Neo4jHelper.java b/neo4j-gremlin/src/main/java/org/apache/tinkerpop/gremlin/neo4j/structure/Neo4jHelper.java
index 7c8c0d0..b205019 100644
--- a/neo4j-gremlin/src/main/java/org/apache/tinkerpop/gremlin/neo4j/structure/Neo4jHelper.java
+++ b/neo4j-gremlin/src/main/java/org/apache/tinkerpop/gremlin/neo4j/structure/Neo4jHelper.java
@@ -19,16 +19,15 @@
package org.apache.tinkerpop.gremlin.neo4j.structure;
import org.apache.tinkerpop.gremlin.structure.Direction;
-import org.apache.tinkerpop.gremlin.structure.Element;
-import org.apache.tinkerpop.gremlin.structure.Vertex;
import org.neo4j.tinkerpop.api.Neo4jNode;
-import org.neo4j.tinkerpop.api.Neo4jRelationship;
/**
* @author Marko A. Rodriguez (http://markorodriguez.com)
*/
public final class Neo4jHelper {
+ private static final String NOT_FOUND_EXCEPTION = "NotFoundException";
+
private Neo4jHelper() {
}
@@ -45,13 +44,16 @@ public final class Neo4jHelper {
try {
node.getKeys();
return false;
- } catch (final IllegalStateException e) {
- return true;
+ } catch (final RuntimeException e) {
+ if (isNotFound(e))
+ return true;
+ else
+ throw e;
}
}
- public static boolean isNotFound(RuntimeException ex) {
- return ex.getClass().getSimpleName().equals("NotFoundException");
+ public static boolean isNotFound(final RuntimeException ex) {
+ return ex.getClass().getSimpleName().equals(NOT_FOUND_EXCEPTION);
}
public static Neo4jNode getVertexPropertyNode(final Neo4jVertexProperty vertexProperty) {
[46/50] [abbrv] incubator-tinkerpop git commit: Merge remote-tracking
branch 'origin/master'
Posted by sp...@apache.org.
Merge remote-tracking branch 'origin/master'
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/863bdab1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/863bdab1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/863bdab1
Branch: refs/heads/TINKERPOP3-923
Commit: 863bdab1f62b82001efabc1cd06b87382d2d263b
Parents: 88abff6 c4fcae6
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Wed Nov 4 14:36:30 2015 -0500
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Wed Nov 4 14:36:30 2015 -0500
----------------------------------------------------------------------
CHANGELOG.asciidoc | 2 +
docs/src/implementations.asciidoc | 9 ++
docs/src/the-graphcomputer.asciidoc | 4 +
.../computer/bulkloading/BulkLoader.java | 31 +++++
.../bulkloading/BulkLoaderVertexProgram.java | 115 ++++++++++++++++---
.../bulkloading/IncrementalBulkLoader.java | 14 ++-
.../process/computer/SparkGraphComputer.java | 32 ++++++
.../process/computer/LocalPropertyTest.java | 100 ++++++++++++++++
.../structure/TinkerGraphPlayTest.java | 19 ++-
9 files changed, 304 insertions(+), 22 deletions(-)
----------------------------------------------------------------------
[27/50] [abbrv] incubator-tinkerpop git commit: minor fix in
MeanGlobalStep
Posted by sp...@apache.org.
minor fix in MeanGlobalStep
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/dfa61c9f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/dfa61c9f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/dfa61c9f
Branch: refs/heads/TINKERPOP3-923
Commit: dfa61c9f7416d2b697e3e8d5350e94137e1e7ff0
Parents: e29a936
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Tue Nov 3 15:51:25 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Tue Nov 3 15:51:25 2015 +0100
----------------------------------------------------------------------
.../gremlin/process/traversal/step/map/MeanGlobalStep.java | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/dfa61c9f/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanGlobalStep.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanGlobalStep.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanGlobalStep.java
index f658271..a4e4272 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanGlobalStep.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanGlobalStep.java
@@ -199,7 +199,7 @@ public final class MeanGlobalStep<S extends Number, E extends Number> extends Re
@Override
public String toString() {
- return div(this.sum, this.count, true).toString();
+ return getFinal().toString();
}
@Override
@@ -219,8 +219,8 @@ public final class MeanGlobalStep<S extends Number, E extends Number> extends Re
}
@Override
- public Double getFinal() {
- return this.doubleValue();
+ public Number getFinal() {
+ return div(this.sum, this.count, true);
}
}
}
\ No newline at end of file
[08/50] [abbrv] incubator-tinkerpop git commit: @RussellSpitzer
explained how to unpersist RDDs works and recommended to drop the RDDs that
are OutputFormatted (as they are in HDFS) or are not persisted with OutputRDD
or Persist.NOTHING is set.
Posted by sp...@apache.org.
@RussellSpitzer explained how to unpersist RDDs works and recommended to drop the RDDs that are OutputFormatted (as they are in HDFS) or are not persisted with OutputRDD or Persist.NOTHING is set.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/1f6c5744
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/1f6c5744
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/1f6c5744
Branch: refs/heads/TINKERPOP3-923
Commit: 1f6c57444e8547f0b69bf3e31494ba4ca6ca685d
Parents: 528ba02
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Fri Oct 30 12:22:01 2015 -0600
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Fri Oct 30 12:22:01 2015 -0600
----------------------------------------------------------------------
.../gremlin/spark/process/computer/SparkGraphComputer.java | 4 ++++
.../gremlin/spark/process/computer/io/OutputFormatRDD.java | 1 +
.../process/computer/io/PersistedInputOutputRDDTest.java | 8 ++++++--
3 files changed, 11 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/1f6c5744/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
index c9e95e3..0656f5a 100644
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
@@ -211,6 +211,10 @@ public final class SparkGraphComputer extends AbstractHadoopGraphComputer {
}
}
+ // unpersist the graphRDD if it will no longer be used
+ if (hadoopConfiguration.get(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD, null) == null || this.persist.equals(GraphComputer.Persist.NOTHING)) {
+ graphRDD.unpersist();
+ }
// update runtime and return the newly computed graph
finalMemory.setRuntime(System.currentTimeMillis() - startTime);
return new DefaultComputerResult(HadoopHelper.getOutputGraph(this.hadoopGraph, this.resultGraph, this.persist), finalMemory.asImmutable());
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/1f6c5744/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputFormatRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputFormatRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputFormatRDD.java
index 56a1297..661dc51 100644
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputFormatRDD.java
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputFormatRDD.java
@@ -45,5 +45,6 @@ public final class OutputFormatRDD implements OutputRDD {
VertexWritable.class,
(Class<OutputFormat<NullWritable, VertexWritable>>) hadoopConfiguration.getClass(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT, OutputFormat.class), hadoopConfiguration);
}
+ graphRDD.unpersist();
}
}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/1f6c5744/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputOutputRDDTest.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputOutputRDDTest.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputOutputRDDTest.java
index 332d80d..a7d1ac0 100644
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputOutputRDDTest.java
+++ b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputOutputRDDTest.java
@@ -142,7 +142,7 @@ public class PersistedInputOutputRDDTest {
writeConfiguration.setProperty(Graph.GRAPH, TinkerGraph.class.getCanonicalName());
writeConfiguration.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_FORMAT, "gryo");
writeConfiguration.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, "target/test-output/tinkergraph.kryo");
- final Graph bulkLoaderGraph = pageRankGraph.compute(SparkGraphComputer.class).persist(GraphComputer.Persist.NOTHING).program(PageRankVertexProgram.build().create(pageRankGraph)).submit().get().graph();
+ final Graph bulkLoaderGraph = pageRankGraph.compute(SparkGraphComputer.class).persist(GraphComputer.Persist.VERTEX_PROPERTIES).program(PageRankVertexProgram.build().create(pageRankGraph)).submit().get().graph();
bulkLoaderGraph.compute(SparkGraphComputer.class)
.persist(GraphComputer.Persist.NOTHING)
.workers(1)
@@ -153,10 +153,14 @@ public class PersistedInputOutputRDDTest {
.program(BulkLoaderVertexProgram.build().userSuppliedIds(true).writeGraph(writeConfiguration).create(bulkLoaderGraph))
.submit().get();
////
+ SparkConf sparkConfiguration = new SparkConf();
+ sparkConfiguration.setAppName("testBulkLoaderVertexProgramChain");
+ JavaSparkContext sparkContext = new JavaSparkContext(SparkContext.getOrCreate(sparkConfiguration));
+ assertFalse(PersistedInputRDD.getPersistedRDD(sparkContext, rddName).isPresent());
+ ////
final Graph graph = TinkerGraph.open();
final GraphTraversalSource g = graph.traversal();
graph.io(IoCore.gryo()).readGraph("target/test-output/tinkergraph.kryo");
-
assertEquals(6l, g.V().count().next().longValue());
assertEquals(6l, g.E().count().next().longValue());
assertEquals("marko", g.V().has("name", "marko").values("name").next());
[11/50] [abbrv] incubator-tinkerpop git commit: @RussellSpitzer has
schooled me in cache()/unpersist(). I now am smart to unpersist() RDDs that
are just dangling around (especailly since we now have persistent contexts).
Finally, I did some refactoring o
Posted by sp...@apache.org.
@RussellSpitzer has schooled me in cache()/unpersist(). I now am smart to unpersist() RDDs that are just dangling around (especailly since we now have persistent contexts). Finally, I did some refactoring of packages (non-breaking) as I'm starting to see the future of what SparkGraph looks like (in other words, being able to use SparkGraph w/o HadoopGraph (back SparkGraph by an RDD). Anywho, that refactor cleaned this up for now and for the future so its good.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/cb972387
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/cb972387
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/cb972387
Branch: refs/heads/TINKERPOP3-923
Commit: cb972387bd369619a39b3ced215997703a8bb9e9
Parents: eeb0d9f
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Fri Oct 30 14:30:10 2015 -0600
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Fri Oct 30 14:30:10 2015 -0600
----------------------------------------------------------------------
docs/src/implementations.asciidoc | 4 +-
.../process/computer/GiraphGraphComputer.java | 8 +-
.../computer/io/GiraphVertexInputFormat.java | 70 ------
.../computer/io/GiraphVertexOutputFormat.java | 65 ------
.../process/computer/io/GiraphVertexReader.java | 67 ------
.../process/computer/io/GiraphVertexWriter.java | 57 -----
.../structure/io/GiraphVertexInputFormat.java | 70 ++++++
.../structure/io/GiraphVertexOutputFormat.java | 65 ++++++
.../giraph/structure/io/GiraphVertexReader.java | 67 ++++++
.../giraph/structure/io/GiraphVertexWriter.java | 57 +++++
.../hadoop/structure/io/InputOutputHelper.java | 22 ++
.../hadoop/structure/util/HadoopHelper.java | 50 -----
.../process/computer/SparkGraphComputer.java | 15 +-
.../process/computer/io/InputFormatRDD.java | 47 ----
.../spark/process/computer/io/InputRDD.java | 41 ----
.../process/computer/io/OutputFormatRDD.java | 49 -----
.../spark/process/computer/io/OutputRDD.java | 31 ---
.../process/computer/io/PersistedInputRDD.java | 60 -----
.../process/computer/io/PersistedOutputRDD.java | 41 ----
.../spark/structure/io/InputFormatRDD.java | 47 ++++
.../spark/structure/io/InputOutputHelper.java | 81 +++++++
.../gremlin/spark/structure/io/InputRDD.java | 41 ++++
.../spark/structure/io/OutputFormatRDD.java | 49 +++++
.../gremlin/spark/structure/io/OutputRDD.java | 31 +++
.../spark/structure/io/PersistedInputRDD.java | 60 +++++
.../spark/structure/io/PersistedOutputRDD.java | 41 ++++
.../process/computer/io/ExampleInputRDD.java | 47 ----
.../process/computer/io/ExampleOutputRDD.java | 45 ----
.../process/computer/io/InputOutputRDDTest.java | 59 -----
.../spark/process/computer/io/InputRDDTest.java | 54 -----
.../process/computer/io/OutputRDDTest.java | 62 ------
.../io/PersistedInputOutputRDDTest.java | 172 ---------------
.../spark/structure/io/ExampleInputRDD.java | 48 ++++
.../spark/structure/io/ExampleOutputRDD.java | 46 ++++
.../spark/structure/io/InputOutputRDDTest.java | 59 +++++
.../spark/structure/io/InputRDDTest.java | 54 +++++
.../spark/structure/io/OutputRDDTest.java | 62 ++++++
.../io/PersistedInputOutputRDDTest.java | 217 +++++++++++++++++++
38 files changed, 1131 insertions(+), 1030 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/docs/src/implementations.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/implementations.asciidoc b/docs/src/implementations.asciidoc
index 8110c5f..7792665 100644
--- a/docs/src/implementations.asciidoc
+++ b/docs/src/implementations.asciidoc
@@ -1001,8 +1001,8 @@ spark.serializer=org.apache.tinkerpop.gremlin.spark.structure.io.gryo.GryoSerial
####################################
# SparkGraphComputer Configuration #
####################################
-gremlin.spark.graphInputRDD=org.apache.tinkerpop.gremlin.spark.process.computer.io.InputRDDFormat
-gremlin.spark.graphOutputRDD=org.apache.tinkerpop.gremlin.spark.process.computer.io.OutputRDDFormat
+gremlin.spark.graphInputRDD=org.apache.tinkerpop.gremlin.spark.structure.io.InputRDDFormat
+gremlin.spark.graphOutputRDD=org.apache.tinkerpop.gremlin.spark.structure.io.OutputRDDFormat
gremlin.spark.persistContext=true
#####################################
# GiraphGraphComputer Configuration #
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
----------------------------------------------------------------------
diff --git a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
index 4c33220..0389750 100644
--- a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
+++ b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
@@ -34,17 +34,17 @@ import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
-import org.apache.tinkerpop.gremlin.giraph.process.computer.io.GiraphVertexInputFormat;
-import org.apache.tinkerpop.gremlin.giraph.process.computer.io.GiraphVertexOutputFormat;
+import org.apache.tinkerpop.gremlin.giraph.structure.io.GiraphVertexInputFormat;
+import org.apache.tinkerpop.gremlin.giraph.structure.io.GiraphVertexOutputFormat;
import org.apache.tinkerpop.gremlin.hadoop.Constants;
import org.apache.tinkerpop.gremlin.hadoop.process.computer.AbstractHadoopGraphComputer;
import org.apache.tinkerpop.gremlin.hadoop.process.computer.util.MapReduceHelper;
import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.InputOutputHelper;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.ObjectWritable;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.ObjectWritableIterator;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
import org.apache.tinkerpop.gremlin.hadoop.structure.util.ConfUtil;
-import org.apache.tinkerpop.gremlin.hadoop.structure.util.HadoopHelper;
import org.apache.tinkerpop.gremlin.process.computer.ComputerResult;
import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
import org.apache.tinkerpop.gremlin.process.computer.MapReduce;
@@ -126,7 +126,7 @@ public final class GiraphGraphComputer extends AbstractHadoopGraphComputer imple
}
this.memory.setRuntime(System.currentTimeMillis() - startTime);
- return new DefaultComputerResult(HadoopHelper.getOutputGraph(this.hadoopGraph, this.resultGraph, this.persist), this.memory.asImmutable());
+ return new DefaultComputerResult(InputOutputHelper.getOutputGraph(ConfUtil.makeApacheConfiguration(this.giraphConfiguration), this.resultGraph, this.persist), this.memory.asImmutable());
});
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexInputFormat.java
----------------------------------------------------------------------
diff --git a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexInputFormat.java b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexInputFormat.java
deleted file mode 100644
index e5407da..0000000
--- a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexInputFormat.java
+++ /dev/null
@@ -1,70 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.giraph.process.computer.io;
-
-import org.apache.giraph.io.VertexInputFormat;
-import org.apache.giraph.io.VertexReader;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.io.NullWritable;
-import org.apache.hadoop.mapreduce.InputFormat;
-import org.apache.hadoop.mapreduce.InputSplit;
-import org.apache.hadoop.mapreduce.JobContext;
-import org.apache.hadoop.mapreduce.TaskAttemptContext;
-import org.apache.hadoop.util.ReflectionUtils;
-import org.apache.tinkerpop.gremlin.hadoop.Constants;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
-
-import java.io.IOException;
-import java.util.List;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public final class GiraphVertexInputFormat extends VertexInputFormat {
-
- private InputFormat<NullWritable, VertexWritable> hadoopGraphInputFormat;
-
- @Override
- public void checkInputSpecs(final Configuration configuration) {
-
- }
-
- @Override
- public List<InputSplit> getSplits(final JobContext context, final int minSplitCountHint) throws IOException, InterruptedException {
- this.constructor(context.getConfiguration());
- return this.hadoopGraphInputFormat.getSplits(context);
- }
-
- @Override
- public VertexReader createVertexReader(final InputSplit split, final TaskAttemptContext context) throws IOException {
- this.constructor(context.getConfiguration());
- try {
- return new GiraphVertexReader(this.hadoopGraphInputFormat.createRecordReader(split, context));
- } catch (InterruptedException e) {
- throw new IOException(e);
- }
- }
-
- private final void constructor(final Configuration configuration) {
- if (null == this.hadoopGraphInputFormat) {
- this.hadoopGraphInputFormat = ReflectionUtils.newInstance(configuration.getClass(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, InputFormat.class, InputFormat.class), configuration);
- }
- }
-
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexOutputFormat.java
----------------------------------------------------------------------
diff --git a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexOutputFormat.java b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexOutputFormat.java
deleted file mode 100644
index c1360c7..0000000
--- a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexOutputFormat.java
+++ /dev/null
@@ -1,65 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.giraph.process.computer.io;
-
-import org.apache.tinkerpop.gremlin.hadoop.Constants;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
-import org.apache.giraph.io.VertexOutputFormat;
-import org.apache.giraph.io.VertexWriter;
-import org.apache.hadoop.conf.Configuration;
-import org.apache.hadoop.io.NullWritable;
-import org.apache.hadoop.mapreduce.JobContext;
-import org.apache.hadoop.mapreduce.OutputCommitter;
-import org.apache.hadoop.mapreduce.OutputFormat;
-import org.apache.hadoop.mapreduce.TaskAttemptContext;
-import org.apache.hadoop.util.ReflectionUtils;
-
-import java.io.IOException;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public final class GiraphVertexOutputFormat extends VertexOutputFormat {
-
- private OutputFormat<NullWritable, VertexWritable> hadoopGraphOutputFormat;
-
- @Override
- public VertexWriter createVertexWriter(final TaskAttemptContext context) throws IOException, InterruptedException {
- this.constructor(context.getConfiguration());
- return new GiraphVertexWriter(this.hadoopGraphOutputFormat);
- }
-
- @Override
- public void checkOutputSpecs(final JobContext context) throws IOException, InterruptedException {
- this.constructor(context.getConfiguration());
- this.hadoopGraphOutputFormat.checkOutputSpecs(context);
- }
-
- @Override
- public OutputCommitter getOutputCommitter(final TaskAttemptContext context) throws IOException, InterruptedException {
- this.constructor(context.getConfiguration());
- return this.hadoopGraphOutputFormat.getOutputCommitter(context);
- }
-
- private final void constructor(final Configuration configuration) {
- if (null == this.hadoopGraphOutputFormat) {
- this.hadoopGraphOutputFormat = ReflectionUtils.newInstance(configuration.getClass(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT, OutputFormat.class, OutputFormat.class), configuration);
- }
- }
-}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexReader.java
----------------------------------------------------------------------
diff --git a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexReader.java b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexReader.java
deleted file mode 100644
index c8ed797..0000000
--- a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexReader.java
+++ /dev/null
@@ -1,67 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.giraph.process.computer.io;
-
-import org.apache.giraph.graph.Vertex;
-import org.apache.giraph.io.VertexReader;
-import org.apache.hadoop.io.NullWritable;
-import org.apache.hadoop.mapreduce.InputSplit;
-import org.apache.hadoop.mapreduce.RecordReader;
-import org.apache.hadoop.mapreduce.TaskAttemptContext;
-import org.apache.tinkerpop.gremlin.giraph.process.computer.GiraphVertex;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
-
-import java.io.IOException;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public final class GiraphVertexReader extends VertexReader {
-
- private RecordReader<NullWritable, VertexWritable> recordReader;
-
- public GiraphVertexReader(final RecordReader<NullWritable, VertexWritable> recordReader) {
- this.recordReader = recordReader;
- }
-
- @Override
- public void initialize(final InputSplit inputSplit, final TaskAttemptContext context) throws IOException, InterruptedException {
- this.recordReader.initialize(inputSplit, context);
- }
-
- @Override
- public boolean nextVertex() throws IOException, InterruptedException {
- return this.recordReader.nextKeyValue();
- }
-
- @Override
- public Vertex getCurrentVertex() throws IOException, InterruptedException {
- return new GiraphVertex(this.recordReader.getCurrentValue());
- }
-
- @Override
- public void close() throws IOException {
- this.recordReader.close();
- }
-
- @Override
- public float getProgress() throws IOException, InterruptedException {
- return this.recordReader.getProgress();
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexWriter.java
----------------------------------------------------------------------
diff --git a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexWriter.java b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexWriter.java
deleted file mode 100644
index d67b736..0000000
--- a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/io/GiraphVertexWriter.java
+++ /dev/null
@@ -1,57 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.giraph.process.computer.io;
-
-import org.apache.giraph.graph.Vertex;
-import org.apache.giraph.io.VertexWriter;
-import org.apache.hadoop.io.NullWritable;
-import org.apache.hadoop.mapreduce.OutputFormat;
-import org.apache.hadoop.mapreduce.RecordWriter;
-import org.apache.hadoop.mapreduce.TaskAttemptContext;
-import org.apache.tinkerpop.gremlin.giraph.process.computer.GiraphVertex;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
-
-import java.io.IOException;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public final class GiraphVertexWriter extends VertexWriter {
- private final OutputFormat<NullWritable, VertexWritable> outputFormat;
- private RecordWriter<NullWritable, VertexWritable> recordWriter;
-
- public GiraphVertexWriter(final OutputFormat<NullWritable, VertexWritable> outputFormat) {
- this.outputFormat = outputFormat;
- }
-
- @Override
- public void initialize(final TaskAttemptContext context) throws IOException, InterruptedException {
- this.recordWriter = this.outputFormat.getRecordWriter(context);
- }
-
- @Override
- public void close(final TaskAttemptContext context) throws IOException, InterruptedException {
- this.recordWriter.close(context);
- }
-
- @Override
- public void writeVertex(final Vertex vertex) throws IOException, InterruptedException {
- this.recordWriter.write(NullWritable.get(), ((GiraphVertex) vertex).getValue());
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexInputFormat.java
----------------------------------------------------------------------
diff --git a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexInputFormat.java b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexInputFormat.java
new file mode 100644
index 0000000..6b6638a
--- /dev/null
+++ b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexInputFormat.java
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.giraph.structure.io;
+
+import org.apache.giraph.io.VertexInputFormat;
+import org.apache.giraph.io.VertexReader;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.mapreduce.InputFormat;
+import org.apache.hadoop.mapreduce.InputSplit;
+import org.apache.hadoop.mapreduce.JobContext;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
+import org.apache.hadoop.util.ReflectionUtils;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+
+import java.io.IOException;
+import java.util.List;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public final class GiraphVertexInputFormat extends VertexInputFormat {
+
+ private InputFormat<NullWritable, VertexWritable> hadoopGraphInputFormat;
+
+ @Override
+ public void checkInputSpecs(final Configuration configuration) {
+
+ }
+
+ @Override
+ public List<InputSplit> getSplits(final JobContext context, final int minSplitCountHint) throws IOException, InterruptedException {
+ this.constructor(context.getConfiguration());
+ return this.hadoopGraphInputFormat.getSplits(context);
+ }
+
+ @Override
+ public VertexReader createVertexReader(final InputSplit split, final TaskAttemptContext context) throws IOException {
+ this.constructor(context.getConfiguration());
+ try {
+ return new GiraphVertexReader(this.hadoopGraphInputFormat.createRecordReader(split, context));
+ } catch (InterruptedException e) {
+ throw new IOException(e);
+ }
+ }
+
+ private final void constructor(final Configuration configuration) {
+ if (null == this.hadoopGraphInputFormat) {
+ this.hadoopGraphInputFormat = ReflectionUtils.newInstance(configuration.getClass(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, InputFormat.class, InputFormat.class), configuration);
+ }
+ }
+
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexOutputFormat.java
----------------------------------------------------------------------
diff --git a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexOutputFormat.java b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexOutputFormat.java
new file mode 100644
index 0000000..2d9b4ba
--- /dev/null
+++ b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexOutputFormat.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.giraph.structure.io;
+
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+import org.apache.giraph.io.VertexOutputFormat;
+import org.apache.giraph.io.VertexWriter;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.mapreduce.JobContext;
+import org.apache.hadoop.mapreduce.OutputCommitter;
+import org.apache.hadoop.mapreduce.OutputFormat;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
+import org.apache.hadoop.util.ReflectionUtils;
+
+import java.io.IOException;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public final class GiraphVertexOutputFormat extends VertexOutputFormat {
+
+ private OutputFormat<NullWritable, VertexWritable> hadoopGraphOutputFormat;
+
+ @Override
+ public VertexWriter createVertexWriter(final TaskAttemptContext context) throws IOException, InterruptedException {
+ this.constructor(context.getConfiguration());
+ return new GiraphVertexWriter(this.hadoopGraphOutputFormat);
+ }
+
+ @Override
+ public void checkOutputSpecs(final JobContext context) throws IOException, InterruptedException {
+ this.constructor(context.getConfiguration());
+ this.hadoopGraphOutputFormat.checkOutputSpecs(context);
+ }
+
+ @Override
+ public OutputCommitter getOutputCommitter(final TaskAttemptContext context) throws IOException, InterruptedException {
+ this.constructor(context.getConfiguration());
+ return this.hadoopGraphOutputFormat.getOutputCommitter(context);
+ }
+
+ private final void constructor(final Configuration configuration) {
+ if (null == this.hadoopGraphOutputFormat) {
+ this.hadoopGraphOutputFormat = ReflectionUtils.newInstance(configuration.getClass(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT, OutputFormat.class, OutputFormat.class), configuration);
+ }
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexReader.java
----------------------------------------------------------------------
diff --git a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexReader.java b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexReader.java
new file mode 100644
index 0000000..2a6ade6
--- /dev/null
+++ b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexReader.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.giraph.structure.io;
+
+import org.apache.giraph.graph.Vertex;
+import org.apache.giraph.io.VertexReader;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.mapreduce.InputSplit;
+import org.apache.hadoop.mapreduce.RecordReader;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
+import org.apache.tinkerpop.gremlin.giraph.process.computer.GiraphVertex;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+
+import java.io.IOException;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public final class GiraphVertexReader extends VertexReader {
+
+ private RecordReader<NullWritable, VertexWritable> recordReader;
+
+ public GiraphVertexReader(final RecordReader<NullWritable, VertexWritable> recordReader) {
+ this.recordReader = recordReader;
+ }
+
+ @Override
+ public void initialize(final InputSplit inputSplit, final TaskAttemptContext context) throws IOException, InterruptedException {
+ this.recordReader.initialize(inputSplit, context);
+ }
+
+ @Override
+ public boolean nextVertex() throws IOException, InterruptedException {
+ return this.recordReader.nextKeyValue();
+ }
+
+ @Override
+ public Vertex getCurrentVertex() throws IOException, InterruptedException {
+ return new GiraphVertex(this.recordReader.getCurrentValue());
+ }
+
+ @Override
+ public void close() throws IOException {
+ this.recordReader.close();
+ }
+
+ @Override
+ public float getProgress() throws IOException, InterruptedException {
+ return this.recordReader.getProgress();
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexWriter.java
----------------------------------------------------------------------
diff --git a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexWriter.java b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexWriter.java
new file mode 100644
index 0000000..43c4b0b
--- /dev/null
+++ b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/structure/io/GiraphVertexWriter.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.giraph.structure.io;
+
+import org.apache.giraph.graph.Vertex;
+import org.apache.giraph.io.VertexWriter;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.mapreduce.OutputFormat;
+import org.apache.hadoop.mapreduce.RecordWriter;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
+import org.apache.tinkerpop.gremlin.giraph.process.computer.GiraphVertex;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+
+import java.io.IOException;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public final class GiraphVertexWriter extends VertexWriter {
+ private final OutputFormat<NullWritable, VertexWritable> outputFormat;
+ private RecordWriter<NullWritable, VertexWritable> recordWriter;
+
+ public GiraphVertexWriter(final OutputFormat<NullWritable, VertexWritable> outputFormat) {
+ this.outputFormat = outputFormat;
+ }
+
+ @Override
+ public void initialize(final TaskAttemptContext context) throws IOException, InterruptedException {
+ this.recordWriter = this.outputFormat.getRecordWriter(context);
+ }
+
+ @Override
+ public void close(final TaskAttemptContext context) throws IOException, InterruptedException {
+ this.recordWriter.close(context);
+ }
+
+ @Override
+ public void writeVertex(final Vertex vertex) throws IOException, InterruptedException {
+ this.recordWriter.write(NullWritable.get(), ((GiraphVertex) vertex).getValue());
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/InputOutputHelper.java
----------------------------------------------------------------------
diff --git a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/InputOutputHelper.java b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/InputOutputHelper.java
index 8a80053..04097c1 100644
--- a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/InputOutputHelper.java
+++ b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/io/InputOutputHelper.java
@@ -18,15 +18,21 @@
*/
package org.apache.tinkerpop.gremlin.hadoop.structure.io;
+import org.apache.commons.configuration.BaseConfiguration;
+import org.apache.commons.configuration.Configuration;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.mapreduce.InputFormat;
import org.apache.hadoop.mapreduce.OutputFormat;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration;
+import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.graphson.GraphSONInputFormat;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.graphson.GraphSONOutputFormat;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoInputFormat;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.gryo.GryoOutputFormat;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.script.ScriptInputFormat;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.script.ScriptOutputFormat;
+import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
@@ -65,4 +71,20 @@ public final class InputOutputHelper {
INPUT_TO_OUTPUT_CACHE.put(inputFormat, outputFormat);
OUTPUT_TO_INPUT_CACHE.put(outputFormat, inputFormat);
}
+
+ public static HadoopGraph getOutputGraph(final Configuration configuration, final GraphComputer.ResultGraph resultGraph, final GraphComputer.Persist persist) {
+ final HadoopConfiguration hadoopConfiguration = new HadoopConfiguration(configuration);
+ final BaseConfiguration newConfiguration = new BaseConfiguration();
+ newConfiguration.copy(hadoopConfiguration);
+ if (resultGraph.equals(GraphComputer.ResultGraph.NEW)) {
+ newConfiguration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, hadoopConfiguration.getOutputLocation() + "/" + Constants.HIDDEN_G);
+ if (hadoopConfiguration.containsKey(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT))
+ newConfiguration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, InputOutputHelper.getInputFormat(hadoopConfiguration.getGraphOutputFormat()).getCanonicalName());
+ newConfiguration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, hadoopConfiguration.getOutputLocation() + "_");
+ newConfiguration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT_HAS_EDGES, persist.equals(GraphComputer.Persist.EDGES));
+ } else {
+ newConfiguration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, hadoopConfiguration.getOutputLocation() + "_");
+ }
+ return HadoopGraph.open(newConfiguration);
+ }
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/util/HadoopHelper.java
----------------------------------------------------------------------
diff --git a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/util/HadoopHelper.java b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/util/HadoopHelper.java
deleted file mode 100644
index ebfdf91..0000000
--- a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/structure/util/HadoopHelper.java
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.hadoop.structure.util;
-
-import org.apache.commons.configuration.BaseConfiguration;
-import org.apache.tinkerpop.gremlin.hadoop.Constants;
-import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.InputOutputHelper;
-import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public final class HadoopHelper {
-
- private HadoopHelper() {
- }
-
- public static HadoopGraph getOutputGraph(final HadoopGraph hadoopGraph, final GraphComputer.ResultGraph resultGraph, final GraphComputer.Persist persist) {
- final BaseConfiguration newConfiguration = new BaseConfiguration();
- newConfiguration.copy(hadoopGraph.configuration());
- if (resultGraph.equals(GraphComputer.ResultGraph.NEW)) {
- newConfiguration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, hadoopGraph.configuration().getOutputLocation() + "/" + Constants.HIDDEN_G);
- if (hadoopGraph.configuration().containsKey(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT))
- newConfiguration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, InputOutputHelper.getInputFormat(hadoopGraph.configuration().getGraphOutputFormat()).getCanonicalName());
- newConfiguration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, hadoopGraph.configuration().getOutputLocation() + "_");
- newConfiguration.setProperty(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT_HAS_EDGES, persist.equals(GraphComputer.Persist.EDGES));
- } else {
- newConfiguration.setProperty(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION, hadoopGraph.configuration().getOutputLocation() + "_");
- }
- return HadoopGraph.open(newConfiguration);
- }
-
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
index 0656f5a..b25073b 100644
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/SparkGraphComputer.java
@@ -37,7 +37,6 @@ import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration;
import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
import org.apache.tinkerpop.gremlin.hadoop.structure.util.ConfUtil;
-import org.apache.tinkerpop.gremlin.hadoop.structure.util.HadoopHelper;
import org.apache.tinkerpop.gremlin.process.computer.ComputerResult;
import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
import org.apache.tinkerpop.gremlin.process.computer.MapReduce;
@@ -45,11 +44,12 @@ import org.apache.tinkerpop.gremlin.process.computer.Memory;
import org.apache.tinkerpop.gremlin.process.computer.VertexProgram;
import org.apache.tinkerpop.gremlin.process.computer.util.DefaultComputerResult;
import org.apache.tinkerpop.gremlin.process.computer.util.MapMemory;
-import org.apache.tinkerpop.gremlin.spark.process.computer.io.InputFormatRDD;
-import org.apache.tinkerpop.gremlin.spark.process.computer.io.InputRDD;
-import org.apache.tinkerpop.gremlin.spark.process.computer.io.OutputFormatRDD;
-import org.apache.tinkerpop.gremlin.spark.process.computer.io.OutputRDD;
import org.apache.tinkerpop.gremlin.spark.process.computer.payload.ViewIncomingPayload;
+import org.apache.tinkerpop.gremlin.spark.structure.io.InputFormatRDD;
+import org.apache.tinkerpop.gremlin.spark.structure.io.InputOutputHelper;
+import org.apache.tinkerpop.gremlin.spark.structure.io.InputRDD;
+import org.apache.tinkerpop.gremlin.spark.structure.io.OutputFormatRDD;
+import org.apache.tinkerpop.gremlin.spark.structure.io.OutputRDD;
import java.io.File;
import java.io.IOException;
@@ -209,6 +209,7 @@ public final class SparkGraphComputer extends AbstractHadoopGraphComputer {
// write the map reduce output back to disk (memory)
SparkExecutor.saveMapReduceRDD(null == reduceRDD ? mapRDD : reduceRDD, mapReduce, finalMemory, hadoopConfiguration);
}
+ mapReduceGraphRDD.unpersist();
}
// unpersist the graphRDD if it will no longer be used
@@ -217,9 +218,9 @@ public final class SparkGraphComputer extends AbstractHadoopGraphComputer {
}
// update runtime and return the newly computed graph
finalMemory.setRuntime(System.currentTimeMillis() - startTime);
- return new DefaultComputerResult(HadoopHelper.getOutputGraph(this.hadoopGraph, this.resultGraph, this.persist), finalMemory.asImmutable());
+ return new DefaultComputerResult(InputOutputHelper.getOutputGraph(apacheConfiguration, this.resultGraph, this.persist), finalMemory.asImmutable());
} finally {
- if (sparkContext != null && !this.hadoopGraph.configuration().getBoolean(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, false))
+ if (sparkContext != null && !apacheConfiguration.getBoolean(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, false))
sparkContext.stop();
}
});
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputFormatRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputFormatRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputFormatRDD.java
deleted file mode 100644
index adb080b..0000000
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputFormatRDD.java
+++ /dev/null
@@ -1,47 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.tinkerpop.gremlin.spark.process.computer.io;
-
-import org.apache.commons.configuration.Configuration;
-import org.apache.hadoop.io.NullWritable;
-import org.apache.hadoop.mapreduce.InputFormat;
-import org.apache.spark.api.java.JavaPairRDD;
-import org.apache.spark.api.java.JavaSparkContext;
-import org.apache.tinkerpop.gremlin.hadoop.Constants;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
-import org.apache.tinkerpop.gremlin.hadoop.structure.util.ConfUtil;
-import scala.Tuple2;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public final class InputFormatRDD implements InputRDD {
-
- @Override
- public JavaPairRDD<Object, VertexWritable> readGraphRDD(final Configuration configuration, final JavaSparkContext sparkContext) {
- final org.apache.hadoop.conf.Configuration hadoopConfiguration = ConfUtil.makeHadoopConfiguration(configuration);
- return sparkContext.newAPIHadoopRDD(hadoopConfiguration,
- (Class<InputFormat<NullWritable, VertexWritable>>) hadoopConfiguration.getClass(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, InputFormat.class),
- NullWritable.class,
- VertexWritable.class)
- .mapToPair(tuple -> new Tuple2<>(tuple._2().get().id(), new VertexWritable(tuple._2().get())))
- .reduceByKey((a, b) -> a); // if this is not done, then the graph is partitioned and you can have duplicate vertices
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputRDD.java
deleted file mode 100644
index 291fcd3..0000000
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/InputRDD.java
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.spark.process.computer.io;
-
-import org.apache.commons.configuration.Configuration;
-import org.apache.spark.api.java.JavaPairRDD;
-import org.apache.spark.api.java.JavaSparkContext;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
-
-/**
- * An InputRDD is used to read data from the underlying graph system and yield the respective adjacency list.
- * Note that {@link InputFormatRDD} is a type of InputRDD that simply uses the specified {@link org.apache.hadoop.mapreduce.InputFormat} to generate the respective graphRDD.
- *
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public interface InputRDD {
-
- /**
- * Read the graphRDD from the underlying graph system.
- * @param configuration the configuration for the {@link org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer}.
- * @param sparkContext the Spark context with the requisite methods for generating a {@link JavaPairRDD}.
- * @return an adjacency list representation of the underlying graph system.
- */
- public JavaPairRDD<Object, VertexWritable> readGraphRDD(final Configuration configuration, final JavaSparkContext sparkContext);
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputFormatRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputFormatRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputFormatRDD.java
deleted file mode 100644
index 56a1297..0000000
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputFormatRDD.java
+++ /dev/null
@@ -1,49 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.tinkerpop.gremlin.spark.process.computer.io;
-
-import org.apache.commons.configuration.Configuration;
-import org.apache.hadoop.io.NullWritable;
-import org.apache.hadoop.mapreduce.OutputFormat;
-import org.apache.spark.api.java.JavaPairRDD;
-import org.apache.tinkerpop.gremlin.hadoop.Constants;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
-import org.apache.tinkerpop.gremlin.hadoop.structure.util.ConfUtil;
-import scala.Tuple2;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public final class OutputFormatRDD implements OutputRDD {
-
- @Override
- public void writeGraphRDD(final Configuration configuration, final JavaPairRDD<Object, VertexWritable> graphRDD) {
- final org.apache.hadoop.conf.Configuration hadoopConfiguration = ConfUtil.makeHadoopConfiguration(configuration);
- final String outputLocation = hadoopConfiguration.get(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION);
- if (null != outputLocation) {
- // map back to a <nullwritable,vertexwritable> stream for output
- graphRDD.mapToPair(tuple -> new Tuple2<>(NullWritable.get(), tuple._2()))
- .saveAsNewAPIHadoopFile(outputLocation + "/" + Constants.HIDDEN_G,
- NullWritable.class,
- VertexWritable.class,
- (Class<OutputFormat<NullWritable, VertexWritable>>) hadoopConfiguration.getClass(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT, OutputFormat.class), hadoopConfiguration);
- }
- }
-}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputRDD.java
deleted file mode 100644
index 2580252..0000000
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/OutputRDD.java
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.spark.process.computer.io;
-
-import org.apache.commons.configuration.Configuration;
-import org.apache.spark.api.java.JavaPairRDD;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public interface OutputRDD {
-
- public void writeGraphRDD(final Configuration configuration, final JavaPairRDD<Object, VertexWritable> graphRDD);
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputRDD.java
deleted file mode 100644
index ad521b3..0000000
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedInputRDD.java
+++ /dev/null
@@ -1,60 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.tinkerpop.gremlin.spark.process.computer.io;
-
-import org.apache.commons.configuration.Configuration;
-import org.apache.spark.api.java.JavaPairRDD;
-import org.apache.spark.api.java.JavaRDD;
-import org.apache.spark.api.java.JavaSparkContext;
-import org.apache.spark.rdd.RDD;
-import org.apache.tinkerpop.gremlin.hadoop.Constants;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
-import scala.Tuple2;
-import scala.collection.Iterator;
-
-import java.util.Optional;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public final class PersistedInputRDD implements InputRDD {
-
- @Override
- public JavaPairRDD<Object, VertexWritable> readGraphRDD(final Configuration configuration, final JavaSparkContext sparkContext) {
- final String inputRDDName = configuration.getString(Constants.GREMLIN_HADOOP_INPUT_LOCATION, null);
- if (null == inputRDDName)
- throw new IllegalArgumentException(PersistedInputRDD.class.getSimpleName() + " requires " + Constants.GREMLIN_HADOOP_INPUT_LOCATION + " in order to retrieve the named graphRDD from the SparkContext");
- if (!PersistedInputRDD.getPersistedRDD(sparkContext, inputRDDName).isPresent())
- throw new IllegalArgumentException("The provided graphRDD name is not in the persisted RDDs of the SparkContext: " + inputRDDName);
- return JavaPairRDD.fromJavaRDD((JavaRDD) PersistedInputRDD.getPersistedRDD(sparkContext, inputRDDName).get().toJavaRDD());
- }
-
- public static Optional<RDD<?>> getPersistedRDD(final JavaSparkContext sparkContext, final String rddName) {
- final Iterator<Tuple2<Object, RDD<?>>> iterator = JavaSparkContext.toSparkContext(sparkContext).
- getPersistentRDDs().
- toList().iterator();
- while (iterator.hasNext()) {
- final Tuple2<Object, RDD<?>> tuple2 = iterator.next();
- if (tuple2._2().toString().contains(rddName))
- return Optional.of(tuple2._2());
- }
- return Optional.empty();
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedOutputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedOutputRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedOutputRDD.java
deleted file mode 100644
index 1832ca3..0000000
--- a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/PersistedOutputRDD.java
+++ /dev/null
@@ -1,41 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-
-package org.apache.tinkerpop.gremlin.spark.process.computer.io;
-
-import org.apache.commons.configuration.Configuration;
-import org.apache.spark.api.java.JavaPairRDD;
-import org.apache.tinkerpop.gremlin.hadoop.Constants;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public final class PersistedOutputRDD implements OutputRDD {
-
- private static final Logger LOGGER = LoggerFactory.getLogger(PersistedOutputRDD.class);
-
- @Override
- public void writeGraphRDD(final Configuration configuration, final JavaPairRDD<Object, VertexWritable> graphRDD) {
- if (!configuration.getBoolean(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, false))
- LOGGER.warn("The SparkContext should be persisted in order for the RDD to persist across jobs. To do so, set " + Constants.GREMLIN_SPARK_PERSIST_CONTEXT + " to true");
- }
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputFormatRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputFormatRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputFormatRDD.java
new file mode 100644
index 0000000..f768939
--- /dev/null
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputFormatRDD.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tinkerpop.gremlin.spark.structure.io;
+
+import org.apache.commons.configuration.Configuration;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.mapreduce.InputFormat;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+import org.apache.tinkerpop.gremlin.hadoop.structure.util.ConfUtil;
+import scala.Tuple2;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public final class InputFormatRDD implements InputRDD {
+
+ @Override
+ public JavaPairRDD<Object, VertexWritable> readGraphRDD(final Configuration configuration, final JavaSparkContext sparkContext) {
+ final org.apache.hadoop.conf.Configuration hadoopConfiguration = ConfUtil.makeHadoopConfiguration(configuration);
+ return sparkContext.newAPIHadoopRDD(hadoopConfiguration,
+ (Class<InputFormat<NullWritable, VertexWritable>>) hadoopConfiguration.getClass(Constants.GREMLIN_HADOOP_GRAPH_INPUT_FORMAT, InputFormat.class),
+ NullWritable.class,
+ VertexWritable.class)
+ .mapToPair(tuple -> new Tuple2<>(tuple._2().get().id(), new VertexWritable(tuple._2().get())))
+ .reduceByKey((a, b) -> a); // if this is not done, then the graph is partitioned and you can have duplicate vertices
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputHelper.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputHelper.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputHelper.java
new file mode 100644
index 0000000..b1e1b0a
--- /dev/null
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputOutputHelper.java
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tinkerpop.gremlin.spark.structure.io;
+
+import org.apache.commons.configuration.BaseConfiguration;
+import org.apache.commons.configuration.Configuration;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopConfiguration;
+import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
+import org.apache.tinkerpop.gremlin.process.computer.GraphComputer;
+
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public final class InputOutputHelper {
+
+ private static Map<Class<? extends InputRDD>, Class<? extends OutputRDD>> INPUT_TO_OUTPUT_CACHE = new ConcurrentHashMap<>();
+ private static Map<Class<? extends OutputRDD>, Class<? extends InputRDD>> OUTPUT_TO_INPUT_CACHE = new ConcurrentHashMap<>();
+
+ static {
+ INPUT_TO_OUTPUT_CACHE.put(PersistedInputRDD.class, PersistedOutputRDD.class);
+ INPUT_TO_OUTPUT_CACHE.put(InputFormatRDD.class, OutputFormatRDD.class);
+ //
+ OUTPUT_TO_INPUT_CACHE.put(PersistedOutputRDD.class, PersistedInputRDD.class);
+ OUTPUT_TO_INPUT_CACHE.put(OutputFormatRDD.class, InputFormatRDD.class);
+ }
+
+ private InputOutputHelper() {
+
+ }
+
+ public static Class<? extends InputRDD> getInputFormat(final Class<? extends OutputRDD> outputRDD) {
+ return OUTPUT_TO_INPUT_CACHE.get(outputRDD);
+ }
+
+ public static Class<? extends OutputRDD> getOutputFormat(final Class<? extends InputRDD> inputRDD) {
+ return INPUT_TO_OUTPUT_CACHE.get(inputRDD);
+ }
+
+ public static void registerInputOutputPair(final Class<? extends InputRDD> inputRDD, final Class<? extends OutputRDD> outputRDD) {
+ INPUT_TO_OUTPUT_CACHE.put(inputRDD, outputRDD);
+ OUTPUT_TO_INPUT_CACHE.put(outputRDD, inputRDD);
+ }
+
+ public static HadoopGraph getOutputGraph(final Configuration configuration, final GraphComputer.ResultGraph resultGraph, final GraphComputer.Persist persist) {
+ try {
+ final HadoopConfiguration hadoopConfiguration = new HadoopConfiguration(configuration);
+ final BaseConfiguration newConfiguration = new BaseConfiguration();
+ newConfiguration.copy(org.apache.tinkerpop.gremlin.hadoop.structure.io.InputOutputHelper.getOutputGraph(configuration, resultGraph, persist).configuration());
+ if (resultGraph.equals(GraphComputer.ResultGraph.NEW) && hadoopConfiguration.containsKey(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD)) {
+ newConfiguration.setProperty(Constants.GREMLIN_SPARK_GRAPH_INPUT_RDD, InputOutputHelper.getInputFormat((Class) Class.forName(hadoopConfiguration.getProperty(Constants.GREMLIN_SPARK_GRAPH_OUTPUT_RDD).toString())).getCanonicalName());
+ if (newConfiguration.getString(Constants.GREMLIN_HADOOP_INPUT_LOCATION, "").endsWith("/" + Constants.HIDDEN_G)) { // Spark RDDs are not namespaced the same as Hadoop
+ newConfiguration.setProperty(Constants.GREMLIN_HADOOP_INPUT_LOCATION, newConfiguration.getString(Constants.GREMLIN_HADOOP_INPUT_LOCATION).replace("/" + Constants.HIDDEN_G, ""));
+ }
+ }
+ return HadoopGraph.open(newConfiguration);
+ } catch (final ClassNotFoundException e) {
+ throw new IllegalArgumentException(e.getMessage(), e);
+ }
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDD.java
new file mode 100644
index 0000000..c84c189
--- /dev/null
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/InputRDD.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.spark.structure.io;
+
+import org.apache.commons.configuration.Configuration;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+
+/**
+ * An InputRDD is used to read data from the underlying graph system and yield the respective adjacency list.
+ * Note that {@link InputFormatRDD} is a type of InputRDD that simply uses the specified {@link org.apache.hadoop.mapreduce.InputFormat} to generate the respective graphRDD.
+ *
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public interface InputRDD {
+
+ /**
+ * Read the graphRDD from the underlying graph system.
+ * @param configuration the configuration for the {@link org.apache.tinkerpop.gremlin.spark.process.computer.SparkGraphComputer}.
+ * @param sparkContext the Spark context with the requisite methods for generating a {@link JavaPairRDD}.
+ * @return an adjacency list representation of the underlying graph system.
+ */
+ public JavaPairRDD<Object, VertexWritable> readGraphRDD(final Configuration configuration, final JavaSparkContext sparkContext);
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputFormatRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputFormatRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputFormatRDD.java
new file mode 100644
index 0000000..cc6ed61
--- /dev/null
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputFormatRDD.java
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tinkerpop.gremlin.spark.structure.io;
+
+import org.apache.commons.configuration.Configuration;
+import org.apache.hadoop.io.NullWritable;
+import org.apache.hadoop.mapreduce.OutputFormat;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+import org.apache.tinkerpop.gremlin.hadoop.structure.util.ConfUtil;
+import scala.Tuple2;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public final class OutputFormatRDD implements OutputRDD {
+
+ @Override
+ public void writeGraphRDD(final Configuration configuration, final JavaPairRDD<Object, VertexWritable> graphRDD) {
+ final org.apache.hadoop.conf.Configuration hadoopConfiguration = ConfUtil.makeHadoopConfiguration(configuration);
+ final String outputLocation = hadoopConfiguration.get(Constants.GREMLIN_HADOOP_OUTPUT_LOCATION);
+ if (null != outputLocation) {
+ // map back to a <nullwritable,vertexwritable> stream for output
+ graphRDD.mapToPair(tuple -> new Tuple2<>(NullWritable.get(), tuple._2()))
+ .saveAsNewAPIHadoopFile(outputLocation + "/" + Constants.HIDDEN_G,
+ NullWritable.class,
+ VertexWritable.class,
+ (Class<OutputFormat<NullWritable, VertexWritable>>) hadoopConfiguration.getClass(Constants.GREMLIN_HADOOP_GRAPH_OUTPUT_FORMAT, OutputFormat.class), hadoopConfiguration);
+ }
+ }
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputRDD.java
new file mode 100644
index 0000000..c2964eb
--- /dev/null
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/OutputRDD.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.spark.structure.io;
+
+import org.apache.commons.configuration.Configuration;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public interface OutputRDD {
+
+ public void writeGraphRDD(final Configuration configuration, final JavaPairRDD<Object, VertexWritable> graphRDD);
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedInputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedInputRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedInputRDD.java
new file mode 100644
index 0000000..e3c27ec
--- /dev/null
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedInputRDD.java
@@ -0,0 +1,60 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tinkerpop.gremlin.spark.structure.io;
+
+import org.apache.commons.configuration.Configuration;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.spark.api.java.JavaRDD;
+import org.apache.spark.api.java.JavaSparkContext;
+import org.apache.spark.rdd.RDD;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+import scala.Tuple2;
+import scala.collection.Iterator;
+
+import java.util.Optional;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public final class PersistedInputRDD implements InputRDD {
+
+ @Override
+ public JavaPairRDD<Object, VertexWritable> readGraphRDD(final Configuration configuration, final JavaSparkContext sparkContext) {
+ final String inputRDDName = configuration.getString(Constants.GREMLIN_HADOOP_INPUT_LOCATION, null);
+ if (null == inputRDDName)
+ throw new IllegalArgumentException(PersistedInputRDD.class.getSimpleName() + " requires " + Constants.GREMLIN_HADOOP_INPUT_LOCATION + " in order to retrieve the named graphRDD from the SparkContext");
+ if (!PersistedInputRDD.getPersistedRDD(sparkContext, inputRDDName).isPresent())
+ throw new IllegalArgumentException("The provided graphRDD name is not in the persisted RDDs of the SparkContext: " + inputRDDName);
+ return JavaPairRDD.fromJavaRDD((JavaRDD) PersistedInputRDD.getPersistedRDD(sparkContext, inputRDDName).get().toJavaRDD());
+ }
+
+ public static Optional<RDD<?>> getPersistedRDD(final JavaSparkContext sparkContext, final String rddName) {
+ final Iterator<Tuple2<Object, RDD<?>>> iterator = JavaSparkContext.toSparkContext(sparkContext).
+ getPersistentRDDs().
+ toList().iterator();
+ while (iterator.hasNext()) {
+ final Tuple2<Object, RDD<?>> tuple2 = iterator.next();
+ if (tuple2._2().toString().contains(rddName))
+ return Optional.of(tuple2._2());
+ }
+ return Optional.empty();
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedOutputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedOutputRDD.java b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedOutputRDD.java
new file mode 100644
index 0000000..abeeaaa
--- /dev/null
+++ b/spark-gremlin/src/main/java/org/apache/tinkerpop/gremlin/spark/structure/io/PersistedOutputRDD.java
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.tinkerpop.gremlin.spark.structure.io;
+
+import org.apache.commons.configuration.Configuration;
+import org.apache.spark.api.java.JavaPairRDD;
+import org.apache.tinkerpop.gremlin.hadoop.Constants;
+import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+public final class PersistedOutputRDD implements OutputRDD {
+
+ private static final Logger LOGGER = LoggerFactory.getLogger(PersistedOutputRDD.class);
+
+ @Override
+ public void writeGraphRDD(final Configuration configuration, final JavaPairRDD<Object, VertexWritable> graphRDD) {
+ if (!configuration.getBoolean(Constants.GREMLIN_SPARK_PERSIST_CONTEXT, false))
+ LOGGER.warn("The SparkContext should be persisted in order for the RDD to persist across jobs. To do so, set " + Constants.GREMLIN_SPARK_PERSIST_CONTEXT + " to true");
+ }
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/ExampleInputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/ExampleInputRDD.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/ExampleInputRDD.java
deleted file mode 100644
index 1fb85a1..0000000
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/ExampleInputRDD.java
+++ /dev/null
@@ -1,47 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.spark.process.computer.io;
-
-import org.apache.commons.configuration.Configuration;
-import org.apache.spark.api.java.JavaPairRDD;
-import org.apache.spark.api.java.JavaSparkContext;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
-import org.apache.tinkerpop.gremlin.structure.T;
-import org.apache.tinkerpop.gremlin.structure.Vertex;
-import org.apache.tinkerpop.gremlin.structure.util.star.StarGraph;
-import scala.Tuple2;
-
-import java.util.ArrayList;
-import java.util.List;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public final class ExampleInputRDD implements InputRDD {
-
- @Override
- public JavaPairRDD<Object, VertexWritable> readGraphRDD(final Configuration configuration, final JavaSparkContext sparkContext) {
- final List<Vertex> list = new ArrayList<>();
- list.add(StarGraph.open().addVertex(T.id, 1l, T.label,"person","age", 29));
- list.add(StarGraph.open().addVertex(T.id, 2l, T.label,"person","age", 27));
- list.add(StarGraph.open().addVertex(T.id, 4l, T.label,"person","age", 32));
- list.add(StarGraph.open().addVertex(T.id, 6l, T.label,"person","age", 35));
- return sparkContext.parallelize(list).mapToPair(vertex -> new Tuple2<>(vertex.id(), new VertexWritable(vertex)));
- }
-}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/cb972387/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/ExampleOutputRDD.java
----------------------------------------------------------------------
diff --git a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/ExampleOutputRDD.java b/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/ExampleOutputRDD.java
deleted file mode 100644
index bb38f7f..0000000
--- a/spark-gremlin/src/test/java/org/apache/tinkerpop/gremlin/spark/process/computer/io/ExampleOutputRDD.java
+++ /dev/null
@@ -1,45 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.spark.process.computer.io;
-
-import org.apache.commons.configuration.Configuration;
-import org.apache.spark.api.java.JavaPairRDD;
-import org.apache.tinkerpop.gremlin.hadoop.structure.io.VertexWritable;
-import org.apache.tinkerpop.gremlin.structure.Vertex;
-
-import java.util.Iterator;
-
-import static org.junit.Assert.assertEquals;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-public final class ExampleOutputRDD implements OutputRDD {
- @Override
- public void writeGraphRDD(final Configuration configuration, final JavaPairRDD<Object, VertexWritable> graphRDD) {
- int totalAge = 0;
- final Iterator<VertexWritable> iterator = graphRDD.values().toLocalIterator();
- while (iterator.hasNext()) {
- final Vertex vertex = iterator.next().get();
- if (vertex.label().equals("person"))
- totalAge = totalAge + vertex.<Integer>value("age");
- }
- assertEquals(123, totalAge);
- }
-}
[28/50] [abbrv] incubator-tinkerpop git commit: integrated
NumnerHelper in MeanLocalStep
Posted by sp...@apache.org.
integrated NumnerHelper in MeanLocalStep
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/baf2ae04
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/baf2ae04
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/baf2ae04
Branch: refs/heads/TINKERPOP3-923
Commit: baf2ae04c32b1497bb72ab69f9264f793c085280
Parents: dfa61c9
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Tue Nov 3 15:51:43 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Tue Nov 3 15:51:43 2015 +0100
----------------------------------------------------------------------
.../process/traversal/step/map/MeanLocalStep.java | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/baf2ae04/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanLocalStep.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanLocalStep.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanLocalStep.java
index e2db6af..fab03bb 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanLocalStep.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MeanLocalStep.java
@@ -18,6 +18,7 @@
*/
package org.apache.tinkerpop.gremlin.process.traversal.step.map;
+import org.apache.tinkerpop.gremlin.process.traversal.NumberHelper;
import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
import org.apache.tinkerpop.gremlin.process.traversal.Traverser;
import org.apache.tinkerpop.gremlin.process.traversal.traverser.TraverserRequirement;
@@ -30,23 +31,23 @@ import java.util.Set;
* @author Marko A. Rodriguez (http://markorodriguez.com)
* @author Daniel Kuppitz (http://gremlin.guru)
*/
-public final class MeanLocalStep<E extends Number, S extends Iterable<E>> extends MapStep<S, Double> {
+public final class MeanLocalStep<E extends Number, S extends Iterable<E>> extends MapStep<S, Number> {
public MeanLocalStep(final Traversal.Admin traversal) {
super(traversal);
}
@Override
- protected Double map(final Traverser.Admin<S> traverser) {
+ protected Number map(final Traverser.Admin<S> traverser) {
final Iterator<E> iterator = traverser.get().iterator();
if (iterator.hasNext()) {
Long counter = 1L;
- Double result = iterator.next().doubleValue();
+ E result = iterator.next();
while (iterator.hasNext()) {
- result += iterator.next().doubleValue();
+ result = (E) NumberHelper.add(result, iterator.next());
counter++;
}
- return result / counter;
+ return NumberHelper.div(result, counter, true);
} else {
return Double.NaN;
}
[13/50] [abbrv] incubator-tinkerpop git commit: a small tweak to
maintain the configuration as it was prior to Giraph getting its little hands
on it.
Posted by sp...@apache.org.
a small tweak to maintain the configuration as it was prior to Giraph getting its little hands on it.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/f9d1fa72
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/f9d1fa72
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/f9d1fa72
Branch: refs/heads/TINKERPOP3-923
Commit: f9d1fa726dd82d2ebee770d9d321425c91a6e268
Parents: cf0c456
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Fri Oct 30 18:26:24 2015 -0600
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Fri Oct 30 18:26:24 2015 -0600
----------------------------------------------------------------------
.../gremlin/giraph/process/computer/GiraphGraphComputer.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/f9d1fa72/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
----------------------------------------------------------------------
diff --git a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
index 0389750..67e8ea3 100644
--- a/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
+++ b/giraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/giraph/process/computer/GiraphGraphComputer.java
@@ -114,6 +114,7 @@ public final class GiraphGraphComputer extends AbstractHadoopGraphComputer imple
public Future<ComputerResult> submit() {
final long startTime = System.currentTimeMillis();
super.validateStatePriorToExecution();
+ final Configuration apacheConfiguration = ConfUtil.makeApacheConfiguration(this.giraphConfiguration);
return CompletableFuture.<ComputerResult>supplyAsync(() -> {
try {
final FileSystem fs = FileSystem.get(this.giraphConfiguration);
@@ -126,7 +127,7 @@ public final class GiraphGraphComputer extends AbstractHadoopGraphComputer imple
}
this.memory.setRuntime(System.currentTimeMillis() - startTime);
- return new DefaultComputerResult(InputOutputHelper.getOutputGraph(ConfUtil.makeApacheConfiguration(this.giraphConfiguration), this.resultGraph, this.persist), this.memory.asImmutable());
+ return new DefaultComputerResult(InputOutputHelper.getOutputGraph(apacheConfiguration, this.resultGraph, this.persist), this.memory.asImmutable());
});
}
[35/50] [abbrv] incubator-tinkerpop git commit: clean up tweaks to
MatchStep optimization.
Posted by sp...@apache.org.
clean up tweaks to MatchStep optimization.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/f644fb42
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/f644fb42
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/f644fb42
Branch: refs/heads/TINKERPOP3-923
Commit: f644fb42b9f5e0f9216deb3f5e3088293e3ca13a
Parents: 956d797
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Tue Nov 3 10:05:25 2015 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Tue Nov 3 10:05:25 2015 -0700
----------------------------------------------------------------------
CHANGELOG.asciidoc | 2 ++
.../process/traversal/step/map/MatchStep.java | 16 +++++++---------
.../process/traversal/step/map/MatchStepTest.java | 2 +-
3 files changed, 10 insertions(+), 10 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/f644fb42/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index e79a0a0..e4eb554 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -25,6 +25,8 @@ image::https://raw.githubusercontent.com/apache/incubator-tinkerpop/master/docs/
TinkerPop 3.1.0 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* `CountMatchAlgorithm`, in OLAP, now biases traversal selection towards those traversals that start at the current traverser location to reduce message passing.
+* Fixed a file stream bug in Hadoop OLTP that showed up if the streamed file was more than 2G of data.
* Bumped to Neo4j 2.3.0.
* Added `PersistedInputRDD` and `PersistedOutputRDD` which enables `SparkGraphComputer` to store the graph RDD in the context between jobs (no HDFS serialization required).
* Renamed the `public static String` configuration variable names of TinkerGraph (deprecated old variables).
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/f644fb42/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStep.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStep.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStep.java
index 1ed202f..38da656 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStep.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStep.java
@@ -67,7 +67,6 @@ public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>>
public enum TraversalType {WHERE_PREDICATE, WHERE_TRAVERSAL, MATCH_TRAVERSAL}
- private TraversalEngine.Type traversalEngineType;
private List<Traversal.Admin<Object, Object>> matchTraversals = new ArrayList<>();
private boolean first = true;
private Set<String> matchStartLabels = new HashSet<>();
@@ -155,12 +154,6 @@ public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>>
}
}
- @Override
- public void onEngine(final TraversalEngine engine) {
- super.onEngine(engine);
- this.traversalEngineType = engine.getType();
- }
-
public ConnectiveStep.Connective getConnective() {
return this.connective;
}
@@ -213,6 +206,8 @@ public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>>
}
public MatchAlgorithm getMatchAlgorithm() {
+ if(null == this.matchAlgorithm)
+ this.initializeMatchAlgorithm(this.traverserStepIdAndLabelsSetByChild ? TraversalEngine.Type.COMPUTER : TraversalEngine.Type.STANDARD);
return this.matchAlgorithm;
}
@@ -224,7 +219,6 @@ public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>>
clone.matchTraversals.add(clone.integrateChild(traversal.clone()));
}
if (this.dedups != null) clone.dedups = new HashSet<>();
- clone.initializeMatchAlgorithm(this.traversalEngineType);
return clone;
}
@@ -304,7 +298,7 @@ public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>>
Traverser.Admin traverser = null;
if (this.first) {
this.first = false;
- this.initializeMatchAlgorithm(this.traversalEngineType);
+ this.initializeMatchAlgorithm(TraversalEngine.Type.STANDARD);
} else {
for (final Traversal.Admin<?, ?> matchTraversal : this.matchTraversals) {
if (matchTraversal.hasNext()) {
@@ -339,6 +333,10 @@ public final class MatchStep<S, E> extends ComputerAwareStep<S, Map<String, E>>
@Override
protected Iterator<Traverser<Map<String, E>>> computerAlgorithm() throws NoSuchElementException {
while (true) {
+ if (this.first) {
+ this.first = false;
+ this.initializeMatchAlgorithm(TraversalEngine.Type.COMPUTER);
+ }
final Traverser.Admin traverser = this.starts.next();
final Path path = traverser.path();
if (!this.matchStartLabels.stream().filter(path::hasLabel).findAny().isPresent())
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/f644fb42/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStepTest.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStepTest.java b/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStepTest.java
index 529e717..4f7c231 100644
--- a/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStepTest.java
+++ b/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/MatchStepTest.java
@@ -314,7 +314,7 @@ public class MatchStepTest extends StepTest {
@Test
public void testComputerAwareCountMatchAlgorithm() {
- // MAKE SURE THE SORT ORDER CHANGES AS MORE RESULTS ARE RETURNED BY ONE OR THE OTHER TRAVERSAL
+ // MAKE SURE OLAP JOBS ARE BIASED TOWARDS STAR GRAPH DATA
final Consumer doNothing = s -> {
};
Traversal.Admin<?, ?> traversal = __.match(
[39/50] [abbrv] incubator-tinkerpop git commit: removed empty code
line
Posted by sp...@apache.org.
removed empty code line
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/f2b90833
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/f2b90833
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/f2b90833
Branch: refs/heads/TINKERPOP3-923
Commit: f2b90833398a20b67f18fc106300986a6b229027
Parents: a2b94c4
Author: Daniel Kuppitz <da...@hotmail.com>
Authored: Wed Nov 4 14:58:13 2015 +0100
Committer: Daniel Kuppitz <da...@hotmail.com>
Committed: Wed Nov 4 14:58:13 2015 +0100
----------------------------------------------------------------------
.../process/computer/bulkloading/BulkLoaderVertexProgram.java | 1 -
1 file changed, 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/f2b90833/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgram.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgram.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgram.java
index 2950bfd..cb36cf4 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgram.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/computer/bulkloading/BulkLoaderVertexProgram.java
@@ -424,7 +424,6 @@ public class BulkLoaderVertexProgram implements VertexProgram<Tuple> {
public BulkLoadingListener() {
this.counter = 0L;
this.isNewVertex = false;
- ;
}
public boolean isNewVertex() {
[05/50] [abbrv] incubator-tinkerpop git commit: The mid-traversal V()
work did not include a test case for testing hasContainer fold-ins for
TinkerGraph nor Neo4jGraph (@dkuppitz). I just added a test. Both pass. Also,
the XXXStratgegySuite classes were
Posted by sp...@apache.org.
The mid-traversal V() work did not include a test case for testing hasContainer fold-ins for TinkerGraph nor Neo4jGraph (@dkuppitz). I just added a test. Both pass. Also, the XXXStratgegySuite classes were in the wrong packages -- reorganized (@dkuppitz). Updated CHANGELOG with note about relocation of GraphStep package.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/9300485e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/9300485e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/9300485e
Branch: refs/heads/TINKERPOP3-923
Commit: 9300485eb3f1a8e4cbf63133e038d64fe6c52372
Parents: ac0924d
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Thu Oct 29 17:04:44 2015 -0600
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Thu Oct 29 17:04:44 2015 -0600
----------------------------------------------------------------------
CHANGELOG.asciidoc | 1 +
.../upgrade-release-3.1.x-incubating.asciidoc | 5 +++
.../gremlin/neo4j/structure/Neo4jGraph.java | 2 +-
.../gremlin/neo4j/Neo4jStrategySuite.java | 44 --------------------
.../gremlin/neo4j/Neo4jStrategyTest.java | 31 --------------
.../traversal/strategy/Neo4jStrategySuite.java | 44 ++++++++++++++++++++
.../traversal/strategy/Neo4jStrategyTest.java | 32 ++++++++++++++
.../Neo4jGraphStepStrategyTest.java | 13 ++++++
.../tinkergraph/structure/TinkerGraph.java | 2 +-
.../tinkergraph/TinkerGraphStrategySuite.java | 44 --------------------
.../tinkergraph/TinkerGraphStrategyTest.java | 31 --------------
.../strategy/TinkerGraphStrategySuite.java | 44 ++++++++++++++++++++
.../strategy/TinkerGraphStrategyTest.java | 32 ++++++++++++++
.../TinkerGraphStepStrategyTest.java | 13 ++++++
14 files changed, 186 insertions(+), 152 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/9300485e/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index d773c02..8bd6e51 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -27,6 +27,7 @@ TinkerPop 3.1.0 (NOT OFFICIALLY RELEASED YET)
* Renamed the `public static String` configuration variable names of TinkerGraph (deprecated old variables).
* Added `GraphComputer.config(key,value)` to allow engine-specific configurations.
+* `GraphStep` is no longer in the `sideEffect`-package and is now in `map`-package (breaking change).
* Added suppport for mid-traversal `V()`-steps (`GraphStep` semantics updated).
* Fixed `Number` handling in `Operator` enums. Prior this change a lot of operations on mixed `Number` types returned a wrong result (wrong data type).
* Fixed a bug in Gremlin Server/Driver serializer where empty buffers were getting returned in certain cases.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/9300485e/docs/src/upgrade-release-3.1.x-incubating.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/upgrade-release-3.1.x-incubating.asciidoc b/docs/src/upgrade-release-3.1.x-incubating.asciidoc
index 1e52cb5..b50dd7e 100644
--- a/docs/src/upgrade-release-3.1.x-incubating.asciidoc
+++ b/docs/src/upgrade-release-3.1.x-incubating.asciidoc
@@ -35,6 +35,7 @@ Additional upgrade information can be found here:
Important Changes
~~~~~~~~~~~~~~~~~
+* `GraphStep` is no longer in `sideEffect`-package, but now in `map`-package as traversals support mid-traversal `V()`.
* Traversals now support mid-traversal `V()`-steps. Graph system providers should ensure that a mid-traversal `V()` can leverage any suitable index.
* The data type of `Operator` enums will now always be the highest common data type of the two given numbers, rather than the data type of the first number, as it's been before.
* The Gephi Plugin has improved integration with Gephi, where manually inserting `store` steps to visualize a running traversal is no longer required.
@@ -144,6 +145,10 @@ Graph Traversal Updates
There were numerous changes to the `GraphTraversal` API. Nearly all changes are backwards compatible with respective
"deprecated" annotations. Please review the respective updates specified in the "Graph System Users" section.
+* `GraphStep` is no longer in `sideEffect` package. Now in `map` package.
+* Make sure mid-traversal `GraphStep` calls are folding `HasContainers` in for index-lookups.
+* Think about copying `TinkerGraphStepStrategyTest` for your implementation so you know folding is happening correctly.
+
Element Removal
+++++++++++++++
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/9300485e/neo4j-gremlin/src/main/java/org/apache/tinkerpop/gremlin/neo4j/structure/Neo4jGraph.java
----------------------------------------------------------------------
diff --git a/neo4j-gremlin/src/main/java/org/apache/tinkerpop/gremlin/neo4j/structure/Neo4jGraph.java b/neo4j-gremlin/src/main/java/org/apache/tinkerpop/gremlin/neo4j/structure/Neo4jGraph.java
index 7616dc6..010e592 100644
--- a/neo4j-gremlin/src/main/java/org/apache/tinkerpop/gremlin/neo4j/structure/Neo4jGraph.java
+++ b/neo4j-gremlin/src/main/java/org/apache/tinkerpop/gremlin/neo4j/structure/Neo4jGraph.java
@@ -71,7 +71,7 @@ import java.util.stream.Stream;
@Graph.OptIn(Graph.OptIn.SUITE_GROOVY_ENVIRONMENT_INTEGRATE)
@Graph.OptIn(Graph.OptIn.SUITE_GROOVY_ENVIRONMENT_PERFORMANCE)
@Graph.OptIn("org.apache.tinkerpop.gremlin.neo4j.NativeNeo4jSuite")
-@Graph.OptIn("org.apache.tinkerpop.gremlin.neo4j.Neo4jStrategySuite")
+@Graph.OptIn("org.apache.tinkerpop.gremlin.neo4j.process.traversal.strategy.Neo4jStrategySuite")
public final class Neo4jGraph implements Graph, WrappedGraph<Neo4jGraphAPI> {
public static final Logger LOGGER = LoggerFactory.getLogger(Neo4jGraph.class);
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/9300485e/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/Neo4jStrategySuite.java
----------------------------------------------------------------------
diff --git a/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/Neo4jStrategySuite.java b/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/Neo4jStrategySuite.java
deleted file mode 100644
index 7c56d0d..0000000
--- a/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/Neo4jStrategySuite.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.neo4j;
-
-import org.apache.tinkerpop.gremlin.AbstractGremlinSuite;
-import org.apache.tinkerpop.gremlin.neo4j.process.traversal.strategy.optimization.Neo4jGraphStepStrategyTest;
-import org.apache.tinkerpop.gremlin.process.traversal.TraversalEngine;
-import org.junit.runners.model.InitializationError;
-import org.junit.runners.model.RunnerBuilder;
-
-/**
- * @author Daniel Kuppitz (http://gremlin.guru)
- */
-public class Neo4jStrategySuite extends AbstractGremlinSuite {
-
- public Neo4jStrategySuite(final Class<?> klass, final RunnerBuilder builder) throws InitializationError {
-
- super(klass, builder,
- new Class<?>[]{
- Neo4jGraphStepStrategyTest.class
- }, new Class<?>[]{
- Neo4jGraphStepStrategyTest.class
- },
- false,
- TraversalEngine.Type.STANDARD);
- }
-
-}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/9300485e/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/Neo4jStrategyTest.java
----------------------------------------------------------------------
diff --git a/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/Neo4jStrategyTest.java b/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/Neo4jStrategyTest.java
deleted file mode 100644
index d0bdc0f..0000000
--- a/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/Neo4jStrategyTest.java
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.neo4j;
-
-import org.apache.tinkerpop.gremlin.GraphProviderClass;
-import org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jGraph;
-import org.junit.runner.RunWith;
-
-/**
- * @author Daniel Kuppitz (http://gremlin.guru)
- */
-@RunWith(Neo4jStrategySuite.class)
-@GraphProviderClass(provider = NoMultiNoMetaNeo4jGraphProvider.class, graph = Neo4jGraph.class)
-public class Neo4jStrategyTest {
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/9300485e/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/Neo4jStrategySuite.java
----------------------------------------------------------------------
diff --git a/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/Neo4jStrategySuite.java b/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/Neo4jStrategySuite.java
new file mode 100644
index 0000000..0004bc2
--- /dev/null
+++ b/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/Neo4jStrategySuite.java
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.neo4j.process.traversal.strategy;
+
+import org.apache.tinkerpop.gremlin.AbstractGremlinSuite;
+import org.apache.tinkerpop.gremlin.neo4j.process.traversal.strategy.optimization.Neo4jGraphStepStrategyTest;
+import org.apache.tinkerpop.gremlin.process.traversal.TraversalEngine;
+import org.junit.runners.model.InitializationError;
+import org.junit.runners.model.RunnerBuilder;
+
+/**
+ * @author Daniel Kuppitz (http://gremlin.guru)
+ */
+public class Neo4jStrategySuite extends AbstractGremlinSuite {
+
+ public Neo4jStrategySuite(final Class<?> klass, final RunnerBuilder builder) throws InitializationError {
+
+ super(klass, builder,
+ new Class<?>[]{
+ Neo4jGraphStepStrategyTest.class
+ }, new Class<?>[]{
+ Neo4jGraphStepStrategyTest.class
+ },
+ false,
+ TraversalEngine.Type.STANDARD);
+ }
+
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/9300485e/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/Neo4jStrategyTest.java
----------------------------------------------------------------------
diff --git a/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/Neo4jStrategyTest.java b/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/Neo4jStrategyTest.java
new file mode 100644
index 0000000..5363e1f
--- /dev/null
+++ b/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/Neo4jStrategyTest.java
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.neo4j.process.traversal.strategy;
+
+import org.apache.tinkerpop.gremlin.GraphProviderClass;
+import org.apache.tinkerpop.gremlin.neo4j.NoMultiNoMetaNeo4jGraphProvider;
+import org.apache.tinkerpop.gremlin.neo4j.structure.Neo4jGraph;
+import org.junit.runner.RunWith;
+
+/**
+ * @author Daniel Kuppitz (http://gremlin.guru)
+ */
+@RunWith(Neo4jStrategySuite.class)
+@GraphProviderClass(provider = NoMultiNoMetaNeo4jGraphProvider.class, graph = Neo4jGraph.class)
+public class Neo4jStrategyTest {
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/9300485e/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/optimization/Neo4jGraphStepStrategyTest.java
----------------------------------------------------------------------
diff --git a/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/optimization/Neo4jGraphStepStrategyTest.java b/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/optimization/Neo4jGraphStepStrategyTest.java
index cc92ef9..0611e23 100644
--- a/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/optimization/Neo4jGraphStepStrategyTest.java
+++ b/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/optimization/Neo4jGraphStepStrategyTest.java
@@ -59,5 +59,18 @@ public class Neo4jGraphStepStrategyTest extends AbstractNeo4jGremlinTest {
assertEquals("name", ((Neo4jGraphStep<?, ?>) traversal.getStartStep()).getHasContainers().get(0).getKey());
assertEquals("marko", ((Neo4jGraphStep<?, ?>) traversal.getStartStep()).getHasContainers().get(0).getValue());
assertEquals(HasStep.class, traversal.getEndStep().getClass());
+ ////
+ traversal = g.V().has("name", "marko").out().V().has("name", "daniel").asAdmin();
+ traversal.applyStrategies();
+ assertEquals(3, traversal.getSteps().size());
+ assertEquals(Neo4jGraphStep.class, traversal.getStartStep().getClass());
+ assertEquals(1, ((Neo4jGraphStep) traversal.getStartStep()).getHasContainers().size());
+ assertEquals("name", ((Neo4jGraphStep<?, ?>) traversal.getStartStep()).getHasContainers().get(0).getKey());
+ assertEquals("marko", ((Neo4jGraphStep<?, ?>) traversal.getStartStep()).getHasContainers().get(0).getValue());
+ assertEquals(Neo4jGraphStep.class, traversal.getSteps().get(2).getClass());
+ assertEquals(1, ((Neo4jGraphStep) traversal.getSteps().get(2)).getHasContainers().size());
+ assertEquals("name", ((Neo4jGraphStep<?, ?>) traversal.getSteps().get(2)).getHasContainers().get(0).getKey());
+ assertEquals("daniel", ((Neo4jGraphStep<?,?>) traversal.getSteps().get(2)).getHasContainers().get(0).getValue());
+ assertEquals(Neo4jGraphStep.class, traversal.getEndStep().getClass());
}
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/9300485e/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java
----------------------------------------------------------------------
diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java
index 4ca542c..9fc1d3e 100644
--- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java
+++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java
@@ -67,7 +67,7 @@ import java.util.stream.Stream;
@Graph.OptIn(Graph.OptIn.SUITE_GROOVY_ENVIRONMENT)
@Graph.OptIn(Graph.OptIn.SUITE_GROOVY_ENVIRONMENT_INTEGRATE)
@Graph.OptIn(Graph.OptIn.SUITE_GROOVY_ENVIRONMENT_PERFORMANCE)
-@Graph.OptIn("org.apache.tinkerpop.gremlin.tinkergraph.TinkerGraphStrategySuite")
+@Graph.OptIn("org.apache.tinkerpop.gremlin.tinkergraph.process.traversal.strategy.TinkerGraphStrategySuite")
public final class TinkerGraph implements Graph {
static {
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/9300485e/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerGraphStrategySuite.java
----------------------------------------------------------------------
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerGraphStrategySuite.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerGraphStrategySuite.java
deleted file mode 100644
index faffe9a..0000000
--- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerGraphStrategySuite.java
+++ /dev/null
@@ -1,44 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.tinkergraph;
-
-import org.apache.tinkerpop.gremlin.AbstractGremlinSuite;
-import org.apache.tinkerpop.gremlin.process.traversal.TraversalEngine;
-import org.apache.tinkerpop.gremlin.tinkergraph.process.traversal.strategy.optimization.TinkerGraphStepStrategyTest;
-import org.junit.runners.model.InitializationError;
-import org.junit.runners.model.RunnerBuilder;
-
-/**
- * @author Daniel Kuppitz (http://gremlin.guru)
- */
-public class TinkerGraphStrategySuite extends AbstractGremlinSuite {
-
- public TinkerGraphStrategySuite(final Class<?> klass, final RunnerBuilder builder) throws InitializationError {
-
- super(klass, builder,
- new Class<?>[]{
- TinkerGraphStepStrategyTest.class
- }, new Class<?>[]{
- TinkerGraphStepStrategyTest.class
- },
- false,
- TraversalEngine.Type.STANDARD);
- }
-
-}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/9300485e/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerGraphStrategyTest.java
----------------------------------------------------------------------
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerGraphStrategyTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerGraphStrategyTest.java
deleted file mode 100644
index d355d4e..0000000
--- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/TinkerGraphStrategyTest.java
+++ /dev/null
@@ -1,31 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing,
- * software distributed under the License is distributed on an
- * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * KIND, either express or implied. See the License for the
- * specific language governing permissions and limitations
- * under the License.
- */
-package org.apache.tinkerpop.gremlin.tinkergraph;
-
-import org.apache.tinkerpop.gremlin.GraphProviderClass;
-import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
-import org.junit.runner.RunWith;
-
-/**
- * @author Daniel Kuppitz (http://gremlin.guru)
- */
-@RunWith(TinkerGraphStrategySuite.class)
-@GraphProviderClass(provider = TinkerGraphProvider.class, graph = TinkerGraph.class)
-public class TinkerGraphStrategyTest {
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/9300485e/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/TinkerGraphStrategySuite.java
----------------------------------------------------------------------
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/TinkerGraphStrategySuite.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/TinkerGraphStrategySuite.java
new file mode 100644
index 0000000..3982918
--- /dev/null
+++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/TinkerGraphStrategySuite.java
@@ -0,0 +1,44 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.tinkergraph.process.traversal.strategy;
+
+import org.apache.tinkerpop.gremlin.AbstractGremlinSuite;
+import org.apache.tinkerpop.gremlin.process.traversal.TraversalEngine;
+import org.apache.tinkerpop.gremlin.tinkergraph.process.traversal.strategy.optimization.TinkerGraphStepStrategyTest;
+import org.junit.runners.model.InitializationError;
+import org.junit.runners.model.RunnerBuilder;
+
+/**
+ * @author Daniel Kuppitz (http://gremlin.guru)
+ */
+public class TinkerGraphStrategySuite extends AbstractGremlinSuite {
+
+ public TinkerGraphStrategySuite(final Class<?> klass, final RunnerBuilder builder) throws InitializationError {
+
+ super(klass, builder,
+ new Class<?>[]{
+ TinkerGraphStepStrategyTest.class
+ }, new Class<?>[]{
+ TinkerGraphStepStrategyTest.class
+ },
+ false,
+ TraversalEngine.Type.STANDARD);
+ }
+
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/9300485e/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/TinkerGraphStrategyTest.java
----------------------------------------------------------------------
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/TinkerGraphStrategyTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/TinkerGraphStrategyTest.java
new file mode 100644
index 0000000..7942626
--- /dev/null
+++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/TinkerGraphStrategyTest.java
@@ -0,0 +1,32 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.tinkerpop.gremlin.tinkergraph.process.traversal.strategy;
+
+import org.apache.tinkerpop.gremlin.GraphProviderClass;
+import org.apache.tinkerpop.gremlin.tinkergraph.TinkerGraphProvider;
+import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
+import org.junit.runner.RunWith;
+
+/**
+ * @author Daniel Kuppitz (http://gremlin.guru)
+ */
+@RunWith(TinkerGraphStrategySuite.class)
+@GraphProviderClass(provider = TinkerGraphProvider.class, graph = TinkerGraph.class)
+public class TinkerGraphStrategyTest {
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/9300485e/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/optimization/TinkerGraphStepStrategyTest.java
----------------------------------------------------------------------
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/optimization/TinkerGraphStepStrategyTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/optimization/TinkerGraphStepStrategyTest.java
index e497808..6e4850d 100644
--- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/optimization/TinkerGraphStepStrategyTest.java
+++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/optimization/TinkerGraphStepStrategyTest.java
@@ -64,6 +64,19 @@ public class TinkerGraphStepStrategyTest extends AbstractGremlinProcessTest {
assertEquals("name", ((TinkerGraphStep<?, ?>) traversal.getStartStep()).getHasContainers().get(0).getKey());
assertEquals("marko", ((TinkerGraphStep<?, ?>) traversal.getStartStep()).getHasContainers().get(0).getValue());
assertEquals(HasStep.class, traversal.getEndStep().getClass());
+ ////
+ traversal = g.V().has("name", "marko").out().V().has("name", "daniel").asAdmin();
+ traversal.applyStrategies();
+ assertEquals(3, traversal.getSteps().size());
+ assertEquals(TinkerGraphStep.class, traversal.getStartStep().getClass());
+ assertEquals(1, ((TinkerGraphStep) traversal.getStartStep()).getHasContainers().size());
+ assertEquals("name", ((TinkerGraphStep<?, ?>) traversal.getStartStep()).getHasContainers().get(0).getKey());
+ assertEquals("marko", ((TinkerGraphStep<?, ?>) traversal.getStartStep()).getHasContainers().get(0).getValue());
+ assertEquals(TinkerGraphStep.class, traversal.getSteps().get(2).getClass());
+ assertEquals(1, ((TinkerGraphStep) traversal.getSteps().get(2)).getHasContainers().size());
+ assertEquals("name", ((TinkerGraphStep<?, ?>) traversal.getSteps().get(2)).getHasContainers().get(0).getKey());
+ assertEquals("daniel", ((TinkerGraphStep<?,?>) traversal.getSteps().get(2)).getHasContainers().get(0).getValue());
+ assertEquals(TinkerGraphStep.class, traversal.getEndStep().getClass());
}
}