You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by ok...@apache.org on 2015/05/22 19:05:27 UTC
incubator-tinkerpop git commit: HadoopGremlinPlugin now supports
SparkGraphComputer. Super good integration tests on the HadoopGremlinPlugin.
Docs updated. Very good stuff here.
Repository: incubator-tinkerpop
Updated Branches:
refs/heads/master 1d3257371 -> 34da329f4
HadoopGremlinPlugin now supports SparkGraphComputer. Super good integration tests on the HadoopGremlinPlugin. Docs updated. Very good stuff here.
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/34da329f
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/34da329f
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/34da329f
Branch: refs/heads/master
Commit: 34da329f41bb56c08bb69108e0694baffccb7e3f
Parents: 1d32573
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Fri May 22 11:05:25 2015 -0600
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Fri May 22 11:05:35 2015 -0600
----------------------------------------------------------------------
docs/src/implementations.asciidoc | 18 ++-
.../traversal/util/TraversalScriptFunction.java | 8 +-
hadoop-gremlin/pom.xml | 14 ++
.../groovy/plugin/HadoopGremlinPlugin.java | 4 +-
.../groovy/plugin/HadoopRemoteAcceptor.java | 53 +++++---
.../groovy/plugin/HadoopGremlinPluginTest.java | 130 ++++++++-----------
...hGraphComputerGroovyPluginIntegrateTest.java | 36 -----
.../GiraphHadoopGremlinPluginIntegrateTest.java | 36 +++++
.../groovy/SparkHadoopGremlinPluginTest.java | 36 +++++
9 files changed, 196 insertions(+), 139 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/34da329f/docs/src/implementations.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/implementations.asciidoc b/docs/src/implementations.asciidoc
index f6311a5..57d0e7a 100644
--- a/docs/src/implementations.asciidoc
+++ b/docs/src/implementations.asciidoc
@@ -661,7 +661,7 @@ GiraphGraphComputer
image:giraph-logo.png[width=100,float=left] link:http://giraph.apache.org[Giraph] is an Apache Software Foundation project focused on OLAP-based graph processing. Giraph makes use of the distributed graph computing paradigm made popular by Google's Pregel. In Giraph, developers write "vertex programs" that get executed at each vertex in parallel. These programs communicate with one another in a bulk synchronous parallel (BSP) manner. This model aligns with TinkerPop3's `GraphComputer` API. TinkerPop3 provides an implementation of `GraphComputer` that works for Giraph called `GiraphGraphComputer`. Moreover, with TinkerPop3's <<mapreduce,MapReduce>>-framework, the standard Giraph/Pregel model is extended to support an arbitrary number of MapReduce phases to aggregate and yield results from the graph. Below are examples using `GiraphGraphComputer` from the <<gremlin-console,Gremlin-Console>>.
-IMPORTANT: Be sure that the link:http://www.slf4j.org/[SLF4J] of Hadoop matches that of Giraph or else there will be conflicts. Simply copy the following jars to the `lib/` of the machines in the Hadoop cluster: `slf4j-api-a.b.c.jar` and `slf4j-log4j12-a.b.c.jar`.
+WARNING: Be sure that the link:http://www.slf4j.org/[SLF4J] of Hadoop matches that of Giraph or else there will be conflicts. Simply copy the following jars to the `lib/` of the machines in the Hadoop cluster: `slf4j-api-a.b.c.jar` and `slf4j-log4j12-a.b.c.jar`.
WARNING: Giraph uses a large number of Hadoop counters. The default for Hadoop is 120. In `mapred-site.xml` it is possible to increase the limit it via the `mapreduce.job.counters.limit` property. A good value to use is 1000. This is a cluster-wide property so be sure to restart the cluster after updating.
@@ -698,7 +698,8 @@ IMPORTANT: The examples above do not use lambdas (i.e. closures in Gremlin-Groov
gremlin> graph = GraphFactory.open('../../../hadoop-gremlin/conf/hadoop-gryo.properties')
==>hadoopgraph[gryoinputformat->gryooutputformat]
gremlin> :remote connect tinkerpop.hadoop graph
-==>hadoopgraph[gryoinputformat->gryooutputformat]
+==>useTraversalSource=graphtraversalsource[hadoopgraph[gryoinputformat->gryooutputformat], giraphgraphcomputer]
+==>useSugar=false
gremlin> :> g.V().group().by{it.value('name')[1]}.by('name')
INFO org.apache.tinkerpop.gremlin.hadoop.process.computer.giraph.GiraphGraphComputer - HadoopGremlin(Giraph): TraversalVertexProgram[GraphStep(vertex), GroupStep(lambda,value(name)), ComputerResultStep]
INFO org.apache.hadoop.mapred.JobClient - Running job: job_201407281259_0039
@@ -736,7 +737,18 @@ gremlin> g.V().out().out().values('name')
==>lop
==>ripple
-CAUTION: The HadoopRemoteAcceptor (`:remote`) currently does not support `SparkGraphComputer`. As such, submitting lambda containing traversals to the Spark cluster is not possible via the Gremlin Console.
+For using lambdas in Gremlin-Groovy, simply provide `:remote connect` a `TraversalSource` which leverages SparkGraphComputer.
+
+[source,test]
+gremlin> graph = GraphFactory.open('../../../hadoop-gremlin/conf/hadoop-gryo.properties')
+==>hadoopgraph[gryoinputformat->gryooutputformat]
+gremlin> g = graph.traversal(computer(SparkGraphComputer))
+==>graphtraversalsource[hadoopgraph[gryoinputformat->gryooutputformat], sparkgraphcomputer]
+gremlin> :remote connect tinkerpop.hadoop graph g
+==>useTraversalSource=graphtraversalsource[hadoopgraph[gryoinputformat->gryooutputformat], sparkgraphcomputer]
+==>useSugar=false
+gremlin> :> g.V().group().by{it.value('name')[1]}.by('name')
+==>[a:[marko, vadas], e:[peter], i:[ripple], o:[lop, josh]]
The `SparkGraphComputer` algorithm leverages Spark's caching abilities to reduce the amount of data shuffled across the wire on each iteration of the <<vertexprogram,`VertexProgram`>>. When the graph is loaded as a Spark RDD (Resilient Distributed Dataset) it is immediately cached as `graphRDD`. The `graphRDD` is a distributed adjacency list which encodes the vertex, its properties, and all its incident edges. On the first iteration, each vertex (in parallel) is passed through `VertexProgram.execute()`. This yields an output of the vertex's mutated state (i.e. updated compute keys -- `propertyX`) and its outgoing messages. This `viewOutgoingRDD` is then reduced to `viewIncomingRDD` where the outgoing messages are sent to their respective vertices. If a `MessageCombiner` exists for the vertex program, then messages are aggregated locally and globally to ultimately yield one incoming message for the vertex. This reduce sequence is the "message pass." If the vertex program does not ter
minate on this iteration, then the `viewIncomingRDD` is joined with the cached `graphRDD` and the process continues. When there are no more iterations, there is a final join and the resultant RDD is stripped of its edges and messages. This `mapReduceRDD` is cached and is processed by each <<mapreduce,`MapReduce`>> job in the <<graphcomputer,`GraphComputer`>> computation.
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/34da329f/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/util/TraversalScriptFunction.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/util/TraversalScriptFunction.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/util/TraversalScriptFunction.java
index 3579a27..35e9991 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/util/TraversalScriptFunction.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/util/TraversalScriptFunction.java
@@ -34,13 +34,13 @@ import java.util.function.Function;
*/
public final class TraversalScriptFunction<S, E> implements Function<Graph, Traversal.Admin<S, E>>, Serializable {
- private final TraversalSource.Builder traversalContextBuilder;
+ private final TraversalSource.Builder traversalSourceBuilder;
private final String scriptEngineName;
private final String traversalScript;
private final Object[] bindings;
- public TraversalScriptFunction(final TraversalSource.Builder traversalContextBuilder, final String scriptEngineName, final String traversalScript, final Object... bindings) {
- this.traversalContextBuilder = traversalContextBuilder;
+ public TraversalScriptFunction(final TraversalSource.Builder traversalSourceBuilder, final String scriptEngineName, final String traversalScript, final Object... bindings) {
+ this.traversalSourceBuilder = traversalSourceBuilder;
this.scriptEngineName = scriptEngineName;
this.traversalScript = traversalScript;
this.bindings = bindings;
@@ -52,7 +52,7 @@ public final class TraversalScriptFunction<S, E> implements Function<Graph, Trav
try {
final ScriptEngine engine = ScriptEngineCache.get(this.scriptEngineName);
final Bindings engineBindings = engine.createBindings();
- engineBindings.put("g", this.traversalContextBuilder.create(graph));
+ engineBindings.put("g", this.traversalSourceBuilder.create(graph));
for (int i = 0; i < this.bindings.length; i = i + 2) {
engineBindings.put((String) this.bindings[i], this.bindings[i + 1]);
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/34da329f/hadoop-gremlin/pom.xml
----------------------------------------------------------------------
diff --git a/hadoop-gremlin/pom.xml b/hadoop-gremlin/pom.xml
index 1bb9c4d..fb1fa22 100644
--- a/hadoop-gremlin/pom.xml
+++ b/hadoop-gremlin/pom.xml
@@ -289,6 +289,20 @@ limitations under the License.
</archive>
</configuration>
</plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-surefire-plugin</artifactId>
+ <configuration>
+ <!-- log4j.configuration>log4j-core.properties</log4j.configuration> -->
+ <!--<argLine>-Xmx2048M</argLine>-->
+ <excludes>
+ <exclude>**/*IntegrateTest.java</exclude>
+ <exclude>**/*PerformanceTest.java</exclude>
+ <!-- this is technically a member of the integration test suite -->
+ <exclude>**/HadoopGremlinPluginTest.java</exclude>
+ </excludes>
+ </configuration>
+ </plugin>
</plugins>
</build>
</project>
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/34da329f/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java
----------------------------------------------------------------------
diff --git a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java
index 8ce5b62..a34153e 100644
--- a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java
+++ b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPlugin.java
@@ -49,6 +49,8 @@ import java.util.Set;
*/
public final class HadoopGremlinPlugin extends AbstractGremlinPlugin {
+ protected static String NAME = "tinkerpop.hadoop";
+
protected static final Set<String> IMPORTS = new HashSet<String>() {{
add("import org.apache.hadoop.hdfs.*");
add("import org.apache.hadoop.conf.*");
@@ -79,7 +81,7 @@ public final class HadoopGremlinPlugin extends AbstractGremlinPlugin {
@Override
public String getName() {
- return "tinkerpop.hadoop";
+ return NAME;
}
@Override
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/34da329f/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopRemoteAcceptor.java
----------------------------------------------------------------------
diff --git a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopRemoteAcceptor.java b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopRemoteAcceptor.java
index 4499cc2..aa6e9f4 100644
--- a/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopRemoteAcceptor.java
+++ b/hadoop-gremlin/src/main/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopRemoteAcceptor.java
@@ -26,11 +26,14 @@ import org.apache.tinkerpop.gremlin.process.computer.ComputerResult;
import org.apache.tinkerpop.gremlin.process.computer.traversal.TraversalVertexProgram;
import org.apache.tinkerpop.gremlin.process.computer.traversal.step.map.ComputerResultStep;
import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
+import org.apache.tinkerpop.gremlin.process.traversal.TraversalSource;
import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.GraphTraversalSource;
import org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal;
import org.codehaus.groovy.tools.shell.Groovysh;
import java.io.IOException;
+import java.util.Collections;
+import java.util.HashMap;
import java.util.List;
/**
@@ -39,11 +42,14 @@ import java.util.List;
public final class HadoopRemoteAcceptor implements RemoteAcceptor {
private static final String USE_SUGAR = "useSugar";
+ private static final String USE_TRAVERSAL_SOURCE = "useTraversalSource";
private static final String SPACE = " ";
private HadoopGraph hadoopGraph;
private Groovysh shell;
- private boolean useSugarPlugin = false;
+ private boolean useSugar = false;
+ private TraversalSource.Builder useTraversalSource = GraphTraversalSource.computer();
+ private TraversalSource traversalSource;
public HadoopRemoteAcceptor(final Groovysh shell) {
this.shell = shell;
@@ -51,35 +57,48 @@ public final class HadoopRemoteAcceptor implements RemoteAcceptor {
@Override
public Object connect(final List<String> args) throws RemoteException {
- if (args.size() != 1) {
- throw new IllegalArgumentException("The variable name of the graph object must be provided");
+ if (args.size() != 1 && args.size() != 2) {
+ throw new IllegalArgumentException("Usage: :remote connect " + HadoopGremlinPlugin.NAME + " <variable name of graph> <optional variable name of traversal source builder>");
}
this.hadoopGraph = (HadoopGraph) this.shell.getInterp().getContext().getVariable(args.get(0));
- return this.hadoopGraph;
+ if (args.size() == 2)
+ this.useTraversalSource = ((TraversalSource) this.shell.getInterp().getContext().getVariable(args.get(1))).asBuilder();
+ this.traversalSource = this.useTraversalSource.create(this.hadoopGraph);
+ ///
+ final HashMap<String, Object> configuration = new HashMap<>();
+ configuration.put(USE_SUGAR, this.useSugar);
+ configuration.put(USE_TRAVERSAL_SOURCE, this.traversalSource);
+ return Collections.unmodifiableMap(configuration);
}
@Override
public Object configure(final List<String> args) throws RemoteException {
for (int i = 0; i < args.size(); i = i + 2) {
if (args.get(i).equals(USE_SUGAR))
- this.useSugarPlugin = Boolean.valueOf(args.get(i + 1));
- else
- throw new IllegalArgumentException("The provided configuration is unknown: " + args);
+ this.useSugar = Boolean.valueOf(args.get(i + 1));
+ else if (args.get(i).equals(USE_TRAVERSAL_SOURCE)) {
+ this.useTraversalSource = ((TraversalSource) this.shell.getInterp().getContext().getVariable(args.get(i + 1))).asBuilder();
+ this.traversalSource = this.useTraversalSource.create(this.hadoopGraph);
+ } else
+ throw new IllegalArgumentException("The provided configuration is unknown: " + args.get(i) + ":" + args.get(i + 1));
}
- return this.hadoopGraph;
+ ///
+ final HashMap<String, Object> configuration = new HashMap<>();
+ configuration.put(USE_SUGAR, this.useSugar);
+ configuration.put(USE_TRAVERSAL_SOURCE, this.traversalSource);
+ return Collections.unmodifiableMap(configuration);
}
@Override
public Object submit(final List<String> args) throws RemoteException {
try {
String script = RemoteAcceptor.getScript(String.join(SPACE, args), this.shell);
- if (this.useSugarPlugin)
+ if (this.useSugar)
script = SugarLoader.class.getCanonicalName() + ".load()\n" + script;
- final TraversalVertexProgram program = TraversalVertexProgram.build().traversal(GraphTraversalSource.computer(), "gremlin-groovy", script).create(this.hadoopGraph);
- final ComputerResult computerResult = this.hadoopGraph.compute().program(program).submit().get();
+ final TraversalVertexProgram program = TraversalVertexProgram.build().traversal(this.useTraversalSource, "gremlin-groovy", script).create(this.hadoopGraph);
+ final ComputerResult computerResult = this.traversalSource.getGraphComputer().get().program(program).submit().get();
this.shell.getInterp().getContext().setVariable(RESULT, computerResult);
-
-
+ ///
final Traversal.Admin<?, ?> traversal = new DefaultTraversal<>(computerResult.graph());
traversal.addStep(new ComputerResultStep<>(traversal, computerResult, false));
return traversal;
@@ -92,12 +111,4 @@ public final class HadoopRemoteAcceptor implements RemoteAcceptor {
public void close() throws IOException {
this.hadoopGraph.close();
}
-
- public HadoopGraph getGraph() {
- return this.hadoopGraph;
- }
-
- public String toString() {
- return "HadoopRemoteAcceptor[" + this.hadoopGraph + "]";
- }
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/34da329f/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPluginTest.java
----------------------------------------------------------------------
diff --git a/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPluginTest.java b/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPluginTest.java
index 768585b..71cde14 100644
--- a/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPluginTest.java
+++ b/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/groovy/plugin/HadoopGremlinPluginTest.java
@@ -21,7 +21,6 @@
package org.apache.tinkerpop.gremlin.hadoop.groovy.plugin;
-import org.apache.tinkerpop.gremlin.GraphManager;
import org.apache.tinkerpop.gremlin.LoadGraphWith;
import org.apache.tinkerpop.gremlin.groovy.plugin.RemoteAcceptor;
import org.apache.tinkerpop.gremlin.groovy.util.TestableConsolePluginAcceptor;
@@ -43,32 +42,16 @@ import static org.junit.Assert.*;
public class HadoopGremlinPluginTest extends AbstractGremlinProcessTest {
- private boolean ignore = false;
-
- @Before
- public void setup() throws Exception {
- if (GraphManager.getGraphProvider() != null) {
- super.setup();
- } else {
- this.ignore = true;
- }
- }
-
@Before
public void setupTest() {
- if (GraphManager.getGraphProvider() != null) {
- try {
- super.setupTest();
- this.console = new TestableConsolePluginAcceptor();
- final HadoopGremlinPlugin plugin = new HadoopGremlinPlugin();
- plugin.pluginTo(this.console);
- this.remote = (HadoopRemoteAcceptor) plugin.remoteAcceptor().get();
- } catch (final Exception e) {
- throw new IllegalStateException(e.getMessage(), e);
- }
-
- } else {
- this.ignore = true;
+ try {
+ super.setupTest();
+ this.console = new TestableConsolePluginAcceptor();
+ final HadoopGremlinPlugin plugin = new HadoopGremlinPlugin();
+ plugin.pluginTo(this.console);
+ this.remote = (HadoopRemoteAcceptor) plugin.remoteAcceptor().get();
+ } catch (final Exception e) {
+ throw new IllegalStateException(e.getMessage(), e);
}
}
@@ -80,66 +63,65 @@ public class HadoopGremlinPluginTest extends AbstractGremlinProcessTest {
@Test
@LoadGraphWith(LoadGraphWith.GraphData.MODERN)
public void shouldReturnResultIterator() throws Exception {
- if (!ignore) {
- this.console.addBinding("graph", this.graph);
- this.remote.connect(Arrays.asList("graph"));
- //
- Traversal<?, ?> traversal = (Traversal<?, ?>) this.remote.submit(Arrays.asList("g.V().count()"));
- assertEquals(6L, traversal.next());
- assertFalse(traversal.hasNext());
- assertNotNull(this.console.getBindings().get(RemoteAcceptor.RESULT));
- }
+ this.console.addBinding("graph", this.graph);
+ this.console.addBinding("g", this.g);
+ this.remote.connect(Arrays.asList("graph", "g"));
+ //
+ Traversal<?, ?> traversal = (Traversal<?, ?>) this.remote.submit(Arrays.asList("g.V().count()"));
+ assertEquals(6L, traversal.next());
+ assertFalse(traversal.hasNext());
+ assertNotNull(this.console.getBindings().get(RemoteAcceptor.RESULT));
}
@Test
@LoadGraphWith(LoadGraphWith.GraphData.MODERN)
public void shouldSupportSugar() throws Exception {
- if (!ignore) {
- this.console.addBinding("graph", this.graph);
- this.remote.connect(Arrays.asList("graph"));
- //
- this.remote.configure(Arrays.asList("useSugar", "true"));
- Traversal<?, ?> traversal = (Traversal<?, ?>) this.remote.submit(Arrays.asList("g.V.name.map{it.length()}.sum"));
- assertEquals(28.0d, traversal.next());
- assertFalse(traversal.hasNext());
- assertNotNull(this.console.getBindings().get(RemoteAcceptor.RESULT));
- }
+ this.console.addBinding("graph", this.graph);
+ this.console.addBinding("g", this.g);
+ this.remote.connect(Arrays.asList("graph"));
+ //
+ this.remote.configure(Arrays.asList("useSugar", "true"));
+ this.remote.connect(Arrays.asList("graph", "g"));
+ Traversal<?, ?> traversal = (Traversal<?, ?>) this.remote.submit(Arrays.asList("g.V.name.map{it.length()}.sum"));
+ assertEquals(28.0d, traversal.next());
+ assertFalse(traversal.hasNext());
+ assertNotNull(this.console.getBindings().get(RemoteAcceptor.RESULT));
}
@Test
@LoadGraphWith(LoadGraphWith.GraphData.MODERN)
public void shouldSupportHDFSMethods() throws Exception {
- if (!ignore) {
- List<String> ls = (List<String>) this.console.eval("hdfs.ls()");
- for (final String line : ls) {
- assertTrue(line.startsWith("-") || line.startsWith("r") || line.startsWith("w") || line.startsWith("x"));
- assertEquals(" ", line.substring(9, 10));
- }
- ls = (List<String>) this.console.eval("local.ls()");
- for (final String line : ls) {
- assertTrue(line.startsWith("-") || line.startsWith("r") || line.startsWith("w") || line.startsWith("x"));
- assertEquals(" ", line.substring(9, 10));
- }
- ////
- this.console.eval("hdfs.copyFromLocal('" + HadoopGraphProvider.PATHS.get("tinkerpop-classic.txt") + "', 'target/tinkerpop-classic.txt')");
- assertTrue((Boolean) this.console.eval("hdfs.exists('target/tinkerpop-classic.txt')"));
- ////
- List<String> head = IteratorUtils.asList(this.console.eval("hdfs.head('target/tinkerpop-classic.txt')"));
- assertEquals(6, head.size());
- for (final String line : head) {
- assertEquals(":", line.substring(1, 2));
- assertTrue(Integer.valueOf(line.substring(0, 1)) <= 6);
- }
- head = IteratorUtils.asList(this.console.eval("hdfs.head('target/tinkerpop-classic.txt',3)"));
- assertEquals(3, head.size());
- for (final String line : head) {
- assertEquals(":", line.substring(1, 2));
- assertTrue(Integer.valueOf(line.substring(0, 1)) <= 3);
- }
- ////
- this.console.eval("hdfs.rm('target/tinkerpop-classic.txt')");
- assertFalse((Boolean) this.console.eval("hdfs.exists('target/tinkerpop-classic.txt')"));
+
+ List<String> ls = (List<String>) this.console.eval("hdfs.ls()");
+ for (final String line : ls) {
+ assertTrue(line.startsWith("-") || line.startsWith("r") || line.startsWith("w") || line.startsWith("x"));
+ assertEquals(" ", line.substring(9, 10));
+ }
+ ls = (List<String>) this.console.eval("local.ls()");
+ for (final String line : ls) {
+ assertTrue(line.startsWith("-") || line.startsWith("r") || line.startsWith("w") || line.startsWith("x"));
+ assertEquals(" ", line.substring(9, 10));
}
+ ////
+ this.console.eval("hdfs.copyFromLocal('" + HadoopGraphProvider.PATHS.get("tinkerpop-classic.txt") + "', 'target/tinkerpop-classic.txt')");
+ assertTrue((Boolean) this.console.eval("hdfs.exists('target/tinkerpop-classic.txt')"));
+ ////
+ List<String> head = IteratorUtils.asList(this.console.eval("hdfs.head('target/tinkerpop-classic.txt')"));
+ assertEquals(6, head.size());
+ for (final String line : head) {
+ assertEquals(":", line.substring(1, 2));
+ assertTrue(Integer.valueOf(line.substring(0, 1)) <= 6);
+ }
+ head = IteratorUtils.asList(this.console.eval("hdfs.head('target/tinkerpop-classic.txt',3)"));
+ assertEquals(3, head.size());
+ for (final String line : head) {
+ assertEquals(":", line.substring(1, 2));
+ assertTrue(Integer.valueOf(line.substring(0, 1)) <= 3);
+ }
+ ////
+ this.console.eval("hdfs.rm('target/tinkerpop-classic.txt')");
+ assertFalse((Boolean) this.console.eval("hdfs.exists('target/tinkerpop-classic.txt')"));
+
}
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/34da329f/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/giraph/groovy/GiraphGraphComputerGroovyPluginIntegrateTest.java
----------------------------------------------------------------------
diff --git a/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/giraph/groovy/GiraphGraphComputerGroovyPluginIntegrateTest.java b/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/giraph/groovy/GiraphGraphComputerGroovyPluginIntegrateTest.java
deleted file mode 100644
index 9dba2bc..0000000
--- a/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/giraph/groovy/GiraphGraphComputerGroovyPluginIntegrateTest.java
+++ /dev/null
@@ -1,36 +0,0 @@
-/*
- *
- * * Licensed to the Apache Software Foundation (ASF) under one
- * * or more contributor license agreements. See the NOTICE file
- * * distributed with this work for additional information
- * * regarding copyright ownership. The ASF licenses this file
- * * to you under the Apache License, Version 2.0 (the
- * * "License"); you may not use this file except in compliance
- * * with the License. You may obtain a copy of the License at
- * *
- * * http://www.apache.org/licenses/LICENSE-2.0
- * *
- * * Unless required by applicable law or agreed to in writing,
- * * software distributed under the License is distributed on an
- * * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- * * KIND, either express or implied. See the License for the
- * * specific language governing permissions and limitations
- * * under the License.
- *
- */
-
-package org.apache.tinkerpop.gremlin.hadoop.process.computer.giraph.groovy;
-
-import org.apache.tinkerpop.gremlin.GraphProviderClass;
-import org.apache.tinkerpop.gremlin.hadoop.groovy.plugin.HadoopPluginSuite;
-import org.apache.tinkerpop.gremlin.hadoop.process.computer.giraph.HadoopGiraphGraphProvider;
-import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
-import org.junit.runner.RunWith;
-
-/**
- * @author Marko A. Rodriguez (http://markorodriguez.com)
- */
-@RunWith(HadoopPluginSuite.class)
-@GraphProviderClass(provider = HadoopGiraphGraphProvider.class, graph = HadoopGraph.class)
-public class GiraphGraphComputerGroovyPluginIntegrateTest {
-}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/34da329f/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/giraph/groovy/GiraphHadoopGremlinPluginIntegrateTest.java
----------------------------------------------------------------------
diff --git a/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/giraph/groovy/GiraphHadoopGremlinPluginIntegrateTest.java b/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/giraph/groovy/GiraphHadoopGremlinPluginIntegrateTest.java
new file mode 100644
index 0000000..f891029
--- /dev/null
+++ b/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/giraph/groovy/GiraphHadoopGremlinPluginIntegrateTest.java
@@ -0,0 +1,36 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one
+ * * or more contributor license agreements. See the NOTICE file
+ * * distributed with this work for additional information
+ * * regarding copyright ownership. The ASF licenses this file
+ * * to you under the Apache License, Version 2.0 (the
+ * * "License"); you may not use this file except in compliance
+ * * with the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing,
+ * * software distributed under the License is distributed on an
+ * * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * * KIND, either express or implied. See the License for the
+ * * specific language governing permissions and limitations
+ * * under the License.
+ *
+ */
+
+package org.apache.tinkerpop.gremlin.hadoop.process.computer.giraph.groovy;
+
+import org.apache.tinkerpop.gremlin.GraphProviderClass;
+import org.apache.tinkerpop.gremlin.hadoop.groovy.plugin.HadoopPluginSuite;
+import org.apache.tinkerpop.gremlin.hadoop.process.computer.giraph.HadoopGiraphGraphProvider;
+import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
+import org.junit.runner.RunWith;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+@RunWith(HadoopPluginSuite.class)
+@GraphProviderClass(provider = HadoopGiraphGraphProvider.class, graph = HadoopGraph.class)
+public class GiraphHadoopGremlinPluginIntegrateTest {
+}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/34da329f/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/spark/groovy/SparkHadoopGremlinPluginTest.java
----------------------------------------------------------------------
diff --git a/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/spark/groovy/SparkHadoopGremlinPluginTest.java b/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/spark/groovy/SparkHadoopGremlinPluginTest.java
new file mode 100644
index 0000000..37f08a1
--- /dev/null
+++ b/hadoop-gremlin/src/test/java/org/apache/tinkerpop/gremlin/hadoop/process/computer/spark/groovy/SparkHadoopGremlinPluginTest.java
@@ -0,0 +1,36 @@
+/*
+ *
+ * * Licensed to the Apache Software Foundation (ASF) under one
+ * * or more contributor license agreements. See the NOTICE file
+ * * distributed with this work for additional information
+ * * regarding copyright ownership. The ASF licenses this file
+ * * to you under the Apache License, Version 2.0 (the
+ * * "License"); you may not use this file except in compliance
+ * * with the License. You may obtain a copy of the License at
+ * *
+ * * http://www.apache.org/licenses/LICENSE-2.0
+ * *
+ * * Unless required by applicable law or agreed to in writing,
+ * * software distributed under the License is distributed on an
+ * * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * * KIND, either express or implied. See the License for the
+ * * specific language governing permissions and limitations
+ * * under the License.
+ *
+ */
+
+package org.apache.tinkerpop.gremlin.hadoop.process.computer.spark.groovy;
+
+import org.apache.tinkerpop.gremlin.GraphProviderClass;
+import org.apache.tinkerpop.gremlin.hadoop.groovy.plugin.HadoopPluginSuite;
+import org.apache.tinkerpop.gremlin.hadoop.process.computer.spark.HadoopSparkGraphProvider;
+import org.apache.tinkerpop.gremlin.hadoop.structure.HadoopGraph;
+import org.junit.runner.RunWith;
+
+/**
+ * @author Marko A. Rodriguez (http://markorodriguez.com)
+ */
+@RunWith(HadoopPluginSuite.class)
+@GraphProviderClass(provider = HadoopSparkGraphProvider.class, graph = HadoopGraph.class)
+public class SparkHadoopGremlinPluginTest {
+}