You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by ok...@apache.org on 2016/03/04 19:21:44 UTC

[27/27] incubator-tinkerpop git commit: When I was adding the UPGRADE docs, I decided to test the example traversal had in the UPGRADE docs and it failed. I added two new traversals to SideEffectTest and tweaked some typing issues to ensure there are no

When I was adding the UPGRADE docs, I decided to test the example traversal had in the UPGRADE docs and it failed. I added two new traversals to SideEffectTest and tweaked some typing issues to ensure there are no ambiguous method calls for withSideEffect().


Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/14fae3e9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/14fae3e9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/14fae3e9

Branch: refs/heads/master
Commit: 14fae3e95a518fb82cc7031caf381a6aa457868a
Parents: a1ee377
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Fri Mar 4 11:21:04 2016 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Fri Mar 4 11:21:04 2016 -0700

----------------------------------------------------------------------
 CHANGELOG.asciidoc                              | 11 +++
 .../upgrade/release-3.2.x-incubating.asciidoc   | 71 ++++++++++++++++++++
 .../process/traversal/TraversalSource.java      | 30 +++++++--
 .../dsl/graph/GraphTraversalSource.java         |  9 ++-
 .../step/sideEffect/GroovySideEffectTest.groovy | 10 +++
 .../gremlin/groovy/loaders/StepLoader.groovy    | 10 +++
 .../step/sideEffect/SideEffectTest.java         | 38 ++++++++++-
 7 files changed, 171 insertions(+), 8 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/14fae3e9/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index de03eb7..92ee4cb 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -24,6 +24,17 @@ TinkerPop 3.2.0 (XXX)
 TinkerPop 3.2.0 (NOT OFFICIALLY RELEASED YET)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
+* Fixed a severe `Traversal` cloning issue that caused inconsistent `TraversalSideEffects`.
+* `TraversalSideEffects` remain consistent and usable across multiple chained OLAP jobs.
+* Added `MemoryTraversalSideEffects` which wraps `Memory` in a `TraversalSideEffects` for use in OLAP.
+* `TraversalSideEffects` are now fully functional in OLAP save that an accurate global view is possible at the start of an iteration (not during).
+* Updated the `TraversalSideEffects` API to support registered reducers and updated `get()`-semantics. (*breaking*)
+* Split existing `profile()` into `ProfileStep` and `ProfileSideEffectStep`.
+* The `profile()`-step acts like a reducing barrier and emits `TraversalMetrics` without the need for `cap()`. (*breaking*)
+* Added `LocalBarrier` interface to allow traversers to remain distributed during an iteration so as to reduce cluster traffic.
+* Added `NoOpBarrierStep` as a `LocalBarrier` implementation of `LambdaCollectingBarrierStep(noOp)`.
+* `AggregateStep` implements `LocalBarrier` and thus, doesn't needlessly communicate its barrier traversers.
+* Fixed an OLAP-based `Barrier` synchronization bug.
 * Fixed a semantic bug in `BranchStep` (and inheriting steps) where barriers reacted locally. (*breaking*)
 * Added `MemoryComputeKey` for specification of `Memory` keys in `VertexProgram`. (*breaking*)
 * Added `VertexComputeKey` for specification of vertex compute properties in `VertexProgram`. (*breaking*)

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/14fae3e9/docs/src/upgrade/release-3.2.x-incubating.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/upgrade/release-3.2.x-incubating.asciidoc b/docs/src/upgrade/release-3.2.x-incubating.asciidoc
index f17e8a6..81963ae 100644
--- a/docs/src/upgrade/release-3.2.x-incubating.asciidoc
+++ b/docs/src/upgrade/release-3.2.x-incubating.asciidoc
@@ -32,6 +32,77 @@ Please see the link:https://github.com/apache/incubator-tinkerpop/blob/3.2.0-inc
 Upgrading for Users
 ~~~~~~~~~~~~~~~~~~~
 
+TraversalSideEffect Update
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There were changes to `TraversalSideEffect` both at the semantic level and at the API level. Users that have traversals
+of the form `sideEffect{...}` that leverage global side-effects should read the following carefully. If the user's traversals do
+not use lambda-based side-effect steps (e.g. `groupCount("m")`), then the changes below will not effect them. Moreover, if user's
+traversal only uses `sideEffect{...}` with closure (non-`TraversalSideEffect`) data references, then changes below will not effect them.
+Finally, providers should not be effected by the changes save any tests cases.
+
+TraversalSideEffect Get API Change
+++++++++++++++++++++++++++++++++++
+
+`TraversalSideEffects` can now logically operate within a distributed OLAP environment. In order to make this possible,
+it is necessary that each side-effect be registered with a reducing `BinaryOperator`. This binary operator will combine
+distributed updates into a single global side-effect at the master traversal. Many of the methods in `TraversalSideEffect`
+have been `Deprecated`, but they are backwards compatible save that `TraversalSideEffects.get()` no longer returns an `Optional`,
+but instead throws an `IllegalArgumentException`. While the `Optional` semantics could have remained, it was deemed best to
+directly return the side-effect value to reduce object creation costs and because all side-effects must be registered apriori,
+there is never a reason why an unknown side-effect key would be used. In short:
+
+```
+// change
+traversal.getSideEffects().get("m").get()
+// to
+traversal.getSideEffects().get("m")
+```
+
+TraversalSideEffect Registration Requirement
+++++++++++++++++++++++++++++++++++++++++++++
+
+All `TraversalSideEffects` must be registered upfront. This is because, in OLAP, side-effects map to `Memory` compute keys
+and as such, must be declared prior to the execution of the `TraversalVertexProgram`. If a user's traversal creates a
+side-effect mid-traversal, it will fail. The traversal must use `GraphTraversalSource.withSideEffect()` to declare
+the side-effects it will use during its execution lifetime. If the user's traversals use standard side-effect Gremlin
+steps (e.g. `group("m")`), then no changes are required.
+
+TraversalSideEffect Add Requirement
++++++++++++++++++++++++++++++++++++
+
+In a distributed environment, a side-effect can not be mutated and be expected to exist in the mutated form at the final,
+aggregated, master traversal. For instance, if the side-effect "myCount" references a `Long`, the `Long` can not be updated
+directly via `sideEffects.set("myCount", sideEffects.get("myCount") + 1)`. Instead, it must rely on the registered reducer
+to do the merging and thus, the `Step` must do `sideEffect.add("mySet",1)`, where the registered reducer is `Operator.sum`.
+Thus, the below will increment "a". If no operator was provided, then the operator is assumed `Operator.assign` and the
+final result of "a" would be 1. Note that `Traverser.sideEffects(key,value)` uses `TraversalSideEffect.add()`.
+
+```
+gremlin> traversal = g.withSideEffect('a',0,sum).V().out().sideEffect{it.sideEffects('a',1)}
+==>v[3]
+==>v[2]
+==>v[4]
+==>v[5]
+==>v[3]
+==>v[3]
+gremlin> traversal.getSideEffects().get('a')
+==>6
+```
+
+See link:https://issues.apache.org/jira/browse/TINKERPOP-1192[TINKERPOP-1192]
+
+ProfileStep Update and GraphTraversal API Change
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The `profile()`-step has been refactored into 2 steps -- `ProfileStep` and `ProfileSideEffectStep`. Users who previously
+used the `profile()` in conjunction with `cap(TraversalMetrics.METRICS_KEY)` can now simply omit the cap step. Users who
+retrieved `TraversalMetrics` from the side-effects after iteration can still do so, but will need to specify a side-effect
+key when using the `profile()`. For example, `profile("myMetrics")`.
+
+See link:https://issues.apache.org/jira/browse/TINKERPOP-958[TINKERPOP-958]
+
+
 BranchStep Bug Fix
 ^^^^^^^^^^^^^^^^^^
 

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/14fae3e9/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/TraversalSource.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/TraversalSource.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/TraversalSource.java
index 4639130..a1a39b6 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/TraversalSource.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/TraversalSource.java
@@ -121,6 +121,15 @@ public interface TraversalSource extends Cloneable {
         return this.withComputer(Graph::compute);
     }
 
+    /**
+     * Add a sideEffect to be used throughout the life of a spawned {@link Traversal}.
+     * This adds a {@link org.apache.tinkerpop.gremlin.process.traversal.strategy.decoration.SideEffectStrategy} to the strategies.
+     *
+     * @param key          the key of the sideEffect
+     * @param initialValue a supplier that produces the initial value of the sideEffect
+     * @param reducer      a reducer to merge sideEffect mutations into a single result
+     * @return a new traversal source with updated strategies
+     */
     public default <A> TraversalSource withSideEffect(final String key, final Supplier<A> initialValue, final BinaryOperator<A> reducer) {
         final TraversalSource clone = this.clone();
         SideEffectStrategy.addSideEffect(clone.getStrategies(), key, (A) initialValue, reducer);
@@ -133,10 +142,11 @@ public interface TraversalSource extends Cloneable {
      *
      * @param key          the key of the sideEffect
      * @param initialValue the initial value of the sideEffect
+     * @param reducer      a reducer to merge sideEffect mutations into a single result
      * @return a new traversal source with updated strategies
      */
-    public default TraversalSource withSideEffect(final String key, final Object initialValue) {
-        return this.withSideEffect(key, new ConstantSupplier<>(initialValue));
+    public default <A> TraversalSource withSideEffect(final String key, final A initialValue, final BinaryOperator<A> reducer) {
+        return this.withSideEffect(key, initialValue instanceof Supplier ? (Supplier) initialValue : new ConstantSupplier<>(initialValue), reducer);
     }
 
     /**
@@ -147,8 +157,20 @@ public interface TraversalSource extends Cloneable {
      * @param initialValue a supplier that produces the initial value of the sideEffect
      * @return a new traversal source with updated strategies
      */
-    public default TraversalSource withSideEffect(final String key, final Supplier initialValue) {
-        return this.withSideEffect(key, initialValue, null);
+    public default <A> TraversalSource withSideEffect(final String key, final Supplier<A> initialValue) {
+        return this.withSideEffect(key, initialValue, (BinaryOperator<A>) null);
+    }
+
+    /**
+     * Add a sideEffect to be used throughout the life of a spawned {@link Traversal}.
+     * This adds a {@link org.apache.tinkerpop.gremlin.process.traversal.strategy.decoration.SideEffectStrategy} to the strategies.
+     *
+     * @param key          the key of the sideEffect
+     * @param initialValue the initial value of the sideEffect
+     * @return a new traversal source with updated strategies
+     */
+    public default <A> TraversalSource withSideEffect(final String key, final A initialValue) {
+        return this.withSideEffect(key, initialValue instanceof Supplier ? (Supplier) initialValue : new ConstantSupplier<>(initialValue));
     }
 
     /**

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/14fae3e9/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversalSource.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversalSource.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversalSource.java
index 9ff6c5d..d06dfac 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversalSource.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversalSource.java
@@ -124,12 +124,17 @@ public class GraphTraversalSource implements TraversalSource {
     }
 
     @Override
-    public GraphTraversalSource withSideEffect(final String key, final Object initialValue) {
+    public <A> GraphTraversalSource withSideEffect(final String key, final A initialValue, final BinaryOperator<A> reducer) {
+        return (GraphTraversalSource) TraversalSource.super.withSideEffect(key, initialValue, reducer);
+    }
+
+    @Override
+    public <A> GraphTraversalSource withSideEffect(final String key, final A initialValue) {
         return (GraphTraversalSource) TraversalSource.super.withSideEffect(key, initialValue);
     }
 
     @Override
-    public GraphTraversalSource withSideEffect(final String key, final Supplier initialValue) {
+    public <A> GraphTraversalSource withSideEffect(final String key, final Supplier<A> initialValue) {
         return (GraphTraversalSource) TraversalSource.super.withSideEffect(key, initialValue);
     }
 

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/14fae3e9/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroovySideEffectTest.groovy
----------------------------------------------------------------------
diff --git a/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroovySideEffectTest.groovy b/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroovySideEffectTest.groovy
index 399b18a..36418f3 100644
--- a/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroovySideEffectTest.groovy
+++ b/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroovySideEffectTest.groovy
@@ -69,5 +69,15 @@ public abstract class GroovySideEffectTest {
                     .cap('a');
                 """);
         }
+
+        @Override
+        public Traversal<Vertex, Integer> get_g_withSideEffectXa_0_sumX_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX() {
+            new ScriptTraversal<>(g, "gremlin-groovy", "g.withSideEffect('a', 0, sum).V.out.sideEffect{it.sideEffects('a', it.bulk() as int)}.cap('a')")
+        }
+
+        @Override
+        public Traversal<Vertex, Integer> get_g_withSideEffectXa_0X_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX() {
+            new ScriptTraversal<>(g, "gremlin-groovy", "g.withSideEffect('a', 0).V.out.sideEffect{it.sideEffects('a', it.bulk() as int)}.cap('a')")
+        }
     }
 }

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/14fae3e9/gremlin-groovy/src/main/groovy/org/apache/tinkerpop/gremlin/groovy/loaders/StepLoader.groovy
----------------------------------------------------------------------
diff --git a/gremlin-groovy/src/main/groovy/org/apache/tinkerpop/gremlin/groovy/loaders/StepLoader.groovy b/gremlin-groovy/src/main/groovy/org/apache/tinkerpop/gremlin/groovy/loaders/StepLoader.groovy
index ccf1a2c..b1cd97f 100644
--- a/gremlin-groovy/src/main/groovy/org/apache/tinkerpop/gremlin/groovy/loaders/StepLoader.groovy
+++ b/gremlin-groovy/src/main/groovy/org/apache/tinkerpop/gremlin/groovy/loaders/StepLoader.groovy
@@ -37,6 +37,16 @@ class StepLoader {
             return ((GraphTraversal) delegate).by(1 == closure.getMaximumNumberOfParameters() ? closure as Function : closure as Comparator);
         }
 
+        TraversalSource.metaClass.withSideEffect = {
+            final String key, final Closure initialValue, final Closure reducer ->
+                return ((TraversalSource) delegate).withSideEffect(key, initialValue as Supplier, reducer as BinaryOperator);
+        }
+
+        TraversalSource.metaClass.withSideEffect = {
+            final String key, final Closure initialValue, final BinaryOperator reducer ->
+                return ((TraversalSource) delegate).withSideEffect(key, initialValue as Supplier, reducer);
+        }
+
         TraversalSource.metaClass.withSideEffect = { final String key, final Closure object ->
             return ((TraversalSource) delegate).withSideEffect(key, object as Supplier);
         }

http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/14fae3e9/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/SideEffectTest.java
----------------------------------------------------------------------
diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/SideEffectTest.java b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/SideEffectTest.java
index e9af4b5..7230f42 100644
--- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/SideEffectTest.java
+++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/SideEffectTest.java
@@ -64,6 +64,10 @@ public abstract class SideEffectTest extends AbstractGremlinProcessTest {
 
     public abstract Traversal<Vertex, Map<String, Long>> get_g_withSideEffectsXa__linkedhashmapX_withSideEffectXb__arraylist__addAllX_withSideEffectXc__arrayList__addAllX_V_groupXaX_byXlabelX_byXcountX_sideEffectXb__1_2_3X_out_out_out_sideEffectXc__bob_danielX_capXaX();
 
+    public abstract Traversal<Vertex, Integer> get_g_withSideEffectXa_0_sumX_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX();
+
+    public abstract Traversal<Vertex, Integer> get_g_withSideEffectXa_0X_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX();
+
     @Test
     @LoadGraphWith(MODERN)
     @IgnoreEngine(TraversalEngine.Type.COMPUTER)
@@ -161,6 +165,26 @@ public abstract class SideEffectTest extends AbstractGremlinProcessTest {
         assertEquals(0, sideEffects.<List>get("c").size());
     }
 
+    @Test
+    @LoadGraphWith(MODERN)
+    public void g_withSideEffectXa_0_sumX_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX() {
+        final Traversal<Vertex, Integer> traversal = get_g_withSideEffectXa_0_sumX_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX();
+        assertEquals(6, traversal.next().intValue());
+        assertFalse(traversal.hasNext());
+        assertEquals(6, traversal.asAdmin().getSideEffects().<Integer>get("a").intValue());
+        assertEquals(1, traversal.asAdmin().getSideEffects().keys().size());
+    }
+
+    @Test
+    @LoadGraphWith(MODERN)
+    public void g_withSideEffectXa_0X_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX() {
+        final Traversal<Vertex, Integer> traversal = get_g_withSideEffectXa_0X_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX();
+        assertEquals(1, traversal.next().intValue());
+        assertFalse(traversal.hasNext());
+        assertEquals(1, traversal.asAdmin().getSideEffects().<Integer>get("a").intValue());
+        assertEquals(1, traversal.asAdmin().getSideEffects().keys().size());
+    }
+
     public static class Traversals extends SideEffectTest {
 
         @Override
@@ -198,14 +222,24 @@ public abstract class SideEffectTest extends AbstractGremlinProcessTest {
         @Override
         public Traversal<Vertex, Map<String, Long>> get_g_withSideEffectsXa__linkedhashmapX_withSideEffectXb__arraylist__addAllX_withSideEffectXc__arrayList__addAllX_V_groupXaX_byXlabelX_byXcountX_sideEffectXb__1_2_3X_out_out_out_sideEffectXc__bob_danielX_capXaX() {
             return g.withSideEffect("a", new LinkedHashMapSupplier())
-                    .withSideEffect("b", (Supplier) ArrayListSupplier.instance(), Operator.addAll)
-                    .withSideEffect("c", (Supplier) ArrayListSupplier.instance(), Operator.addAll)
+                    .withSideEffect("b", ArrayListSupplier.instance(), Operator.addAll)
+                    .withSideEffect("c", ArrayListSupplier.instance(), Operator.addAll)
                     .V().group("a").by(T.label).by(__.count())
                     .sideEffect(t -> t.sideEffects("b", new LinkedList<>(Arrays.asList(1, 2, 3))))
                     .out().out().out()
                     .sideEffect(t -> t.sideEffects("c", new LinkedList<>(Arrays.asList("bob", "daniel"))))
                     .cap("a");
         }
+
+        @Override
+        public Traversal<Vertex, Integer> get_g_withSideEffectXa_0_sumX_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX() {
+            return g.withSideEffect("a", 0, Operator.sum).V().out().sideEffect(t -> t.sideEffects("a", (int) t.bulk())).cap("a");
+        }
+
+        @Override
+        public Traversal<Vertex, Integer> get_g_withSideEffectXa_0X_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX() {
+            return g.withSideEffect("a", 0).V().out().sideEffect(t -> t.sideEffects("a", (int) t.bulk())).cap("a");
+        }
     }
 
     private static class LinkedHashMapSupplier implements Supplier<LinkedHashMap> {