You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by ok...@apache.org on 2016/03/04 19:21:44 UTC
[27/27] incubator-tinkerpop git commit: When I was adding the UPGRADE
docs,
I decided to test the example traversal had in the UPGRADE docs and it failed.
I added two new traversals to SideEffectTest and tweaked some typing issues
to ensure there are no
When I was adding the UPGRADE docs, I decided to test the example traversal had in the UPGRADE docs and it failed. I added two new traversals to SideEffectTest and tweaked some typing issues to ensure there are no ambiguous method calls for withSideEffect().
Project: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/commit/14fae3e9
Tree: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/tree/14fae3e9
Diff: http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/diff/14fae3e9
Branch: refs/heads/master
Commit: 14fae3e95a518fb82cc7031caf381a6aa457868a
Parents: a1ee377
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Fri Mar 4 11:21:04 2016 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Fri Mar 4 11:21:04 2016 -0700
----------------------------------------------------------------------
CHANGELOG.asciidoc | 11 +++
.../upgrade/release-3.2.x-incubating.asciidoc | 71 ++++++++++++++++++++
.../process/traversal/TraversalSource.java | 30 +++++++--
.../dsl/graph/GraphTraversalSource.java | 9 ++-
.../step/sideEffect/GroovySideEffectTest.groovy | 10 +++
.../gremlin/groovy/loaders/StepLoader.groovy | 10 +++
.../step/sideEffect/SideEffectTest.java | 38 ++++++++++-
7 files changed, 171 insertions(+), 8 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/14fae3e9/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index de03eb7..92ee4cb 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -24,6 +24,17 @@ TinkerPop 3.2.0 (XXX)
TinkerPop 3.2.0 (NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Fixed a severe `Traversal` cloning issue that caused inconsistent `TraversalSideEffects`.
+* `TraversalSideEffects` remain consistent and usable across multiple chained OLAP jobs.
+* Added `MemoryTraversalSideEffects` which wraps `Memory` in a `TraversalSideEffects` for use in OLAP.
+* `TraversalSideEffects` are now fully functional in OLAP save that an accurate global view is possible at the start of an iteration (not during).
+* Updated the `TraversalSideEffects` API to support registered reducers and updated `get()`-semantics. (*breaking*)
+* Split existing `profile()` into `ProfileStep` and `ProfileSideEffectStep`.
+* The `profile()`-step acts like a reducing barrier and emits `TraversalMetrics` without the need for `cap()`. (*breaking*)
+* Added `LocalBarrier` interface to allow traversers to remain distributed during an iteration so as to reduce cluster traffic.
+* Added `NoOpBarrierStep` as a `LocalBarrier` implementation of `LambdaCollectingBarrierStep(noOp)`.
+* `AggregateStep` implements `LocalBarrier` and thus, doesn't needlessly communicate its barrier traversers.
+* Fixed an OLAP-based `Barrier` synchronization bug.
* Fixed a semantic bug in `BranchStep` (and inheriting steps) where barriers reacted locally. (*breaking*)
* Added `MemoryComputeKey` for specification of `Memory` keys in `VertexProgram`. (*breaking*)
* Added `VertexComputeKey` for specification of vertex compute properties in `VertexProgram`. (*breaking*)
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/14fae3e9/docs/src/upgrade/release-3.2.x-incubating.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/upgrade/release-3.2.x-incubating.asciidoc b/docs/src/upgrade/release-3.2.x-incubating.asciidoc
index f17e8a6..81963ae 100644
--- a/docs/src/upgrade/release-3.2.x-incubating.asciidoc
+++ b/docs/src/upgrade/release-3.2.x-incubating.asciidoc
@@ -32,6 +32,77 @@ Please see the link:https://github.com/apache/incubator-tinkerpop/blob/3.2.0-inc
Upgrading for Users
~~~~~~~~~~~~~~~~~~~
+TraversalSideEffect Update
+^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+There were changes to `TraversalSideEffect` both at the semantic level and at the API level. Users that have traversals
+of the form `sideEffect{...}` that leverage global side-effects should read the following carefully. If the user's traversals do
+not use lambda-based side-effect steps (e.g. `groupCount("m")`), then the changes below will not effect them. Moreover, if user's
+traversal only uses `sideEffect{...}` with closure (non-`TraversalSideEffect`) data references, then changes below will not effect them.
+Finally, providers should not be effected by the changes save any tests cases.
+
+TraversalSideEffect Get API Change
+++++++++++++++++++++++++++++++++++
+
+`TraversalSideEffects` can now logically operate within a distributed OLAP environment. In order to make this possible,
+it is necessary that each side-effect be registered with a reducing `BinaryOperator`. This binary operator will combine
+distributed updates into a single global side-effect at the master traversal. Many of the methods in `TraversalSideEffect`
+have been `Deprecated`, but they are backwards compatible save that `TraversalSideEffects.get()` no longer returns an `Optional`,
+but instead throws an `IllegalArgumentException`. While the `Optional` semantics could have remained, it was deemed best to
+directly return the side-effect value to reduce object creation costs and because all side-effects must be registered apriori,
+there is never a reason why an unknown side-effect key would be used. In short:
+
+```
+// change
+traversal.getSideEffects().get("m").get()
+// to
+traversal.getSideEffects().get("m")
+```
+
+TraversalSideEffect Registration Requirement
+++++++++++++++++++++++++++++++++++++++++++++
+
+All `TraversalSideEffects` must be registered upfront. This is because, in OLAP, side-effects map to `Memory` compute keys
+and as such, must be declared prior to the execution of the `TraversalVertexProgram`. If a user's traversal creates a
+side-effect mid-traversal, it will fail. The traversal must use `GraphTraversalSource.withSideEffect()` to declare
+the side-effects it will use during its execution lifetime. If the user's traversals use standard side-effect Gremlin
+steps (e.g. `group("m")`), then no changes are required.
+
+TraversalSideEffect Add Requirement
++++++++++++++++++++++++++++++++++++
+
+In a distributed environment, a side-effect can not be mutated and be expected to exist in the mutated form at the final,
+aggregated, master traversal. For instance, if the side-effect "myCount" references a `Long`, the `Long` can not be updated
+directly via `sideEffects.set("myCount", sideEffects.get("myCount") + 1)`. Instead, it must rely on the registered reducer
+to do the merging and thus, the `Step` must do `sideEffect.add("mySet",1)`, where the registered reducer is `Operator.sum`.
+Thus, the below will increment "a". If no operator was provided, then the operator is assumed `Operator.assign` and the
+final result of "a" would be 1. Note that `Traverser.sideEffects(key,value)` uses `TraversalSideEffect.add()`.
+
+```
+gremlin> traversal = g.withSideEffect('a',0,sum).V().out().sideEffect{it.sideEffects('a',1)}
+==>v[3]
+==>v[2]
+==>v[4]
+==>v[5]
+==>v[3]
+==>v[3]
+gremlin> traversal.getSideEffects().get('a')
+==>6
+```
+
+See link:https://issues.apache.org/jira/browse/TINKERPOP-1192[TINKERPOP-1192]
+
+ProfileStep Update and GraphTraversal API Change
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The `profile()`-step has been refactored into 2 steps -- `ProfileStep` and `ProfileSideEffectStep`. Users who previously
+used the `profile()` in conjunction with `cap(TraversalMetrics.METRICS_KEY)` can now simply omit the cap step. Users who
+retrieved `TraversalMetrics` from the side-effects after iteration can still do so, but will need to specify a side-effect
+key when using the `profile()`. For example, `profile("myMetrics")`.
+
+See link:https://issues.apache.org/jira/browse/TINKERPOP-958[TINKERPOP-958]
+
+
BranchStep Bug Fix
^^^^^^^^^^^^^^^^^^
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/14fae3e9/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/TraversalSource.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/TraversalSource.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/TraversalSource.java
index 4639130..a1a39b6 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/TraversalSource.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/TraversalSource.java
@@ -121,6 +121,15 @@ public interface TraversalSource extends Cloneable {
return this.withComputer(Graph::compute);
}
+ /**
+ * Add a sideEffect to be used throughout the life of a spawned {@link Traversal}.
+ * This adds a {@link org.apache.tinkerpop.gremlin.process.traversal.strategy.decoration.SideEffectStrategy} to the strategies.
+ *
+ * @param key the key of the sideEffect
+ * @param initialValue a supplier that produces the initial value of the sideEffect
+ * @param reducer a reducer to merge sideEffect mutations into a single result
+ * @return a new traversal source with updated strategies
+ */
public default <A> TraversalSource withSideEffect(final String key, final Supplier<A> initialValue, final BinaryOperator<A> reducer) {
final TraversalSource clone = this.clone();
SideEffectStrategy.addSideEffect(clone.getStrategies(), key, (A) initialValue, reducer);
@@ -133,10 +142,11 @@ public interface TraversalSource extends Cloneable {
*
* @param key the key of the sideEffect
* @param initialValue the initial value of the sideEffect
+ * @param reducer a reducer to merge sideEffect mutations into a single result
* @return a new traversal source with updated strategies
*/
- public default TraversalSource withSideEffect(final String key, final Object initialValue) {
- return this.withSideEffect(key, new ConstantSupplier<>(initialValue));
+ public default <A> TraversalSource withSideEffect(final String key, final A initialValue, final BinaryOperator<A> reducer) {
+ return this.withSideEffect(key, initialValue instanceof Supplier ? (Supplier) initialValue : new ConstantSupplier<>(initialValue), reducer);
}
/**
@@ -147,8 +157,20 @@ public interface TraversalSource extends Cloneable {
* @param initialValue a supplier that produces the initial value of the sideEffect
* @return a new traversal source with updated strategies
*/
- public default TraversalSource withSideEffect(final String key, final Supplier initialValue) {
- return this.withSideEffect(key, initialValue, null);
+ public default <A> TraversalSource withSideEffect(final String key, final Supplier<A> initialValue) {
+ return this.withSideEffect(key, initialValue, (BinaryOperator<A>) null);
+ }
+
+ /**
+ * Add a sideEffect to be used throughout the life of a spawned {@link Traversal}.
+ * This adds a {@link org.apache.tinkerpop.gremlin.process.traversal.strategy.decoration.SideEffectStrategy} to the strategies.
+ *
+ * @param key the key of the sideEffect
+ * @param initialValue the initial value of the sideEffect
+ * @return a new traversal source with updated strategies
+ */
+ public default <A> TraversalSource withSideEffect(final String key, final A initialValue) {
+ return this.withSideEffect(key, initialValue instanceof Supplier ? (Supplier) initialValue : new ConstantSupplier<>(initialValue));
}
/**
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/14fae3e9/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversalSource.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversalSource.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversalSource.java
index 9ff6c5d..d06dfac 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversalSource.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/dsl/graph/GraphTraversalSource.java
@@ -124,12 +124,17 @@ public class GraphTraversalSource implements TraversalSource {
}
@Override
- public GraphTraversalSource withSideEffect(final String key, final Object initialValue) {
+ public <A> GraphTraversalSource withSideEffect(final String key, final A initialValue, final BinaryOperator<A> reducer) {
+ return (GraphTraversalSource) TraversalSource.super.withSideEffect(key, initialValue, reducer);
+ }
+
+ @Override
+ public <A> GraphTraversalSource withSideEffect(final String key, final A initialValue) {
return (GraphTraversalSource) TraversalSource.super.withSideEffect(key, initialValue);
}
@Override
- public GraphTraversalSource withSideEffect(final String key, final Supplier initialValue) {
+ public <A> GraphTraversalSource withSideEffect(final String key, final Supplier<A> initialValue) {
return (GraphTraversalSource) TraversalSource.super.withSideEffect(key, initialValue);
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/14fae3e9/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroovySideEffectTest.groovy
----------------------------------------------------------------------
diff --git a/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroovySideEffectTest.groovy b/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroovySideEffectTest.groovy
index 399b18a..36418f3 100644
--- a/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroovySideEffectTest.groovy
+++ b/gremlin-groovy-test/src/main/groovy/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/GroovySideEffectTest.groovy
@@ -69,5 +69,15 @@ public abstract class GroovySideEffectTest {
.cap('a');
""");
}
+
+ @Override
+ public Traversal<Vertex, Integer> get_g_withSideEffectXa_0_sumX_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX() {
+ new ScriptTraversal<>(g, "gremlin-groovy", "g.withSideEffect('a', 0, sum).V.out.sideEffect{it.sideEffects('a', it.bulk() as int)}.cap('a')")
+ }
+
+ @Override
+ public Traversal<Vertex, Integer> get_g_withSideEffectXa_0X_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX() {
+ new ScriptTraversal<>(g, "gremlin-groovy", "g.withSideEffect('a', 0).V.out.sideEffect{it.sideEffects('a', it.bulk() as int)}.cap('a')")
+ }
}
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/14fae3e9/gremlin-groovy/src/main/groovy/org/apache/tinkerpop/gremlin/groovy/loaders/StepLoader.groovy
----------------------------------------------------------------------
diff --git a/gremlin-groovy/src/main/groovy/org/apache/tinkerpop/gremlin/groovy/loaders/StepLoader.groovy b/gremlin-groovy/src/main/groovy/org/apache/tinkerpop/gremlin/groovy/loaders/StepLoader.groovy
index ccf1a2c..b1cd97f 100644
--- a/gremlin-groovy/src/main/groovy/org/apache/tinkerpop/gremlin/groovy/loaders/StepLoader.groovy
+++ b/gremlin-groovy/src/main/groovy/org/apache/tinkerpop/gremlin/groovy/loaders/StepLoader.groovy
@@ -37,6 +37,16 @@ class StepLoader {
return ((GraphTraversal) delegate).by(1 == closure.getMaximumNumberOfParameters() ? closure as Function : closure as Comparator);
}
+ TraversalSource.metaClass.withSideEffect = {
+ final String key, final Closure initialValue, final Closure reducer ->
+ return ((TraversalSource) delegate).withSideEffect(key, initialValue as Supplier, reducer as BinaryOperator);
+ }
+
+ TraversalSource.metaClass.withSideEffect = {
+ final String key, final Closure initialValue, final BinaryOperator reducer ->
+ return ((TraversalSource) delegate).withSideEffect(key, initialValue as Supplier, reducer);
+ }
+
TraversalSource.metaClass.withSideEffect = { final String key, final Closure object ->
return ((TraversalSource) delegate).withSideEffect(key, object as Supplier);
}
http://git-wip-us.apache.org/repos/asf/incubator-tinkerpop/blob/14fae3e9/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/SideEffectTest.java
----------------------------------------------------------------------
diff --git a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/SideEffectTest.java b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/SideEffectTest.java
index e9af4b5..7230f42 100644
--- a/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/SideEffectTest.java
+++ b/gremlin-test/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/sideEffect/SideEffectTest.java
@@ -64,6 +64,10 @@ public abstract class SideEffectTest extends AbstractGremlinProcessTest {
public abstract Traversal<Vertex, Map<String, Long>> get_g_withSideEffectsXa__linkedhashmapX_withSideEffectXb__arraylist__addAllX_withSideEffectXc__arrayList__addAllX_V_groupXaX_byXlabelX_byXcountX_sideEffectXb__1_2_3X_out_out_out_sideEffectXc__bob_danielX_capXaX();
+ public abstract Traversal<Vertex, Integer> get_g_withSideEffectXa_0_sumX_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX();
+
+ public abstract Traversal<Vertex, Integer> get_g_withSideEffectXa_0X_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX();
+
@Test
@LoadGraphWith(MODERN)
@IgnoreEngine(TraversalEngine.Type.COMPUTER)
@@ -161,6 +165,26 @@ public abstract class SideEffectTest extends AbstractGremlinProcessTest {
assertEquals(0, sideEffects.<List>get("c").size());
}
+ @Test
+ @LoadGraphWith(MODERN)
+ public void g_withSideEffectXa_0_sumX_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX() {
+ final Traversal<Vertex, Integer> traversal = get_g_withSideEffectXa_0_sumX_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX();
+ assertEquals(6, traversal.next().intValue());
+ assertFalse(traversal.hasNext());
+ assertEquals(6, traversal.asAdmin().getSideEffects().<Integer>get("a").intValue());
+ assertEquals(1, traversal.asAdmin().getSideEffects().keys().size());
+ }
+
+ @Test
+ @LoadGraphWith(MODERN)
+ public void g_withSideEffectXa_0X_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX() {
+ final Traversal<Vertex, Integer> traversal = get_g_withSideEffectXa_0X_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX();
+ assertEquals(1, traversal.next().intValue());
+ assertFalse(traversal.hasNext());
+ assertEquals(1, traversal.asAdmin().getSideEffects().<Integer>get("a").intValue());
+ assertEquals(1, traversal.asAdmin().getSideEffects().keys().size());
+ }
+
public static class Traversals extends SideEffectTest {
@Override
@@ -198,14 +222,24 @@ public abstract class SideEffectTest extends AbstractGremlinProcessTest {
@Override
public Traversal<Vertex, Map<String, Long>> get_g_withSideEffectsXa__linkedhashmapX_withSideEffectXb__arraylist__addAllX_withSideEffectXc__arrayList__addAllX_V_groupXaX_byXlabelX_byXcountX_sideEffectXb__1_2_3X_out_out_out_sideEffectXc__bob_danielX_capXaX() {
return g.withSideEffect("a", new LinkedHashMapSupplier())
- .withSideEffect("b", (Supplier) ArrayListSupplier.instance(), Operator.addAll)
- .withSideEffect("c", (Supplier) ArrayListSupplier.instance(), Operator.addAll)
+ .withSideEffect("b", ArrayListSupplier.instance(), Operator.addAll)
+ .withSideEffect("c", ArrayListSupplier.instance(), Operator.addAll)
.V().group("a").by(T.label).by(__.count())
.sideEffect(t -> t.sideEffects("b", new LinkedList<>(Arrays.asList(1, 2, 3))))
.out().out().out()
.sideEffect(t -> t.sideEffects("c", new LinkedList<>(Arrays.asList("bob", "daniel"))))
.cap("a");
}
+
+ @Override
+ public Traversal<Vertex, Integer> get_g_withSideEffectXa_0_sumX_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX() {
+ return g.withSideEffect("a", 0, Operator.sum).V().out().sideEffect(t -> t.sideEffects("a", (int) t.bulk())).cap("a");
+ }
+
+ @Override
+ public Traversal<Vertex, Integer> get_g_withSideEffectXa_0X_V_out_sideEffectXsideEffectsXa_bulkXX_capXaX() {
+ return g.withSideEffect("a", 0).V().out().sideEffect(t -> t.sideEffects("a", (int) t.bulk())).cap("a");
+ }
}
private static class LinkedHashMapSupplier implements Supplier<LinkedHashMap> {