You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by ok...@apache.org on 2017/02/16 18:09:50 UTC
[1/4] tinkerpop git commit: Fixed an optimization bug in
LazyBarrierStrategy. RemoteStrategy should be responsible for network I/O
concerns (and using bulking as the optimization).
Repository: tinkerpop
Updated Branches:
refs/heads/master 9a39bdef1 -> 768ade6bd
Fixed an optimization bug in LazyBarrierStrategy. RemoteStrategy should be responsible for network I/O concerns (and using bulking as the optimization).
Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/080306b0
Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/080306b0
Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/080306b0
Branch: refs/heads/master
Commit: 080306b099b3ca5dbf5bf5041447fd1a0557b9cb
Parents: 9d74372
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Wed Feb 15 15:07:53 2017 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Wed Feb 15 15:07:53 2017 -0700
----------------------------------------------------------------------
CHANGELOG.asciidoc | 3 ++-
.../upgrade/release-3.2.x-incubating.asciidoc | 26 ++++++++++++++++++++
.../strategy/decoration/RemoteStrategy.java | 8 +++++-
.../optimization/LazyBarrierStrategy.java | 2 +-
.../optimization/LazyBarrierStrategyTest.java | 19 +++++++-------
.../Neo4jGraphStepStrategyTest.java | 2 +-
.../TinkerGraphStepStrategyTest.java | 2 +-
7 files changed, 47 insertions(+), 15 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/080306b0/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index b550fe4..cf7312e 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -26,7 +26,8 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima
TinkerPop 3.2.5 (Release Date: NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
+* Fixed an optimization bug in `LazyBarrierStrategy` around appending barriers to the end of a `Traversal`.
+* `RemoteStrategy` is smart to try and append a `NoOpBarrierStep` in order to bulk traversers prior to network I/O.
[[release-3-2-4]]
TinkerPop 3.2.4 (Release Date: February 8, 2017)
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/080306b0/docs/src/upgrade/release-3.2.x-incubating.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/upgrade/release-3.2.x-incubating.asciidoc b/docs/src/upgrade/release-3.2.x-incubating.asciidoc
index c199238..1247433 100644
--- a/docs/src/upgrade/release-3.2.x-incubating.asciidoc
+++ b/docs/src/upgrade/release-3.2.x-incubating.asciidoc
@@ -22,6 +22,32 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima
*Nine Inch Gremlins*
+TinkerPop 3.2.5
+---------------
+
+*Release Date: NOT OFFICIALLY RELEASED YET*
+
+Please see the link:https://github.com/apache/tinkerpop/blob/3.2.5/CHANGELOG.asciidoc#release-3-2-5[changelog] for a complete list of all the modifications that are part of this release.
+
+Upgrading for Users
+~~~~~~~~~~~~~~~~~~~
+
+Upgrading for Providers
+~~~~~~~~~~~~~~~~~~~~~~~
+
+LazyBarrierStrategy No Longer End Appends Barriers
+++++++++++++++++++++++++++++++++++++++++++++++++++
+
+`LazyBarrierStrategy` was trying to do to much by considering `Traverser` effects on network I/O by appending an
+`NoOpBarrierStrategy` to the end of the root traversal. This should not be accomplished by `LazyBarrierStrategy`,
+but instead by `RemoteStrategy`. `RemoteStrategy` now tries to barrier-append. This may effect the reasoninig logic in
+some `ProviderStrategies`. Most likely not, but just be aware.
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP-1627[TINKERPOP-1627]
+
+TinkerGraph Deserialization
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
TinkerPop 3.2.4
---------------
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/080306b0/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/remote/traversal/strategy/decoration/RemoteStrategy.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/remote/traversal/strategy/decoration/RemoteStrategy.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/remote/traversal/strategy/decoration/RemoteStrategy.java
index 38ceb95..14d3b36 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/remote/traversal/strategy/decoration/RemoteStrategy.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/remote/traversal/strategy/decoration/RemoteStrategy.java
@@ -24,6 +24,8 @@ import org.apache.tinkerpop.gremlin.process.remote.RemoteGraph;
import org.apache.tinkerpop.gremlin.process.remote.traversal.step.map.RemoteStep;
import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
import org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy;
+import org.apache.tinkerpop.gremlin.process.traversal.step.Barrier;
+import org.apache.tinkerpop.gremlin.process.traversal.step.map.NoOpBarrierStep;
import org.apache.tinkerpop.gremlin.process.traversal.step.sideEffect.ProfileSideEffectStep;
import org.apache.tinkerpop.gremlin.process.traversal.step.util.EmptyStep;
import org.apache.tinkerpop.gremlin.process.traversal.strategy.AbstractTraversalStrategy;
@@ -47,7 +49,7 @@ public final class RemoteStrategy extends AbstractTraversalStrategy<TraversalStr
implements TraversalStrategy.DecorationStrategy {
private static final RemoteStrategy INSTANCE = new RemoteStrategy();
-
+ protected static final int MAX_BARRIER_SIZE = 1000;
private final Optional<RemoteConnection> remoteConnection;
private static final Set<Class<? extends DecorationStrategy>> POSTS = Collections.singleton(VertexProgramStrategy.class);
@@ -90,6 +92,10 @@ public final class RemoteStrategy extends AbstractTraversalStrategy<TraversalStr
if (!(traversal.getParent() instanceof EmptyStep))
return;
+ // given that we send traversers over the wire, try our best to bulk to limit network traffic
+ if (!(traversal.getEndStep() instanceof Barrier))
+ traversal.addStep(new NoOpBarrierStep<>(traversal, MAX_BARRIER_SIZE));
+
// verifications to ensure unsupported steps do not exist in the traversal
if (Boolean.valueOf(System.getProperty("is.testing", "false")) &&
(TraversalHelper.hasStepOfAssignableClassRecursively(ProfileSideEffectStep.class, traversal) ||
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/080306b0/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/optimization/LazyBarrierStrategy.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/optimization/LazyBarrierStrategy.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/optimization/LazyBarrierStrategy.java
index 1f2d758..29c04a7 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/optimization/LazyBarrierStrategy.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/optimization/LazyBarrierStrategy.java
@@ -86,7 +86,7 @@ public final class LazyBarrierStrategy extends AbstractTraversalStrategy<Travers
(((GraphStep) step).getIds().length == 0 && !(step.getNextStep() instanceof HasStep))))) {
if (foundFlatMap && !labeledPath &&
!(step.getNextStep() instanceof Barrier) &&
- (!(step.getNextStep() instanceof EmptyStep) || step.getTraversal().getParent() instanceof EmptyStep)) {
+ !(step.getNextStep() instanceof EmptyStep)) {
final Step noOpBarrierStep = new NoOpBarrierStep<>(traversal, MAX_BARRIER_SIZE);
TraversalHelper.copyLabels(step, noOpBarrierStep, true);
TraversalHelper.insertAfterStep(noOpBarrierStep, step, traversal);
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/080306b0/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/optimization/LazyBarrierStrategyTest.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/optimization/LazyBarrierStrategyTest.java b/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/optimization/LazyBarrierStrategyTest.java
index e1b9a94..f67abd1 100644
--- a/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/optimization/LazyBarrierStrategyTest.java
+++ b/gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/strategy/optimization/LazyBarrierStrategyTest.java
@@ -18,7 +18,6 @@
*/
package org.apache.tinkerpop.gremlin.process.traversal.strategy.optimization;
-import com.sun.org.apache.regexp.internal.RE;
import org.apache.tinkerpop.gremlin.process.traversal.P;
import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
import org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategies;
@@ -88,20 +87,20 @@ public class LazyBarrierStrategyTest {
{__.repeat(__.out()).times(4), __.repeat(__.out()).times(4), Collections.emptyList()},
{__.repeat(__.out()).times(4), __.out().barrier(REPEAT_SIZE).out().barrier(REPEAT_SIZE).out().barrier(REPEAT_SIZE).out().barrier(REPEAT_SIZE), Collections.singletonList(RepeatUnrollStrategy.instance())},
{__.out().out().as("a").select("a").out(), __.out().out().barrier(LAZY_SIZE).as("a").select("a").out(), Collections.emptyList()},
- {__.out().out().as("a").select("a").out(), __.out().out().barrier(LAZY_SIZE).as("a").select("a").barrier(PATH_SIZE).out().barrier(LAZY_SIZE), Collections.singletonList(PathRetractionStrategy.instance())},
- {__.out().out().as("a").out().select("a").out(), __.out().out().barrier(LAZY_SIZE).as("a").out().select("a").barrier(PATH_SIZE).out().barrier(LAZY_SIZE), Collections.singletonList(PathRetractionStrategy.instance())},
- {__.out().out().out().limit(10).out(), __.out().out().barrier(LAZY_SIZE).out().limit(10).out().barrier(LAZY_SIZE), Collections.emptyList()},
+ {__.out().out().as("a").select("a").out(), __.out().out().barrier(LAZY_SIZE).as("a").select("a").barrier(PATH_SIZE).out(), Collections.singletonList(PathRetractionStrategy.instance())},
+ {__.out().out().as("a").out().select("a").out(), __.out().out().barrier(LAZY_SIZE).as("a").out().select("a").barrier(PATH_SIZE).out(), Collections.singletonList(PathRetractionStrategy.instance())},
+ {__.out().out().out().limit(10).out(), __.out().out().barrier(LAZY_SIZE).out().limit(10).out(), Collections.emptyList()},
{__.V().out().in().where(P.neq("a")), __.V().out().barrier(LAZY_SIZE).in().barrier(LAZY_SIZE).where(P.neq("a")), Collections.emptyList()},
{__.V().as("a").out().in().where(P.neq("a")), __.V().as("a").out().in().where(P.neq("a")), Collections.emptyList()},
{__.out().out().in().where(P.neq("a")), __.out().out().barrier(LAZY_SIZE).in().barrier(LAZY_SIZE).where(P.neq("a")), Collections.emptyList()},
{__.out().as("a").out().in().where(P.neq("a")), __.out().as("a").out().in().where(P.neq("a")), Collections.emptyList()},
- {__.out().as("a").out().in().where(P.neq("a")).out().out(), __.out().as("a").out().in().where(P.neq("a")).barrier(LAZY_SIZE).out().barrier(LAZY_SIZE).out().barrier(LAZY_SIZE), Collections.singletonList(PathRetractionStrategy.instance())},
- {__.out().as("a").out().as("b").in().where(P.neq("a")).out().out(), __.out().as("a").out().as("b").in().where(P.neq("a")).barrier(PATH_SIZE).out().barrier(LAZY_SIZE).out().barrier(LAZY_SIZE), Collections.singletonList(PathRetractionStrategy.instance())},
+ {__.out().as("a").out().in().where(P.neq("a")).out().out(), __.out().as("a").out().in().where(P.neq("a")).barrier(LAZY_SIZE).out().barrier(LAZY_SIZE).out(), Collections.singletonList(PathRetractionStrategy.instance())},
+ {__.out().as("a").out().as("b").in().where(P.neq("a")).out().out(), __.out().as("a").out().as("b").in().where(P.neq("a")).barrier(PATH_SIZE).out().barrier(LAZY_SIZE).out(), Collections.singletonList(PathRetractionStrategy.instance())},
{__.out().as("a").out().as("b").in().where(P.neq("a")).out().out(), __.out().as("a").out().as("b").in().where(P.neq("a")).out().out(), Collections.emptyList()},
- {__.out().as("a").out().as("b").in().where(P.neq("a")).out().select("b").out(), __.out().as("a").out().as("b").in().where(P.neq("a")).barrier(PATH_SIZE).out().select("b").barrier(PATH_SIZE).out().barrier(LAZY_SIZE), Collections.singletonList(PathRetractionStrategy.instance())},
- {__.out().as("a").out().as("b").in().where(P.neq("a")).out().select("b").out().out(), __.out().as("a").out().as("b").in().where(P.neq("a")).barrier(PATH_SIZE).out().select("b").barrier(PATH_SIZE).out().barrier(LAZY_SIZE).out().barrier(LAZY_SIZE), Collections.singletonList(PathRetractionStrategy.instance())},
- {__.V().out().out().groupCount().by(__.out().out().out()).out(), __.V().out().barrier(LAZY_SIZE).out().groupCount().by(__.out().out().barrier(LAZY_SIZE).out()).out().barrier(LAZY_SIZE), Collections.emptyList()},
- {__.V().out().out().groupCount().by(__.out().out().out()).out().as("a"), __.V().out().barrier(LAZY_SIZE).out().groupCount().by(__.out().out().barrier(LAZY_SIZE).out()).out().barrier(LAZY_SIZE).as("a"), Collections.emptyList()}
+ {__.out().as("a").out().as("b").in().where(P.neq("a")).out().select("b").out(), __.out().as("a").out().as("b").in().where(P.neq("a")).barrier(PATH_SIZE).out().select("b").barrier(PATH_SIZE).out(), Collections.singletonList(PathRetractionStrategy.instance())},
+ {__.out().as("a").out().as("b").in().where(P.neq("a")).out().select("b").out().out(), __.out().as("a").out().as("b").in().where(P.neq("a")).barrier(PATH_SIZE).out().select("b").barrier(PATH_SIZE).out().barrier(LAZY_SIZE).out(), Collections.singletonList(PathRetractionStrategy.instance())},
+ {__.V().out().out().groupCount().by(__.out().out().out()).out(), __.V().out().barrier(LAZY_SIZE).out().groupCount().by(__.out().out().barrier(LAZY_SIZE).out()).out(), Collections.emptyList()},
+ {__.V().out().out().groupCount().by(__.out().out().out()).out().as("a"), __.V().out().barrier(LAZY_SIZE).out().groupCount().by(__.out().out().barrier(LAZY_SIZE).out()).out().as("a"), Collections.emptyList()}
});
}
}
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/080306b0/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/optimization/Neo4jGraphStepStrategyTest.java
----------------------------------------------------------------------
diff --git a/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/optimization/Neo4jGraphStepStrategyTest.java b/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/optimization/Neo4jGraphStepStrategyTest.java
index 77a2610..1150cf3 100644
--- a/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/optimization/Neo4jGraphStepStrategyTest.java
+++ b/neo4j-gremlin/src/test/java/org/apache/tinkerpop/gremlin/neo4j/process/traversal/strategy/optimization/Neo4jGraphStepStrategyTest.java
@@ -130,7 +130,7 @@ public class Neo4jGraphStepStrategyTest {
{__.V().has("name", P.eq("marko").and(P.eq("bob").or(P.eq("stephen")))).out("knows"),
g_V("name", eq("marko"), "name", P.eq("bob").or(eq("stephen"))).out("knows"), Collections.emptyList()},
///////
- {__.V().out().out().V().has("name", "marko").out(), g_V().out().barrier(LAZY_SIZE).out().barrier(LAZY_SIZE).asAdmin().addStep(V("name", eq("marko"))).barrier(LAZY_SIZE).out().barrier(LAZY_SIZE), Arrays.asList(InlineFilterStrategy.instance(), FilterRankingStrategy.instance(), LazyBarrierStrategy.instance())},
+ {__.V().out().out().V().has("name", "marko").out(), g_V().out().barrier(LAZY_SIZE).out().barrier(LAZY_SIZE).asAdmin().addStep(V("name", eq("marko"))).barrier(LAZY_SIZE).out(), Arrays.asList(InlineFilterStrategy.instance(), FilterRankingStrategy.instance(), LazyBarrierStrategy.instance())},
{__.V().out().out().V().has("name", "marko").as("a").out(), g_V().out().barrier(LAZY_SIZE).out().barrier(LAZY_SIZE).asAdmin().addStep(V("name", eq("marko"))).barrier(LAZY_SIZE).as("a").out(), Arrays.asList(InlineFilterStrategy.instance(), FilterRankingStrategy.instance(), LazyBarrierStrategy.instance())},
{__.V().out().V().has("age", gt(32)).barrier(10).has("name", "marko").as("a"), g_V().out().barrier(LAZY_SIZE).asAdmin().addStep(V("age", gt(32), "name", eq("marko"))).barrier(LAZY_SIZE).barrier(10).as("a"), Arrays.asList(InlineFilterStrategy.instance(), FilterRankingStrategy.instance(), LazyBarrierStrategy.instance())},
{__.V().out().V().has("age", gt(32)).barrier(10).has("name", "marko").as("a"), g_V().out().barrier(LAZY_SIZE).asAdmin().addStep(V("age", gt(32), "name", eq("marko"))).barrier(LAZY_SIZE).barrier(10).as("a"), TraversalStrategies.GlobalCache.getStrategies(Neo4jGraph.class).toList()},
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/080306b0/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/optimization/TinkerGraphStepStrategyTest.java
----------------------------------------------------------------------
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/optimization/TinkerGraphStepStrategyTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/optimization/TinkerGraphStepStrategyTest.java
index c62bd5a..b44d90f 100644
--- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/optimization/TinkerGraphStepStrategyTest.java
+++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/process/traversal/strategy/optimization/TinkerGraphStepStrategyTest.java
@@ -131,7 +131,7 @@ public class TinkerGraphStepStrategyTest {
{__.V().has("name", P.eq("marko").and(P.eq("bob").or(P.eq("stephen")))).out("knows"),
g_V("name", eq("marko"), "name", P.eq("bob").or(eq("stephen"))).out("knows"), Collections.emptyList()},
///////
- {__.V().out().out().V().has("name", "marko").out(), g_V().out().barrier(LAZY_SIZE).out().barrier(LAZY_SIZE).asAdmin().addStep(V("name", eq("marko"))).barrier(LAZY_SIZE).out().barrier(LAZY_SIZE), Arrays.asList(InlineFilterStrategy.instance(), FilterRankingStrategy.instance(), LazyBarrierStrategy.instance())},
+ {__.V().out().out().V().has("name", "marko").out(), g_V().out().barrier(LAZY_SIZE).out().barrier(LAZY_SIZE).asAdmin().addStep(V("name", eq("marko"))).barrier(LAZY_SIZE).out(), Arrays.asList(InlineFilterStrategy.instance(), FilterRankingStrategy.instance(), LazyBarrierStrategy.instance())},
{__.V().out().out().V().has("name", "marko").as("a").out(), g_V().out().barrier(LAZY_SIZE).out().barrier(LAZY_SIZE).asAdmin().addStep(V("name", eq("marko"))).barrier(LAZY_SIZE).as("a").out(), Arrays.asList(InlineFilterStrategy.instance(), FilterRankingStrategy.instance(), LazyBarrierStrategy.instance())},
{__.V().out().V().has("age", gt(32)).barrier(10).has("name", "marko").as("a"), g_V().out().barrier(LAZY_SIZE).asAdmin().addStep(V("age", gt(32), "name", eq("marko"))).barrier(LAZY_SIZE).barrier(10).as("a"), Arrays.asList(InlineFilterStrategy.instance(), FilterRankingStrategy.instance(), LazyBarrierStrategy.instance())},
{__.V().out().V().has("age", gt(32)).barrier(10).has("name", "marko").as("a"), g_V().out().barrier(LAZY_SIZE).asAdmin().addStep(V("age", gt(32), "name", eq("marko"))).barrier(LAZY_SIZE).barrier(10).as("a"), TraversalStrategies.GlobalCache.getStrategies(TinkerGraph.class).toList()},
[4/4] tinkerpop git commit: Merge branch 'tp32'
Posted by ok...@apache.org.
Merge branch 'tp32'
Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/768ade6b
Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/768ade6b
Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/768ade6b
Branch: refs/heads/master
Commit: 768ade6bdc853c556d0a2812005be4796bb261f9
Parents: 9a39bde b717615
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Thu Feb 16 11:09:42 2017 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Thu Feb 16 11:09:42 2017 -0700
----------------------------------------------------------------------
CHANGELOG.asciidoc | 3 ++-
.../upgrade/release-3.2.x-incubating.asciidoc | 26 ++++++++++++++++++++
.../strategy/decoration/RemoteStrategy.java | 1 -
.../optimization/LazyBarrierStrategy.java | 2 +-
.../optimization/LazyBarrierStrategyTest.java | 19 +++++++-------
.../gremlin/server/util/TraverserIterator.java | 21 +++++++++++++---
.../Neo4jGraphStepStrategyTest.java | 2 +-
.../TinkerGraphStepStrategyTest.java | 2 +-
8 files changed, 58 insertions(+), 18 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/768ade6b/CHANGELOG.asciidoc
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/768ade6b/docs/src/upgrade/release-3.2.x-incubating.asciidoc
----------------------------------------------------------------------
[2/4] tinkerpop git commit: moved the bulking logic for GremlinServer
into TraverserIterator. Basically,
at the point of returning traversers over the network,
try and bulk as many as you can to reduce traffic.
Posted by ok...@apache.org.
moved the bulking logic for GremlinServer into TraverserIterator. Basically, at the point of returning traversers over the network, try and bulk as many as you can to reduce traffic.
Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/b635c801
Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/b635c801
Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/b635c801
Branch: refs/heads/master
Commit: b635c801d07aca4d9b62e962f5bf9b2326b732cf
Parents: 080306b
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Thu Feb 16 08:35:51 2017 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Thu Feb 16 08:35:51 2017 -0700
----------------------------------------------------------------------
CHANGELOG.asciidoc | 2 +-
.../strategy/decoration/RemoteStrategy.java | 7 -------
.../gremlin/server/util/TraverserIterator.java | 21 +++++++++++++++++---
3 files changed, 19 insertions(+), 11 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/b635c801/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index cf7312e..30fb41e 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -27,7 +27,7 @@ TinkerPop 3.2.5 (Release Date: NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* Fixed an optimization bug in `LazyBarrierStrategy` around appending barriers to the end of a `Traversal`.
-* `RemoteStrategy` is smart to try and append a `NoOpBarrierStep` in order to bulk traversers prior to network I/O.
+* `TraverserIterator` in GremlinServer is smart to try and bulk traversers prior to network I/O.
[[release-3-2-4]]
TinkerPop 3.2.4 (Release Date: February 8, 2017)
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/b635c801/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/remote/traversal/strategy/decoration/RemoteStrategy.java
----------------------------------------------------------------------
diff --git a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/remote/traversal/strategy/decoration/RemoteStrategy.java b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/remote/traversal/strategy/decoration/RemoteStrategy.java
index 14d3b36..d1b14d5 100644
--- a/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/remote/traversal/strategy/decoration/RemoteStrategy.java
+++ b/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/remote/traversal/strategy/decoration/RemoteStrategy.java
@@ -24,8 +24,6 @@ import org.apache.tinkerpop.gremlin.process.remote.RemoteGraph;
import org.apache.tinkerpop.gremlin.process.remote.traversal.step.map.RemoteStep;
import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
import org.apache.tinkerpop.gremlin.process.traversal.TraversalStrategy;
-import org.apache.tinkerpop.gremlin.process.traversal.step.Barrier;
-import org.apache.tinkerpop.gremlin.process.traversal.step.map.NoOpBarrierStep;
import org.apache.tinkerpop.gremlin.process.traversal.step.sideEffect.ProfileSideEffectStep;
import org.apache.tinkerpop.gremlin.process.traversal.step.util.EmptyStep;
import org.apache.tinkerpop.gremlin.process.traversal.strategy.AbstractTraversalStrategy;
@@ -49,7 +47,6 @@ public final class RemoteStrategy extends AbstractTraversalStrategy<TraversalStr
implements TraversalStrategy.DecorationStrategy {
private static final RemoteStrategy INSTANCE = new RemoteStrategy();
- protected static final int MAX_BARRIER_SIZE = 1000;
private final Optional<RemoteConnection> remoteConnection;
private static final Set<Class<? extends DecorationStrategy>> POSTS = Collections.singleton(VertexProgramStrategy.class);
@@ -92,10 +89,6 @@ public final class RemoteStrategy extends AbstractTraversalStrategy<TraversalStr
if (!(traversal.getParent() instanceof EmptyStep))
return;
- // given that we send traversers over the wire, try our best to bulk to limit network traffic
- if (!(traversal.getEndStep() instanceof Barrier))
- traversal.addStep(new NoOpBarrierStep<>(traversal, MAX_BARRIER_SIZE));
-
// verifications to ensure unsupported steps do not exist in the traversal
if (Boolean.valueOf(System.getProperty("is.testing", "false")) &&
(TraversalHelper.hasStepOfAssignableClassRecursively(ProfileSideEffectStep.class, traversal) ||
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/b635c801/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/util/TraverserIterator.java
----------------------------------------------------------------------
diff --git a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/util/TraverserIterator.java b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/util/TraverserIterator.java
index d4bf5c0..8ea6e55 100644
--- a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/util/TraverserIterator.java
+++ b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/util/TraverserIterator.java
@@ -22,6 +22,8 @@ import org.apache.tinkerpop.gremlin.process.remote.traversal.DefaultRemoteTraver
import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
import org.apache.tinkerpop.gremlin.process.traversal.Traverser;
import org.apache.tinkerpop.gremlin.process.traversal.strategy.decoration.HaltedTraverserStrategy;
+import org.apache.tinkerpop.gremlin.process.traversal.traverser.TraverserRequirement;
+import org.apache.tinkerpop.gremlin.process.traversal.traverser.util.TraverserSet;
import java.util.Iterator;
@@ -32,9 +34,12 @@ public class TraverserIterator implements Iterator<Object> {
private final Traversal.Admin traversal;
private final HaltedTraverserStrategy haltedTraverserStrategy;
+ private final TraverserSet bulker = new TraverserSet();
+ private final int barrierSize;
public TraverserIterator(final Traversal.Admin traversal) {
this.traversal = traversal;
+ this.barrierSize = traversal.getTraverserRequirements().contains(TraverserRequirement.ONE_BULK) ? 1 : 1000;
this.haltedTraverserStrategy = traversal.getStrategies().getStrategy(HaltedTraverserStrategy.class).orElse(
Boolean.valueOf(System.getProperty("is.testing", "false")) ?
HaltedTraverserStrategy.detached() :
@@ -42,17 +47,27 @@ public class TraverserIterator implements Iterator<Object> {
}
public Traversal.Admin getTraversal() {
- return traversal;
+ return this.traversal;
}
@Override
public boolean hasNext() {
- return this.traversal.hasNext();
+ if (this.bulker.isEmpty())
+ this.fillBulker();
+ return !this.bulker.isEmpty();
}
@Override
public Object next() {
- final Traverser.Admin t = this.haltedTraverserStrategy.halt(traversal.nextTraverser());
+ if (this.bulker.isEmpty())
+ this.fillBulker();
+ final Traverser.Admin t = this.haltedTraverserStrategy.halt(this.bulker.remove());
return new DefaultRemoteTraverser<>(t.get(), t.bulk());
}
+
+ private final void fillBulker() {
+ while (this.traversal.hasNext() && this.bulker.size() < this.barrierSize) {
+ this.bulker.add(this.traversal.nextTraverser());
+ }
+ }
}
\ No newline at end of file
[3/4] tinkerpop git commit: Merge branch 'TINKERPOP-1627' into tp32
Posted by ok...@apache.org.
Merge branch 'TINKERPOP-1627' into tp32
Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/b717615d
Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/b717615d
Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/b717615d
Branch: refs/heads/master
Commit: b717615dfe6176413f223316f925c975d81ba1c1
Parents: 4a328b9 b635c80
Author: Marko A. Rodriguez <ok...@gmail.com>
Authored: Thu Feb 16 11:09:32 2017 -0700
Committer: Marko A. Rodriguez <ok...@gmail.com>
Committed: Thu Feb 16 11:09:32 2017 -0700
----------------------------------------------------------------------
CHANGELOG.asciidoc | 3 ++-
.../upgrade/release-3.2.x-incubating.asciidoc | 26 ++++++++++++++++++++
.../strategy/decoration/RemoteStrategy.java | 1 -
.../optimization/LazyBarrierStrategy.java | 2 +-
.../optimization/LazyBarrierStrategyTest.java | 19 +++++++-------
.../gremlin/server/util/TraverserIterator.java | 21 +++++++++++++---
.../Neo4jGraphStepStrategyTest.java | 2 +-
.../TinkerGraphStepStrategyTest.java | 2 +-
8 files changed, 58 insertions(+), 18 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/b717615d/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --cc CHANGELOG.asciidoc
index e3c78f9,30fb41e..de78d24
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@@ -26,8 -26,8 +26,9 @@@ image::https://raw.githubusercontent.co
TinkerPop 3.2.5 (Release Date: NOT OFFICIALLY RELEASED YET)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+* Refactor `SparkContext` handler to support external kill and stop operations.
-
+ * Fixed an optimization bug in `LazyBarrierStrategy` around appending barriers to the end of a `Traversal`.
+ * `TraverserIterator` in GremlinServer is smart to try and bulk traversers prior to network I/O.
[[release-3-2-4]]
TinkerPop 3.2.4 (Release Date: February 8, 2017)