You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by sp...@apache.org on 2016/08/09 17:22:25 UTC
[1/2] tinkerpop git commit: Added a way to allow an OpProcessor to
take configuration from server Settings.
Repository: tinkerpop
Updated Branches:
refs/heads/TINKERPOP-1278 d6f6e29bc -> 4f5b8f9b1
Added a way to allow an OpProcessor to take configuration from server Settings.
Used this new method to init config options for TraversalOpProcessor's cache.
Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/25563531
Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/25563531
Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/25563531
Branch: refs/heads/TINKERPOP-1278
Commit: 25563531182461fbd263e5fb32135f677d35ef1c
Parents: b4c2686
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Tue Aug 9 13:20:43 2016 -0400
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Tue Aug 9 13:20:43 2016 -0400
----------------------------------------------------------------------
CHANGELOG.asciidoc | 1 +
.../src/reference/gremlin-applications.asciidoc | 5 +-
.../tinkerpop/gremlin/server/GremlinServer.java | 6 +--
.../tinkerpop/gremlin/server/OpProcessor.java | 8 +++
.../tinkerpop/gremlin/server/op/OpLoader.java | 14 ++++++
.../op/traversal/TraversalOpProcessor.java | 53 ++++++++++++++++++--
6 files changed, 78 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/25563531/CHANGELOG.asciidoc
----------------------------------------------------------------------
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index b2d3c60..a9a615c 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -28,6 +28,7 @@ TinkerPop 3.2.2 (NOT OFFICIALLY RELEASED YET)
* Removed support for submitting a Java serialized `Traversal` to Gremlin Server.
* Removed a largely internal feature that supported automatic unrolling of traversers in the Gremlin Driver.
+* Made it possible to directly initialize `OpProcessor` implementations with server `Settings`.
* Fixed a potential leak of a `ReferenceCounted` resource in Gremlin Server.
* Added class registrations for `Map.Entry` implementations to `GryoMapper`.
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/25563531/docs/src/reference/gremlin-applications.asciidoc
----------------------------------------------------------------------
diff --git a/docs/src/reference/gremlin-applications.asciidoc b/docs/src/reference/gremlin-applications.asciidoc
index 67ae364..81823ea 100644
--- a/docs/src/reference/gremlin-applications.asciidoc
+++ b/docs/src/reference/gremlin-applications.asciidoc
@@ -355,7 +355,7 @@ $ bin/gremlin-server.sh conf/gremlin-server-modern.yaml
[INFO] GremlinServer - Configuring Gremlin Server from conf/gremlin-server-modern.yaml
[INFO] MetricManager - Configured Metrics Slf4jReporter configured with interval=180000ms and loggerName=org.apache.tinkerpop.gremlin.server.Settings$Slf4jReporterMetrics
-[INFO] Graphs - Graph [graph] was successfully configured via [conf/tinkergraph-empty.properties].
+[INFO] GraphManager - Graph [graph] was successfully configured via [conf/tinkergraph-empty.properties].
[INFO] ServerGremlinExecutor - Initialized Gremlin thread pool. Threads in pool named with pattern gremlin-*
[INFO] ScriptEngines - Loaded gremlin-groovy ScriptEngine
[INFO] GremlinExecutor - Initialized gremlin-groovy ScriptEngine with scripts/generate-modern.groovy
@@ -364,9 +364,12 @@ $ bin/gremlin-server.sh conf/gremlin-server-modern.yaml
[INFO] OpLoader - Adding the standard OpProcessor.
[INFO] OpLoader - Adding the control OpProcessor.
[INFO] OpLoader - Adding the session OpProcessor.
+[INFO] OpLoader - Adding the traversal OpProcessor.
+[INFO] TraversalOpProcessor - Initialized cache for TraversalOpProcessor with size 1000 and expiration time of 600000 ms
[INFO] GremlinServer - Executing start up LifeCycleHook
[INFO] Logger$info - Loading 'modern' graph data.
[INFO] AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo with org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0
+[INFO] AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo-lite with org.apache.tinkerpop.gremlin.driver.ser.GryoLiteMessageSerializerV1d0
[INFO] AbstractChannelizer - Configured application/vnd.gremlin-v1.0+gryo-stringd with org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0
[INFO] GremlinServer$1 - Gremlin Server configured with worker thread pool of 1, gremlin pool of 8 and boss thread pool of 1.
[INFO] GremlinServer$1 - Channel started at port 8182.
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/25563531/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/GremlinServer.java
----------------------------------------------------------------------
diff --git a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/GremlinServer.java b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/GremlinServer.java
index 4d76e30..0b9ffcd 100644
--- a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/GremlinServer.java
+++ b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/GremlinServer.java
@@ -109,10 +109,8 @@ public class GremlinServer {
serverGremlinExecutor = new ServerGremlinExecutor<>(settings, null, workerGroup, EventLoopGroup.class);
gremlinExecutorService = serverGremlinExecutor.getGremlinExecutorService();
- // force an early load of the OpLoader (even if not used by a Channelizer). this is an expensive operation
- // as it uses ServiceLoader and if left uninitialized puts a burden on the first client request that passes
- // through an OpProcessor
- OpLoader.getProcessors();
+ // initialize the OpLoader with configurations being passed to each OpProcessor implementation loaded
+ OpLoader.init(settings);
}
/**
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/25563531/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/OpProcessor.java
----------------------------------------------------------------------
diff --git a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/OpProcessor.java b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/OpProcessor.java
index 27875ac..a99ed77 100644
--- a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/OpProcessor.java
+++ b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/OpProcessor.java
@@ -36,6 +36,14 @@ public interface OpProcessor extends AutoCloseable {
public String getName();
/**
+ * Initialize the {@code OpProcessor} with settings from the server. This method should only be called once at
+ * server startup by a single thread.
+ */
+ public default void init(final Settings settings) {
+ // do nothing by default
+ }
+
+ /**
* Given the context (which contains the RequestMessage), return back a Consumer function that will be
* executed with the context. A typical implementation will simply check the "op" field on the RequestMessage
* and return the Consumer function for that particular operation.
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/25563531/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/op/OpLoader.java
----------------------------------------------------------------------
diff --git a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/op/OpLoader.java b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/op/OpLoader.java
index 96b415c..3ef805f 100644
--- a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/op/OpLoader.java
+++ b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/op/OpLoader.java
@@ -19,6 +19,7 @@
package org.apache.tinkerpop.gremlin.server.op;
import org.apache.tinkerpop.gremlin.server.OpProcessor;
+import org.apache.tinkerpop.gremlin.server.Settings;
import org.apache.tinkerpop.gremlin.server.op.standard.StandardOpProcessor;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -50,6 +51,19 @@ public final class OpLoader {
});
}
+ private static volatile boolean initialized = false;
+
+ /**
+ * Initialize the {@code OpLoader} with server settings. This method should only be called once at startup but is
+ * designed to be idempotent.
+ */
+ public static synchronized void init(final Settings settings) {
+ if (!initialized) {
+ processors.values().forEach(processor -> processor.init(settings));
+ initialized = true;
+ }
+ }
+
/**
* Gets an {@link OpProcessor} by its name. If it cannot be found an {@link Optional#EMPTY} is returned.
*/
http://git-wip-us.apache.org/repos/asf/tinkerpop/blob/25563531/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/op/traversal/TraversalOpProcessor.java
----------------------------------------------------------------------
diff --git a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/op/traversal/TraversalOpProcessor.java b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/op/traversal/TraversalOpProcessor.java
index 914741f..a2df857 100644
--- a/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/op/traversal/TraversalOpProcessor.java
+++ b/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/op/traversal/TraversalOpProcessor.java
@@ -35,6 +35,7 @@ import org.apache.tinkerpop.gremlin.server.Context;
import org.apache.tinkerpop.gremlin.server.GraphManager;
import org.apache.tinkerpop.gremlin.server.GremlinServer;
import org.apache.tinkerpop.gremlin.server.OpProcessor;
+import org.apache.tinkerpop.gremlin.server.Settings;
import org.apache.tinkerpop.gremlin.server.op.AbstractOpProcessor;
import org.apache.tinkerpop.gremlin.server.op.OpProcessorException;
import org.apache.tinkerpop.gremlin.server.util.MetricManager;
@@ -67,10 +68,37 @@ public class TraversalOpProcessor extends AbstractOpProcessor {
public static final String OP_PROCESSOR_NAME = "traversal";
public static final Timer traversalOpTimer = MetricManager.INSTANCE.getTimer(name(GremlinServer.class, "op", "traversal"));
- private static final Cache<UUID, TraversalSideEffects> cache = Caffeine.newBuilder()
- .expireAfterWrite(10, TimeUnit.MINUTES)
- .maximumSize(10_000)
- .build();
+ public static final Settings.ProcessorSettings DEFAULT_SETTINGS = new Settings.ProcessorSettings();
+
+ /**
+ * Configuration setting for how long a cached side-effect will be available before it is evicted from the cache.
+ */
+ public static final String CONFIG_CACHE_EXPIRATION_TIME = "cacheExpirationTime";
+
+ /**
+ * Default timeout for a cached side-effect is ten minutes.
+ */
+ public static final long DEFAULT_CACHE_EXPIRATION_TIME = 600000;
+
+ /**
+ * Configuration setting for the maximum number of entries the cache will have.
+ */
+ public static final String CONFIG_CACHE_MAX_SIZE = "cacheMaxSize";
+
+ /**
+ * Default size of the max size of the cache.
+ */
+ public static final long DEFAULT_CACHE_MAX_SIZE = 1000;
+
+ static {
+ DEFAULT_SETTINGS.className = TraversalOpProcessor.class.getCanonicalName();
+ DEFAULT_SETTINGS.config = new HashMap<String, Object>() {{
+ put(CONFIG_CACHE_EXPIRATION_TIME, DEFAULT_CACHE_EXPIRATION_TIME);
+ put(CONFIG_CACHE_MAX_SIZE, DEFAULT_CACHE_MAX_SIZE);
+ }};
+ }
+
+ private static Cache<UUID, TraversalSideEffects> cache = null;
public TraversalOpProcessor() {
super(true);
@@ -87,6 +115,23 @@ public class TraversalOpProcessor extends AbstractOpProcessor {
}
@Override
+ public void init(final Settings settings) {
+ final Settings.ProcessorSettings processorSettings = settings.processors.stream()
+ .filter(p -> p.className.equals(TraversalOpProcessor.class.getCanonicalName()))
+ .findAny().orElse(TraversalOpProcessor.DEFAULT_SETTINGS);
+ final long maxSize = Long.parseLong(processorSettings.config.get(TraversalOpProcessor.CONFIG_CACHE_MAX_SIZE).toString());
+ final long expirationTime = Long.parseLong(processorSettings.config.get(TraversalOpProcessor.CONFIG_CACHE_EXPIRATION_TIME).toString());
+
+ cache = Caffeine.newBuilder()
+ .expireAfterWrite(expirationTime, TimeUnit.MILLISECONDS)
+ .maximumSize(maxSize)
+ .build();
+
+ logger.info("Initialized cache for {} with size {} and expiration time of {} ms",
+ TraversalOpProcessor.class.getSimpleName(), maxSize, expirationTime);
+ }
+
+ @Override
public ThrowingConsumer<Context> select(final Context ctx) throws OpProcessorException {
final RequestMessage message = ctx.getRequestMessage();
logger.debug("Selecting processor for RequestMessage {}", message);
[2/2] tinkerpop git commit: Merge remote-tracking branch
'origin/TINKERPOP-1278' into TINKERPOP-1278
Posted by sp...@apache.org.
Merge remote-tracking branch 'origin/TINKERPOP-1278' into TINKERPOP-1278
Project: http://git-wip-us.apache.org/repos/asf/tinkerpop/repo
Commit: http://git-wip-us.apache.org/repos/asf/tinkerpop/commit/4f5b8f9b
Tree: http://git-wip-us.apache.org/repos/asf/tinkerpop/tree/4f5b8f9b
Diff: http://git-wip-us.apache.org/repos/asf/tinkerpop/diff/4f5b8f9b
Branch: refs/heads/TINKERPOP-1278
Commit: 4f5b8f9b152ea4615ed1fdbd00af0af873acd373
Parents: 2556353 d6f6e29
Author: Stephen Mallette <sp...@genoprime.com>
Authored: Tue Aug 9 13:22:14 2016 -0400
Committer: Stephen Mallette <sp...@genoprime.com>
Committed: Tue Aug 9 13:22:14 2016 -0400
----------------------------------------------------------------------
.../gremlin/process/traversal/Translator.java | 4 ++++
.../gremlin/python/jsr223/PythonProvider.java | 6 +-----
.../jsr223/TinkerGraphGroovyTranslatorProvider.java | 6 +-----
.../jsr223/TinkerGraphJavaTranslatorProvider.java | 16 +---------------
.../structure/io/graphson/GraphSONTranslator.java | 5 ++---
.../TinkerGraphGraphSONTranslatorProvider.java | 10 ++--------
6 files changed, 11 insertions(+), 36 deletions(-)
----------------------------------------------------------------------