You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@tinkerpop.apache.org by sp...@apache.org on 2026/08/20 18:29:39 UTC
(tinkerpop) branch tinkergraph-storage updated (1b16518103 -> 819bd8c17c)
This is an automated email from the ASF dual-hosted git repository.
spmallette pushed a change to branch tinkergraph-storage
in repository https://gitbox.apache.org/repos/asf/tinkerpop.git
from 1b16518103 Document TinkerStorageGraph durability, compaction, and locking
new ce004eec0b Extract AbstractLogStorage base from GraphBinaryStorage
new dc72ba579a Encode TinkerStorageGraph elements as dictionary-referenced components
new 7832c81a9c Add opt-in vertex-property id persistence to TinkerStorageGraph storage
new a7273bc473 Document TinkerStorageGraph compact encoding and vertex-property id option
new f32a598ea3 Broaden TinkerStorage TCK: value types, ids, varints, unicode round-trips
new 379deede92 Test TinkerStorage durability, lifecycle, and dictionary crash window
new 7788cb2bd1 Reject corrupt TinkerStorage frames with a clear error
new 73dbe40cb2 Document TinkerStorageGraph single-writer stance; enforce meta-property conflict
new 819bd8c17c Document TinkerStorageGraph persistence and fix stale class doc
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGELOG.asciidoc | 2 +-
.../reference/implementations-tinkergraph.asciidoc | 26 +
...o.properties => tinkergraph-storage.properties} | 4 +
.../conf/tinkergraph-credentials.properties | 3 +
.../conf/tinkerstoragegraph-persistent.properties | 40 ++
.../gremlin/tinkergraph/structure/TinkerGraph.java | 8 +
.../tinkergraph/structure/TinkerStorageGraph.java | 20 +-
...hBinaryStorage.java => AbstractLogStorage.java} | 442 +++++-------
.../structure/storage/GraphBinaryStorage.java | 777 +++++++++------------
.../structure/TinkerStorageGraphTest.java | 8 +-
.../AbstractTinkerStorageConformanceTest.java | 318 +++++++++
.../structure/storage/GraphBinaryStorageTest.java | 202 +++++-
.../storage/StorageCrashConsistencyTest.java | 58 +-
13 files changed, 1152 insertions(+), 756 deletions(-)
rename gremlin-console/conf/{tinkergraph-gryo.properties => tinkergraph-storage.properties} (69%)
create mode 100644 gremlin-server/conf/tinkerstoragegraph-persistent.properties
copy tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/{GraphBinaryStorage.java => AbstractLogStorage.java} (64%)
(tinkerpop) 09/09: Document TinkerStorageGraph persistence and fix stale class doc
Posted by sp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
spmallette pushed a commit to branch tinkergraph-storage
in repository https://gitbox.apache.org/repos/asf/tinkerpop.git
commit 819bd8c17cf642fe275b6b8392d6be6af480788d
Author: Stephen Mallette <st...@amazon.com>
AuthorDate: Thu Aug 20 18:26:32 2026 +0000
Document TinkerStorageGraph persistence and fix stale class doc
Rewrite the TinkerStorageGraph class Javadoc, which still described disk
storage as planned future work. Add a commented, persistence-enabled server
sample (tinkerstoragegraph-persistent.properties) and a Gremlin Server
subsection in the persistence reference. Rename the misnamed console
tinkergraph-gryo.properties to tinkergraph-storage.properties and comment the
credentials sample.
Assisted-by: Claude Code:claude-opus-4-8
---
.../reference/implementations-tinkergraph.asciidoc | 18 ++++++++++
...o.properties => tinkergraph-storage.properties} | 4 +++
.../conf/tinkergraph-credentials.properties | 3 ++
.../conf/tinkerstoragegraph-persistent.properties | 40 ++++++++++++++++++++++
.../tinkergraph/structure/TinkerStorageGraph.java | 14 ++++++--
5 files changed, 76 insertions(+), 3 deletions(-)
diff --git a/docs/src/reference/implementations-tinkergraph.asciidoc b/docs/src/reference/implementations-tinkergraph.asciidoc
index 7f7db73fbe..6b3a8d086c 100644
--- a/docs/src/reference/implementations-tinkergraph.asciidoc
+++ b/docs/src/reference/implementations-tinkergraph.asciidoc
@@ -471,6 +471,24 @@ the storage format version it was written with. Opening a store written in a for
with a clear error rather than misreading the data. There is no in-place format migration. To move a graph across an
incompatible storage format, export it with the `io()` step before upgrading and read it back afterward.
+Gremlin Server persists a graph the same way. A graph in the server's `graphs` configuration is pointed at a
+properties file that sets the storage keys, and the server opens that graph at startup, commits to it as clients
+request, and closes it on shutdown. A graph configured without a storage engine remains transactional but in-memory,
+so its data is lost when the server stops.
+
+[source,properties]
+----
+gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph
+gremlin.tinkergraph.storage=graphbinary
+gremlin.tinkergraph.graphLocation=/data/mygraph
+----
+
+[source,yaml]
+----
+graphs: {
+ graph: conf/tinkerstoragegraph-persistent.properties }
+----
+
Persistence is distinct from interchange. `TinkerMemoryGraph` is purely in-memory and does not persist. To move data in
or out of any TinkerGraph in an interchange format such as GraphML, GraphSON, or Gryo, use the `io()` step directly:
diff --git a/gremlin-console/conf/tinkergraph-gryo.properties b/gremlin-console/conf/tinkergraph-storage.properties
similarity index 69%
rename from gremlin-console/conf/tinkergraph-gryo.properties
rename to gremlin-console/conf/tinkergraph-storage.properties
index 312dc3f9b6..2a04bb7d37 100644
--- a/gremlin-console/conf/tinkergraph-gryo.properties
+++ b/gremlin-console/conf/tinkergraph-storage.properties
@@ -15,7 +15,11 @@
# specific language governing permissions and limitations
# under the License.
+# Sample configuration for a durable, transactional TinkerStorageGraph. Opening a graph with this configuration
+# (for example via GraphFactory) produces a TinkerStorageGraph that persists committed transactions to graphLocation.
gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph
+# built-in storage engine; without this key the graph is transactional but in-memory only
gremlin.tinkergraph.storage=graphbinary
+# directory holding the durable data (created if absent; a location may be opened by only one graph at a time)
gremlin.tinkergraph.graphLocation=/tmp/tinkergraph
diff --git a/gremlin-server/conf/tinkergraph-credentials.properties b/gremlin-server/conf/tinkergraph-credentials.properties
index 4597fabcd3..fb4225b376 100644
--- a/gremlin-server/conf/tinkergraph-credentials.properties
+++ b/gremlin-server/conf/tinkergraph-credentials.properties
@@ -16,5 +16,8 @@
# under the License.
gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph
gremlin.tinkergraph.vertexIdManager=LONG
+# This credential store is an in-memory TinkerGraph. TinkerGraph no longer auto-loads from disk on open, so
+# SimpleAuthenticator reads the store explicitly from graphLocation using graphFormat at startup. (A durable
+# TinkerStorageGraph, by contrast, manages its own persistence and needs no graphFormat.)
gremlin.tinkergraph.graphLocation=data/credentials.kryo
gremlin.tinkergraph.graphFormat=gryo
\ No newline at end of file
diff --git a/gremlin-server/conf/tinkerstoragegraph-persistent.properties b/gremlin-server/conf/tinkerstoragegraph-persistent.properties
new file mode 100644
index 0000000000..dac2460298
--- /dev/null
+++ b/gremlin-server/conf/tinkerstoragegraph-persistent.properties
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Sample configuration for a durable, transactional TinkerStorageGraph on Gremlin Server. Reference it from the
+# server's "graphs" block, for example in gremlin-server-transaction.yaml:
+# graphs: { graph: conf/tinkerstoragegraph-persistent.properties }
+gremlin.graph=org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph
+
+# Enables durable persistence. "graphbinary" is the built-in storage engine; the fully qualified class name of a
+# custom TinkerStorage implementation may be used instead. Without this key the graph is transactional but in-memory.
+gremlin.tinkergraph.storage=graphbinary
+# Directory holding the durable data. Required when a storage engine is set. It is created if absent and may be
+# opened by only one graph at a time (single writer, whether in this JVM or another process).
+gremlin.tinkergraph.graphLocation=/tmp/tinkerstoragegraph
+
+gremlin.tinkergraph.vertexIdManager=LONG
+gremlin.tinkergraph.edgeIdManager=LONG
+gremlin.tinkergraph.vertexPropertyIdManager=LONG
+
+# Optional storage tuning (defaults shown, uncomment to change):
+# "commit" forces each transaction to disk (survives OS crash/power loss); "os" is faster but survives only a JVM crash
+#gremlin.tinkergraph.storage.sync=commit
+# auto-compact the append log once it grows past this many bytes (default 67108864 = 64MB; 0 disables)
+#gremlin.tinkergraph.storage.compactThreshold=67108864
+# persist auto-generated vertex-property ids across a reopen (default false)
+#gremlin.tinkergraph.storage.preserveVertexPropertyIds=false
diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java
index 803cc27dcf..ae9fcb2b48 100644
--- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java
+++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java
@@ -49,9 +49,17 @@ import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
/**
- * The transactional implementation of the {@link TinkerGraph} interface, in-memory with optional persistence on
- * calls to {@link #close()}. It is planned that this implementation will optionally support simple storage to disk
- * built on its transaction functionality.
+ * The transactional implementation of the {@link TinkerGraph} interface. It provides {@code read committed}
+ * transaction isolation with optimistic locking and, when a storage engine is configured, durable persistence to
+ * disk. With no storage engine configured it is an in-memory transactional graph that retains nothing across
+ * restarts.
+ * <p/>
+ * Persistence is pluggable through the {@link org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.TinkerStorage}
+ * SPI and enabled with the {@code gremlin.tinkergraph.storage} and {@code gremlin.tinkergraph.graphLocation}
+ * configuration keys. Each committed transaction is written through to the storage engine before the in-memory commit
+ * is applied, and reopening the same location replays the persisted commits to rebuild the graph. A storage location
+ * is single-writer: it is guarded by an exclusive {@link org.apache.tinkerpop.gremlin.tinkergraph.structure.storage.DirectoryLock}
+ * so a second open of the same directory fails rather than corrupting the data.
*
* @author Valentyn Kahamlyk
*/
(tinkerpop) 07/09: Reject corrupt TinkerStorage frames with a clear error
Posted by sp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
spmallette pushed a commit to branch tinkergraph-storage
in repository https://gitbox.apache.org/repos/asf/tinkerpop.git
commit 7788cb2bd14e6dd0e7238e385bb57f920b2a0e3f
Author: Stephen Mallette <st...@amazon.com>
AuthorDate: Wed Aug 19 16:36:57 2026 +0000
Reject corrupt TinkerStorage frames with a clear error
Surface an unknown value type code as a Corrupt storage IOException instead of
an opaque NullPointerException, matching the codec's other integrity checks.
Add decode-path tests for the four malformed-frame cases that clear CRC framing
but are internally invalid: unknown op code, dictionary append gap, dictionary
id redefinition, and unknown value type code.
Assisted-by: Claude Code:claude-opus-4-8
---
.../structure/storage/GraphBinaryStorage.java | 5 +-
.../structure/storage/GraphBinaryStorageTest.java | 69 ++++++++++++++++++++++
2 files changed, 73 insertions(+), 1 deletion(-)
diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java
index 7c6d0cff77..e99e3c8192 100644
--- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java
+++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java
@@ -391,7 +391,10 @@ public final class GraphBinaryStorage extends AbstractLogStorage {
@SuppressWarnings({"unchecked", "rawtypes"})
private Object readScalar(final ByteBufferBuffer buf) throws IOException {
- final DataType dataType = DataType.get(Byte.toUnsignedInt(buf.readByte()));
+ final int code = Byte.toUnsignedInt(buf.readByte());
+ final DataType dataType = DataType.get(code);
+ if (dataType == null)
+ throw new IOException(String.format("Corrupt storage: unknown value type code 0x%02X", code));
if (dataType == DataType.UNSPECIFIED_NULL)
return null;
final TypeSerializer serializer = registry.getSerializer(dataType);
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java
index f5347f0920..bcb8605ba2 100644
--- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java
+++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java
@@ -22,15 +22,20 @@ import org.apache.commons.configuration2.Configuration;
import org.apache.tinkerpop.gremlin.structure.T;
import org.apache.tinkerpop.gremlin.structure.Vertex;
import org.apache.tinkerpop.gremlin.structure.VertexProperty;
+import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge;
+import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph;
import org.junit.Test;
import java.io.DataInputStream;
import java.io.File;
+import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.file.Files;
+import java.util.HashMap;
import java.util.Iterator;
+import java.util.Map;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
@@ -503,6 +508,70 @@ public class GraphBinaryStorageTest extends AbstractTinkerStorageConformanceTest
}
}
+ // --- decode-error paths -----------------------------------------------------------------------------------------
+ // A frame that clears CRC framing can still be internally malformed (a bug in an older writer, or a targeted flip
+ // that happens to keep the checksum valid). The codec must reject each such frame with a clear IOException rather
+ // than a silent wrong answer or an opaque runtime crash. These drive decodeFrame directly with hand-built frames;
+ // values below 128 are single-byte varints, so the frames are written as literal bytes.
+
+ private static final byte OP_DEL_VERTEX = 2;
+ private static final byte OP_DICT_APPEND = 5;
+
+ /** Decode one frame payload against a fresh codec, returning nothing — the maps are throwaway. */
+ private static void decode(final GraphBinaryStorage codec, final byte[] frame) throws IOException {
+ final Map<Object, DetachedVertex> vertices = new HashMap<>();
+ final Map<Object, DetachedEdge> edges = new HashMap<>();
+ codec.decodeFrame(frame, vertices, edges);
+ }
+
+ @Test
+ public void shouldRejectFrameWithUnknownOpCode() {
+ // entryCount=1, op=99 (no such op)
+ try {
+ decode(new GraphBinaryStorage(), new byte[]{ 1, 99 });
+ fail("expected an IOException for an unknown op code");
+ } catch (final IOException expected) {
+ assertTrue(expected.getMessage(), expected.getMessage().contains("Unknown storage op code"));
+ }
+ }
+
+ @Test
+ public void shouldRejectDictionaryAppendGap() {
+ // entryCount=1, OP_DICT_APPEND, id=5 into an empty dictionary (expected next id is 0) -> gap
+ try {
+ decode(new GraphBinaryStorage(), new byte[]{ 1, OP_DICT_APPEND, 5, 1, (byte) 'x' });
+ fail("expected an IOException for a dictionary append gap");
+ } catch (final IOException expected) {
+ assertTrue(expected.getMessage(), expected.getMessage().contains("dictionary append gap"));
+ }
+ }
+
+ @Test
+ public void shouldRejectDictionaryRedefinition() {
+ // append id 0 = "a", then re-append id 0 = "b" on the same codec -> redefinition (a mismatch, not an
+ // idempotent re-append of the same string)
+ final GraphBinaryStorage codec = new GraphBinaryStorage();
+ try {
+ decode(codec, new byte[]{ 1, OP_DICT_APPEND, 0, 1, (byte) 'a' });
+ decode(codec, new byte[]{ 1, OP_DICT_APPEND, 0, 1, (byte) 'b' });
+ fail("expected an IOException for a redefined dictionary id");
+ } catch (final IOException expected) {
+ assertTrue(expected.getMessage(), expected.getMessage().contains("redefined"));
+ }
+ }
+
+ @Test
+ public void shouldRejectUnknownValueTypeCode() {
+ // OP_DEL_VERTEX reads a scalar id; feed it a value type code (0xFF) no serializer claims. Without a guard this
+ // is an opaque NullPointerException; the codec must surface it as corruption instead.
+ try {
+ decode(new GraphBinaryStorage(), new byte[]{ 1, OP_DEL_VERTEX, (byte) 0xFF });
+ fail("expected an IOException for an unknown value type code");
+ } catch (final IOException expected) {
+ assertTrue(expected.getMessage(), expected.getMessage().contains("unknown value type code"));
+ }
+ }
+
private static String rootMessage(final Throwable t) {
Throwable cur = t;
while (cur.getCause() != null && cur.getCause() != cur)
(tinkerpop) 08/09: Document TinkerStorageGraph single-writer stance; enforce meta-property conflict
Posted by sp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
spmallette pushed a commit to branch tinkergraph-storage
in repository https://gitbox.apache.org/repos/asf/tinkerpop.git
commit 73dbe40cb2d855be2afc54bfb9f92a9e62a8d584
Author: Stephen Mallette <st...@amazon.com>
AuthorDate: Thu Aug 20 15:07:03 2026 +0000
Document TinkerStorageGraph single-writer stance; enforce meta-property conflict
Add a comment explaining why supportsConcurrentAccess() is false: a persistent
TinkerStorageGraph is a single-writer store (DirectoryLock), and the feature
denotes multiple connections/instances sharing the same data, not the
intra-instance multi-thread transaction access the graph already provides.
Re-enable the stale commented-out fail() in the concurrent meta-property test
and strengthen its assertion so it proves the losing transaction rolled back,
turning a decorative test into a real guard for conflict detection.
Assisted-by: Claude Code:claude-opus-4-8
---
.../gremlin/tinkergraph/structure/TinkerStorageGraph.java | 6 ++++++
.../gremlin/tinkergraph/structure/TinkerStorageGraphTest.java | 8 +++++---
2 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java
index 0eced39af1..803cc27dcf 100644
--- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java
+++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraph.java
@@ -545,6 +545,12 @@ public final class TinkerStorageGraph extends AbstractTinkerGraph {
private TinkerGraphGraphFeatures() {
}
+ /**
+ * A persistent {@link TinkerStorageGraph} is a single-writer store: {@code DirectoryLock} permits only one
+ * graph instance to open a given storage directory at a time. This feature denotes multiple connections /
+ * instances sharing the same data — not the intra-instance, multi-thread transaction access that the
+ * thread-local {@link TinkerTransaction} already provides — so it is {@code false}.
+ */
@Override
public boolean supportsConcurrentAccess() {
return false;
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraphTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraphTest.java
index af5e79124a..67bb6bffdd 100644
--- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraphTest.java
+++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerStorageGraphTest.java
@@ -1415,14 +1415,16 @@ public class TinkerStorageGraphTest {
try {
gtx.tx().commit();
- //fail("should throw TransactionException");
+ fail("should throw TransactionException");
} catch (TransactionException e) {
}
- // should be only 1 vertex with updated property
+ // tx2 committed first and wins; tx1 conflicted and rolled back, so its meta1 was never persisted
assertEquals(1L, (long) gtx.V().count().next());
- assertEquals("tx2", gtx.V(v1.id()).properties("test").values("meta1", "meta2").next());
+ assertEquals(1L, (long) gtx.V(v1.id()).properties("test").properties().count().next());
+ assertEquals("tx2", gtx.V(v1.id()).properties("test").values("meta2").next());
+ assertFalse(gtx.V(v1.id()).properties("test").values("meta1").hasNext());
}
@Test
(tinkerpop) 04/09: Document TinkerStorageGraph compact encoding and vertex-property id option
Posted by sp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
spmallette pushed a commit to branch tinkergraph-storage
in repository https://gitbox.apache.org/repos/asf/tinkerpop.git
commit a7273bc473a93f6e9c7bdc4a4af99713ef125ee9
Author: Stephen Mallette <st...@amazon.com>
AuthorDate: Wed Aug 19 14:41:17 2026 +0000
Document TinkerStorageGraph compact encoding and vertex-property id option
Add gremlin.tinkergraph.storage.preserveVertexPropertyIds to the TinkerGraph
configuration table and note in the persistence section that element and edge
ids are always preserved on reopen while auto-generated vertex-property ids are
regenerated by default. Fold the compact dictionary encoding and the new
option into the storage CHANGELOG entry.
Assisted-by: Claude Code:claude-opus-4-8
---
CHANGELOG.asciidoc | 2 +-
docs/src/reference/implementations-tinkergraph.asciidoc | 8 ++++++++
2 files changed, 9 insertions(+), 1 deletion(-)
diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc
index 5e7a7fa022..73da9ca801 100644
--- a/CHANGELOG.asciidoc
+++ b/CHANGELOG.asciidoc
@@ -28,7 +28,7 @@ image::https://raw.githubusercontent.com/apache/tinkerpop/master/docs/static/ima
* Fixed `gremlin-go` to report a malformed or truncated GraphBinary response as a deserialization error rather than a bare decoder message.
* Made `TinkerGraph` an interface and renamed the in-memory implementation to `TinkerMemoryGraph`; `TinkerGraph.open()` and `gremlin.graph=...TinkerGraph` behave as before. *(breaking)*
* Renamed `TinkerTransactionGraph` to `TinkerStorageGraph`. *(breaking)*
-* Added a pluggable storage layer to `TinkerStorageGraph` that durably persists each committed transaction to disk, selected with the `gremlin.tinkergraph.storage` config key and shipping a GraphBinary engine, with a `gremlin.tinkergraph.storage.sync` key to choose `commit` (fsync per commit) or `os` durability; a storage location is locked to a single writer, so opening one already in use fails fast; the append log auto-compacts once it exceeds `gremlin.tinkergraph.storage.compactThresh [...]
+* Added a pluggable storage layer to `TinkerStorageGraph` that durably persists each committed transaction to disk, selected with the `gremlin.tinkergraph.storage` config key and shipping a GraphBinary engine, with a `gremlin.tinkergraph.storage.sync` key to choose `commit` (fsync per commit) or `os` durability; a storage location is locked to a single writer, so opening one already in use fails fast; the append log auto-compacts once it exceeds `gremlin.tinkergraph.storage.compactThresh [...]
* Removed automatic persistence from `TinkerMemoryGraph`, which is now purely in-memory and ignores `gremlin.tinkergraph.graphLocation`/`graphFormat`. Use `TinkerStorageGraph` for durability or `g.io()` for interchange. *(breaking)*
[[release-4-0-0-beta-3]]
diff --git a/docs/src/reference/implementations-tinkergraph.asciidoc b/docs/src/reference/implementations-tinkergraph.asciidoc
index b657d0c9f4..7f7db73fbe 100644
--- a/docs/src/reference/implementations-tinkergraph.asciidoc
+++ b/docs/src/reference/implementations-tinkergraph.asciidoc
@@ -186,6 +186,10 @@ may be lost on an operating system crash or power loss. Only meaningful when `gr
automatically compacts its append log, bounding the log growth and restart time of a long-running graph that is never
explicitly closed. Defaults to `67108864` (64 MB). A value of `0` disables automatic compaction, leaving it to
`close()` or an explicit `compact()`. Only meaningful when `gremlin.tinkergraph.storage` is set.
+|gremlin.tinkergraph.storage.preserveVertexPropertyIds |Whether a `TinkerStorageGraph` storage engine persists
+auto-generated vertex-property ids so they are stable across a close and reopen. Defaults to `false`, which
+regenerates those ids on load and keeps the store smaller. Element and edge ids are always preserved regardless. Only
+meaningful when `gremlin.tinkergraph.storage` is set.
|=========================================================
NOTE: To use <<tinkergraph-gremlin-tx, transactions>> and <<tinkergraph-gremlin-persistence, persistence>>, configure
@@ -456,6 +460,10 @@ committed state. Compaction runs when the graph is closed and can be requested e
unbounded log, compaction also runs automatically once the log grows past `gremlin.tinkergraph.storage.compactThreshold`
bytes. Setting that threshold to `0` disables automatic compaction.
+Element and edge ids are always preserved across a reopen. Auto-generated vertex-property ids are not, by default,
+so a vertex property may receive a different id after a reopen. Setting `gremlin.tinkergraph.storage.preserveVertexPropertyIds`
+to `true` persists those ids as well, at the cost of a larger store.
+
A storage location may be opened by only one graph at a time. `TinkerStorageGraph` takes an exclusive lock on the
storage directory when it opens, so a second attempt to open the same location, whether from the same JVM or another
process, fails rather than corrupting the data. The lock is released when the graph is closed. A store also records
(tinkerpop) 01/09: Extract AbstractLogStorage base from GraphBinaryStorage
Posted by sp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
spmallette pushed a commit to branch tinkergraph-storage
in repository https://gitbox.apache.org/repos/asf/tinkerpop.git
commit ce004eec0b48246f02c70dda3c9fd0e5d79dc856
Author: Stephen Mallette <st...@amazon.com>
AuthorDate: Wed Aug 19 14:16:14 2026 +0000
Extract AbstractLogStorage base from GraphBinaryStorage
Move the codec-agnostic log-structured machinery — file layout, VERSION marker,
CRC framing, replay fold, SyncMode durability, and crash-safe + threshold
compaction — into a new abstract AbstractLogStorage. GraphBinaryStorage now
supplies only the element codec through four hooks (encodeCommit, decodeFrame,
writeSnapshot, beginReplay). Pure refactor: on-disk format and behavior are
unchanged, so a new codec can be built by overriding the hooks alone.
Assisted-by: Claude Code:claude-opus-4-8
---
...hBinaryStorage.java => AbstractLogStorage.java} | 442 +++++++-----------
.../structure/storage/GraphBinaryStorage.java | 512 ++-------------------
2 files changed, 202 insertions(+), 752 deletions(-)
diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractLogStorage.java
similarity index 63%
copy from tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java
copy to tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractLogStorage.java
index 2aaa7643cc..b2bdf249f5 100644
--- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java
+++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractLogStorage.java
@@ -19,20 +19,15 @@
package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage;
import org.apache.commons.configuration2.Configuration;
-import org.apache.tinkerpop.gremlin.structure.Edge;
-import org.apache.tinkerpop.gremlin.structure.Vertex;
-import org.apache.tinkerpop.gremlin.structure.io.binary.GraphBinaryReader;
-import org.apache.tinkerpop.gremlin.structure.io.binary.GraphBinaryWriter;
-import org.apache.tinkerpop.gremlin.structure.io.binary.TypeSerializerRegistry;
import org.apache.tinkerpop.gremlin.structure.util.Attachable;
import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge;
-import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedFactory;
import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.AbstractTinkerGraph;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerEdge;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex;
+import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.DataInputStream;
import java.io.DataOutputStream;
@@ -42,7 +37,6 @@ import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
-import java.io.OutputStream;
import java.io.UncheckedIOException;
import java.nio.channels.FileChannel;
import java.nio.file.AtomicMoveNotSupportedException;
@@ -51,40 +45,24 @@ import java.nio.file.StandardCopyOption;
import java.nio.file.StandardOpenOption;
import java.util.Arrays;
import java.util.Collection;
-import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.Map;
import java.util.zip.CRC32;
/**
- * A durable {@link TinkerStorage} engine that persists a {@code TinkerStorageGraph} as an append-only commit log
- * ("write-ahead log") serialized with GraphBinary. On each committed transaction the changeset is appended to
- * {@code log.gbin} as a single record; on open the optional {@code snapshot.gbin} is read followed by the log, with the
- * folded result re-applied to the in-memory graph. {@link #compact(AbstractTinkerGraph)} rewrites the snapshot from the
- * current committed state and truncates the log.
+ * Log-structured durable storage machinery shared by {@link TinkerStorage} engines, independent of how an element is
+ * encoded. It persists a {@code TinkerStorageGraph} as an append-only commit log ({@code log.gbin}) plus an optional
+ * folded {@code snapshot.gbin}; on open the snapshot is read followed by the log, last-write-wins per element id.
* <p/>
- * On commit the appended record is made durable according to the configured {@link SyncMode}: {@link SyncMode#COMMIT}
- * (default) {@code fsync}s so the commit survives an OS crash or power loss, while {@link SyncMode#OS} only flushes to
- * the operating system.
+ * This base owns everything that is not the element codec: the on-disk file layout, the single-source-of-truth
+ * {@code VERSION} marker, length+CRC frame framing (which tells an interrupted trailing append apart from genuine
+ * corruption), the replay fold loop, durability via {@link SyncMode}, and crash-safe atomic compaction with a
+ * size threshold. Concrete engines supply only the codec through {@link #encodeCommit}, {@link #decodeFrame},
+ * {@link #writeSnapshot}, and (optionally) {@link #beginReplay} for per-replay decode state.
* <p/>
- * On-disk compatibility: the store's format version is recorded once in a {@code VERSION} marker file (magic +
- * version), which is the single source of truth even when a store momentarily holds a snapshot and a log written at
- * different times. Individual files carry only the magic for identity/corruption detection. Opening a store whose
- * marker names an unsupported version fails loudly — records are never misread across a format change. The engine
- * keeps this simple by not attempting in-place migration: the supported path across an incompatible format bump is to
- * export via {@code g.io()} before upgrading. Additive changes should prefer new record opcodes; unknown opcodes are
- * a hard error (a durable store must not silently drop records it cannot parse), so a genuinely incompatible change
- * bumps the version.
- * <p/>
- * The in-memory graph remains authoritative (write-through). This engine does not support graphs larger than memory.
- * <p/>
- * Known limitation (write amplification): a commit records each changed element in full — a single property change on
- * a large element rewrites the whole element to the log. Elements are typically small and automatic compaction bounds
- * the resulting log growth, so this is accepted rather than mitigated with per-property deltas, which would complicate
- * the {@link TinkerStorageMutation} contract and the replay fold. The snapshot, by contrast, is streamed one element
- * at a time so compaction never holds a second full copy of the graph in heap.
+ * The in-memory graph remains authoritative (write-through); this machinery does not support graphs larger than memory.
*/
-public final class GraphBinaryStorage implements TinkerStorage {
+public abstract class AbstractLogStorage implements TinkerStorage {
/**
* Magic bytes ("TGSB" — TinkerGraph Storage Binary) at the start of every storage file, so a file can be
@@ -101,8 +79,7 @@ public final class GraphBinaryStorage implements TinkerStorage {
static final byte FORMAT_VERSION = 1;
/**
- * Store-level version marker file, holding {@link #MAGIC} followed by the one-byte {@link #FORMAT_VERSION}. It is
- * the single source of truth for the store's format version.
+ * Store-level version marker file, holding {@link #MAGIC} followed by the one-byte {@link #FORMAT_VERSION}.
*/
static final String VERSION_FILE = "VERSION";
@@ -112,27 +89,19 @@ public final class GraphBinaryStorage implements TinkerStorage {
*/
static final int HEADER_SIZE = MAGIC.length;
- private static final byte OP_PUT_VERTEX = 1;
- private static final byte OP_DEL_VERTEX = 2;
- private static final byte OP_PUT_EDGE = 3;
- private static final byte OP_DEL_EDGE = 4;
-
static final String SNAPSHOT_FILE = "snapshot.gbin";
static final String LOG_FILE = "log.gbin";
- private final GraphBinaryWriter writer = new GraphBinaryWriter(TypeSerializerRegistry.INSTANCE);
- private final GraphBinaryReader reader = new GraphBinaryReader(TypeSerializerRegistry.INSTANCE);
+ /**
+ * Default automatic-compaction threshold: 64 MB of appended log since the last compaction.
+ */
+ static final long DEFAULT_COMPACT_THRESHOLD_BYTES = 64L * 1024 * 1024;
private File directory;
private File snapshotFile;
private File logFile;
private File versionFile;
- /**
- * Default automatic-compaction threshold: 64 MB of appended log since the last compaction.
- */
- static final long DEFAULT_COMPACT_THRESHOLD_BYTES = 64L * 1024 * 1024;
-
private DataOutputStream logOut;
private FileOutputStream logFos;
private SyncMode syncMode = SyncMode.COMMIT;
@@ -140,11 +109,42 @@ public final class GraphBinaryStorage implements TinkerStorage {
private long logBytesSinceCompaction = 0;
private boolean closed = false;
+ // ----------------------------------------------------------------------------------------- codec hooks
+
+ /**
+ * Encode a committing transaction's changeset into a single record payload (the framing is added by the caller).
+ */
+ protected abstract byte[] encodeCommit(long txVersion,
+ Collection<TinkerStorageMutation<TinkerVertex>> changedVertices,
+ Collection<TinkerStorageMutation<TinkerEdge>> changedEdges) throws IOException;
+
+ /**
+ * Decode one record payload, folding its puts and deletes into the supplied maps (last-write-wins per id).
+ */
+ protected abstract void decodeFrame(byte[] record,
+ Map<Object, DetachedVertex> vertices,
+ Map<Object, DetachedEdge> edges) throws IOException;
+
+ /**
+ * Write the entire current committed state of the graph to {@code out} as framed records (via {@link #writeFrame}),
+ * for compaction. The fixed {@link #MAGIC} header has already been written to {@code out}.
+ */
+ protected abstract void writeSnapshot(AbstractTinkerGraph graph, DataOutputStream out) throws IOException;
+
+ /**
+ * Reset any per-replay decode state (e.g. a dictionary) before a fold begins. Default is a no-op.
+ */
+ protected void beginReplay() {
+ // no-op by default
+ }
+
+ // ----------------------------------------------------------------------------------------- lifecycle
+
@Override
public void open(final AbstractTinkerGraph graph, final Configuration config) {
final String location = config.getString(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null);
if (null == location)
- throw new IllegalStateException(String.format("%s must be set to use the GraphBinary storage engine",
+ throw new IllegalStateException(String.format("%s must be set to use a durable storage engine",
TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION));
this.directory = new File(location);
this.snapshotFile = new File(directory, SNAPSHOT_FILE);
@@ -159,68 +159,9 @@ public final class GraphBinaryStorage implements TinkerStorage {
establishStoreVersion();
}
- /**
- * Read and validate the store-level version marker, or create it for a new store. This is the single source of
- * truth for the store's format version: a marker naming an unsupported version, or bad magic, fails the open
- * loudly rather than risking a misread. The supported path across an incompatible format bump is to export the
- * graph via {@code g.io()} before upgrading.
- */
- private void establishStoreVersion() {
- // an existing store (has a snapshot or log) written before the marker existed is treated as version 1
- final boolean storeHasData = snapshotFile.exists() || logFile.exists();
- if (!versionFile.exists()) {
- if (storeHasData && FORMAT_VERSION != 1)
- throw new IllegalStateException(String.format(
- "Storage location %s has data but no version marker; cannot confirm it is format version %d",
- directory, FORMAT_VERSION));
- writeStoreVersion();
- return;
- }
- try (final DataInputStream in = new DataInputStream(new java.io.BufferedInputStream(new FileInputStream(versionFile)))) {
- final byte[] magic = new byte[MAGIC.length];
- readFully(in, magic);
- if (!Arrays.equals(magic, MAGIC))
- throw new IOException(String.format("%s is not a TinkerGraph storage version marker (bad magic)", versionFile));
- final byte version = in.readByte();
- if (version != FORMAT_VERSION)
- throw new IOException(String.format(
- "Unsupported storage format version %d at %s (this build writes %d); export via g.io() before upgrading",
- version, directory, FORMAT_VERSION));
- } catch (IOException ex) {
- throw new UncheckedIOException(String.format("Could not read storage version marker %s", versionFile), ex);
- }
- }
-
- /**
- * Write the store-level version marker ({@link #MAGIC} + {@link #FORMAT_VERSION}) durably.
- */
- private void writeStoreVersion() {
- try (final FileOutputStream fos = new FileOutputStream(versionFile);
- final DataOutputStream out = new DataOutputStream(fos)) {
- out.write(MAGIC);
- out.writeByte(FORMAT_VERSION);
- out.flush();
- fos.getFD().sync();
- } catch (IOException ex) {
- throw new UncheckedIOException(String.format("Could not write storage version marker %s", versionFile), ex);
- }
- }
-
- /**
- * Ensure the backing directory exists, creating it if necessary. Called on open and again before writing a
- * snapshot, since {@code close()} may be invoked more than once and the directory may have been removed in between.
- */
- private void ensureDirectory() {
- if (directory.exists()) {
- if (!directory.isDirectory())
- throw new IllegalStateException(String.format("Storage location %s exists but is not a directory", directory));
- } else if (!directory.mkdirs()) {
- throw new IllegalStateException(String.format("Could not create storage directory %s", directory));
- }
- }
-
@Override
public void replay(final AbstractTinkerGraph graph) {
+ beginReplay();
// Fold snapshot then log into final state: last write per id wins, deletes remove.
final Map<Object, DetachedVertex> vertices = new LinkedHashMap<>();
final Map<Object, DetachedEdge> edges = new LinkedHashMap<>();
@@ -242,84 +183,13 @@ public final class GraphBinaryStorage implements TinkerStorage {
graph.tx().commit();
}
- /**
- * Read every record in a file, folding puts and deletes into the supplied maps.
- */
- private void foldRecords(final File file, final Map<Object, DetachedVertex> vertices, final Map<Object, DetachedEdge> edges) {
- final long fileLength = file.length();
- try (final DataInputStream in = new DataInputStream(new java.io.BufferedInputStream(new FileInputStream(file)))) {
- long remaining = readAndVerifyHeader(in, file, fileLength);
- while (true) {
- final byte[] record = readFrame(in, remaining);
- if (record == null)
- break;
- // account for the header (length + crc) and payload just consumed
- remaining -= 2L * Integer.BYTES + record.length;
- applyRecord(record, vertices, edges);
- }
- } catch (IOException ex) {
- throw new UncheckedIOException(String.format("Could not read storage file %s", file), ex);
- }
- }
-
- /**
- * Read and validate the per-file header (magic only), returning the number of record bytes that follow it. An
- * empty file (freshly created, no header yet) is treated as having no records. The store's format version is
- * validated once against the {@link #VERSION_FILE} marker in {@link #open}, not here.
- */
- private long readAndVerifyHeader(final DataInputStream in, final File file, final long fileLength) throws IOException {
- if (fileLength == 0)
- return 0;
- if (fileLength < HEADER_SIZE)
- throw new IOException(String.format("Corrupt storage file %s: shorter than its %d-byte header", file, HEADER_SIZE));
- final byte[] magic = new byte[MAGIC.length];
- readFully(in, magic);
- if (!Arrays.equals(magic, MAGIC))
- throw new IOException(String.format("%s is not a TinkerGraph storage file (bad magic)", file));
- return fileLength - HEADER_SIZE;
- }
-
- private void applyRecord(final byte[] record, final Map<Object, DetachedVertex> vertices, final Map<Object, DetachedEdge> edges) throws IOException {
- // the format version is validated once per file in readAndVerifyHeader, so records no longer repeat it
- final ByteBufferBuffer buffer = new ByteBufferBuffer(record);
- buffer.readLong(); // txVersion, retained for diagnostics/future use
- final int entryCount = buffer.readInt();
- for (int i = 0; i < entryCount; i++) {
- final byte op = buffer.readByte();
- switch (op) {
- case OP_PUT_VERTEX: {
- final Vertex v = reader.read(buffer);
- vertices.put(v.id(), (DetachedVertex) v);
- break;
- }
- case OP_DEL_VERTEX: {
- final Object id = reader.read(buffer);
- vertices.remove(id);
- break;
- }
- case OP_PUT_EDGE: {
- final Edge e = reader.read(buffer);
- edges.put(e.id(), (DetachedEdge) e);
- break;
- }
- case OP_DEL_EDGE: {
- final Object id = reader.read(buffer);
- edges.remove(id);
- break;
- }
- default:
- throw new IOException("Unknown storage op code: " + op);
- }
- }
- }
-
@Override
public void persist(final long txVersion,
final Collection<TinkerStorageMutation<TinkerVertex>> changedVertices,
final Collection<TinkerStorageMutation<TinkerEdge>> changedEdges) {
ensureLogOpen();
try {
- final byte[] frame = encodeRecord(txVersion, changedVertices, changedEdges);
+ final byte[] frame = encodeCommit(txVersion, changedVertices, changedEdges);
writeFrame(logOut, frame);
logBytesSinceCompaction += 2L * Integer.BYTES + frame.length; // length + crc prefixes + payload
} catch (IOException ex) {
@@ -327,39 +197,6 @@ public final class GraphBinaryStorage implements TinkerStorage {
}
}
- /**
- * Serialize a commit record: txVersion, entry count, then each entry as an op byte followed by either the
- * serialized element (put) or the serialized id (delete). The format version lives in the file header, not the
- * record.
- */
- private byte[] encodeRecord(final long txVersion,
- final Collection<TinkerStorageMutation<TinkerVertex>> changedVertices,
- final Collection<TinkerStorageMutation<TinkerEdge>> changedEdges) throws IOException {
- final ByteBufferBuffer buffer = new ByteBufferBuffer();
- buffer.writeLong(txVersion);
- buffer.writeInt(changedVertices.size() + changedEdges.size());
- for (final TinkerStorageMutation<TinkerVertex> m : changedVertices) {
- if (m.isDeleted()) {
- buffer.writeByte(OP_DEL_VERTEX);
- writer.write(m.id(), buffer);
- } else {
- buffer.writeByte(OP_PUT_VERTEX);
- // detach to a stable form independent of the transactional element
- writer.write(DetachedFactory.detach(m.element(), true), buffer);
- }
- }
- for (final TinkerStorageMutation<TinkerEdge> m : changedEdges) {
- if (m.isDeleted()) {
- buffer.writeByte(OP_DEL_EDGE);
- writer.write(m.id(), buffer);
- } else {
- buffer.writeByte(OP_PUT_EDGE);
- writer.write(DetachedFactory.detach(m.element(), true), buffer);
- }
- }
- return buffer.toWrittenArray();
- }
-
@Override
public void flush() {
if (closed)
@@ -428,66 +265,96 @@ public final class GraphBinaryStorage implements TinkerStorage {
compact(graph);
}
+ @Override
+ public void close() {
+ closeLog();
+ closed = true;
+ }
+
+ // ----------------------------------------------------------------------------------------- version marker
+
/**
- * Atomically move {@code source} onto {@code target}, replacing any existing target. Falls back to a non-atomic
- * replacing move on filesystems that do not support atomic moves.
+ * Read and validate the store-level version marker, or create it for a new store. This is the single source of
+ * truth for the store's format version: a marker naming an unsupported version, or bad magic, fails the open
+ * loudly rather than risking a misread.
*/
- private static void atomicMove(final File source, final File target) throws IOException {
- try {
- Files.move(source.toPath(), target.toPath(),
- StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
- } catch (AtomicMoveNotSupportedException anse) {
- Files.move(source.toPath(), target.toPath(), StandardCopyOption.REPLACE_EXISTING);
+ private void establishStoreVersion() {
+ final boolean storeHasData = snapshotFile.exists() || logFile.exists();
+ if (!versionFile.exists()) {
+ if (storeHasData && FORMAT_VERSION != 1)
+ throw new IllegalStateException(String.format(
+ "Storage location %s has data but no version marker; cannot confirm it is format version %d",
+ directory, FORMAT_VERSION));
+ writeStoreVersion();
+ return;
+ }
+ try (final DataInputStream in = new DataInputStream(new BufferedInputStream(new FileInputStream(versionFile)))) {
+ final byte[] magic = new byte[MAGIC.length];
+ readFully(in, magic);
+ if (!Arrays.equals(magic, MAGIC))
+ throw new IOException(String.format("%s is not a TinkerGraph storage version marker (bad magic)", versionFile));
+ final byte version = in.readByte();
+ if (version != FORMAT_VERSION)
+ throw new IOException(String.format(
+ "Unsupported storage format version %d at %s (this build writes %d); export via g.io() before upgrading",
+ version, directory, FORMAT_VERSION));
+ } catch (IOException ex) {
+ throw new UncheckedIOException(String.format("Could not read storage version marker %s", versionFile), ex);
}
}
- /**
- * fsync the storage directory so that recent namespace changes (a rename into place, a file deletion) are durable.
- * A directory fsync is required because those operations only update the directory entry, which the earlier file
- * fsync does not cover.
- */
- private void syncDirectory() {
- try (final FileChannel dirChannel = FileChannel.open(directory.toPath(), StandardOpenOption.READ)) {
- dirChannel.force(true);
+ private void writeStoreVersion() {
+ try (final FileOutputStream fos = new FileOutputStream(versionFile);
+ final DataOutputStream out = new DataOutputStream(fos)) {
+ out.write(MAGIC);
+ out.writeByte(FORMAT_VERSION);
+ out.flush();
+ fos.getFD().sync();
} catch (IOException ex) {
- // some platforms (notably Windows) cannot open a directory as a channel; the atomic rename is the
- // durability guarantee there, so treat inability to sync the directory as non-fatal
+ throw new UncheckedIOException(String.format("Could not write storage version marker %s", versionFile), ex);
}
}
- /**
- * Write the entire current committed state of the graph to {@code out} as a stream of single-element put records,
- * one frame per vertex and per edge. Each frame is an ordinary put record (see {@link #encodeRecord}) with an
- * entry count of one, so {@link #foldRecords} reconstructs the graph from these frames exactly as it would from a
- * commit log. Writing one element at a time keeps peak memory bounded to a single element rather than materializing
- * the whole graph as one byte array, so a snapshot never needs to hold a second full copy of the graph in heap.
- */
- private void writeSnapshot(final AbstractTinkerGraph graph, final DataOutputStream out) throws IOException {
- final Iterator<Vertex> vertexIterator = graph.vertices();
- while (vertexIterator.hasNext())
- writeElementFrame(out, OP_PUT_VERTEX, vertexIterator.next());
- final Iterator<Edge> edgeIterator = graph.edges();
- while (edgeIterator.hasNext())
- writeElementFrame(out, OP_PUT_EDGE, edgeIterator.next());
+ private void ensureDirectory() {
+ if (directory.exists()) {
+ if (!directory.isDirectory())
+ throw new IllegalStateException(String.format("Storage location %s exists but is not a directory", directory));
+ } else if (!directory.mkdirs()) {
+ throw new IllegalStateException(String.format("Could not create storage directory %s", directory));
+ }
}
- /**
- * Encode a single element as a one-entry put record and write it as a framed record to {@code out}. Only one
- * element's bytes are held in memory at a time.
- */
- private void writeElementFrame(final DataOutputStream out, final byte op, final Object element) throws IOException {
- final ByteBufferBuffer buffer = new ByteBufferBuffer();
- buffer.writeLong(0L); // snapshot records have no single tx version
- buffer.writeInt(1);
- buffer.writeByte(op);
- writer.write(DetachedFactory.detach(element, true), buffer);
- writeFrame(out, buffer.toWrittenArray());
+ // ----------------------------------------------------------------------------------------- fold / framing
+
+ private void foldRecords(final File file, final Map<Object, DetachedVertex> vertices, final Map<Object, DetachedEdge> edges) {
+ final long fileLength = file.length();
+ try (final DataInputStream in = new DataInputStream(new BufferedInputStream(new FileInputStream(file)))) {
+ long remaining = readAndVerifyHeader(in, file, fileLength);
+ while (true) {
+ final byte[] record = readFrame(in, remaining);
+ if (record == null)
+ break;
+ remaining -= 2L * Integer.BYTES + record.length;
+ decodeFrame(record, vertices, edges);
+ }
+ } catch (IOException ex) {
+ throw new UncheckedIOException(String.format("Could not read storage file %s", file), ex);
+ }
}
- @Override
- public void close() {
- closeLog();
- closed = true;
+ /**
+ * Read and validate the per-file header (magic only), returning the number of record bytes that follow it.
+ */
+ private long readAndVerifyHeader(final DataInputStream in, final File file, final long fileLength) throws IOException {
+ if (fileLength == 0)
+ return 0;
+ if (fileLength < HEADER_SIZE)
+ throw new IOException(String.format("Corrupt storage file %s: shorter than its %d-byte header", file, HEADER_SIZE));
+ final byte[] magic = new byte[MAGIC.length];
+ readFully(in, magic);
+ if (!Arrays.equals(magic, MAGIC))
+ throw new IOException(String.format("%s is not a TinkerGraph storage file (bad magic)", file));
+ return fileLength - HEADER_SIZE;
}
private void ensureLogOpen() {
@@ -505,14 +372,6 @@ public final class GraphBinaryStorage implements TinkerStorage {
}
}
- /**
- * Write the per-file header ({@link #MAGIC}) at the start of a storage file. The format version is recorded once
- * per store in the {@link #VERSION_FILE} marker, not per file.
- */
- private static void writeHeader(final DataOutputStream out) throws IOException {
- out.write(MAGIC);
- }
-
private void closeLog() {
if (logOut != null) {
try {
@@ -527,12 +386,19 @@ public final class GraphBinaryStorage implements TinkerStorage {
}
}
+ /**
+ * Write the per-file header ({@link #MAGIC}) at the start of a storage file.
+ */
+ private static void writeHeader(final DataOutputStream out) throws IOException {
+ out.write(MAGIC);
+ }
+
/**
* Write a framed record: a 4-byte big-endian payload length, a 4-byte CRC32 of the payload, then the payload.
* The checksum lets a reader tell a bit-flip inside a complete frame (corruption) from a short final frame left
- * by an interrupted append (truncation).
+ * by an interrupted append (truncation). Available to codec subclasses writing per-element snapshot frames.
*/
- private static void writeFrame(final DataOutputStream out, final byte[] payload) throws IOException {
+ protected static void writeFrame(final DataOutputStream out, final byte[] payload) throws IOException {
final CRC32 crc = new CRC32();
crc.update(payload);
out.writeInt(payload.length);
@@ -541,14 +407,9 @@ public final class GraphBinaryStorage implements TinkerStorage {
}
/**
- * Read a framed record, or return {@code null} at end of the readable log. A frame that is only partially present
- * — the file ends inside the header or payload — is treated as an interrupted trailing append (truncation) and
- * ends reading so earlier committed records still load. A frame that is fully present but whose stored CRC does
- * not match its payload is genuine corruption and is raised, rather than silently dropping it and everything
- * after it.
- *
- * @param remaining bytes left in the file at the current position; used to distinguish a short trailing frame
- * (truncation) from a complete frame, and to bound the payload allocation against a garbage length
+ * Read a framed record, or return {@code null} at end of the readable log. A frame only partially present is
+ * treated as an interrupted trailing append (truncation) and ends reading; a fully-present frame whose stored CRC
+ * does not match is genuine corruption and is raised.
*/
private static byte[] readFrame(final DataInputStream in, final long remaining) throws IOException {
if (remaining == 0)
@@ -588,4 +449,29 @@ public final class GraphBinaryStorage implements TinkerStorage {
off += read;
}
}
+
+ /**
+ * Atomically move {@code source} onto {@code target}, replacing any existing target. Falls back to a non-atomic
+ * replacing move on filesystems that do not support atomic moves.
+ */
+ private static void atomicMove(final File source, final File target) throws IOException {
+ try {
+ Files.move(source.toPath(), target.toPath(),
+ StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
+ } catch (AtomicMoveNotSupportedException anse) {
+ Files.move(source.toPath(), target.toPath(), StandardCopyOption.REPLACE_EXISTING);
+ }
+ }
+
+ /**
+ * fsync the storage directory so that recent namespace changes (a rename into place, a file deletion) are durable.
+ */
+ private void syncDirectory() {
+ try (final FileChannel dirChannel = FileChannel.open(directory.toPath(), StandardOpenOption.READ)) {
+ dirChannel.force(true);
+ } catch (IOException ex) {
+ // some platforms (notably Windows) cannot open a directory as a channel; the atomic rename is the
+ // durability guarantee there, so treat inability to sync the directory as non-fatal
+ }
+ }
}
diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java
index 2aaa7643cc..a5f3fd1c5c 100644
--- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java
+++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java
@@ -18,65 +18,29 @@
*/
package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage;
-import org.apache.commons.configuration2.Configuration;
import org.apache.tinkerpop.gremlin.structure.Edge;
import org.apache.tinkerpop.gremlin.structure.Vertex;
import org.apache.tinkerpop.gremlin.structure.io.binary.GraphBinaryReader;
import org.apache.tinkerpop.gremlin.structure.io.binary.GraphBinaryWriter;
import org.apache.tinkerpop.gremlin.structure.io.binary.TypeSerializerRegistry;
-import org.apache.tinkerpop.gremlin.structure.util.Attachable;
import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge;
import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedFactory;
import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.AbstractTinkerGraph;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerEdge;
-import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex;
-import java.io.BufferedOutputStream;
-import java.io.DataInputStream;
import java.io.DataOutputStream;
-import java.io.EOFException;
-import java.io.File;
-import java.io.FileInputStream;
-import java.io.FileOutputStream;
import java.io.IOException;
-import java.io.InputStream;
-import java.io.OutputStream;
-import java.io.UncheckedIOException;
-import java.nio.channels.FileChannel;
-import java.nio.file.AtomicMoveNotSupportedException;
-import java.nio.file.Files;
-import java.nio.file.StandardCopyOption;
-import java.nio.file.StandardOpenOption;
-import java.util.Arrays;
import java.util.Collection;
import java.util.Iterator;
-import java.util.LinkedHashMap;
import java.util.Map;
-import java.util.zip.CRC32;
/**
- * A durable {@link TinkerStorage} engine that persists a {@code TinkerStorageGraph} as an append-only commit log
- * ("write-ahead log") serialized with GraphBinary. On each committed transaction the changeset is appended to
- * {@code log.gbin} as a single record; on open the optional {@code snapshot.gbin} is read followed by the log, with the
- * folded result re-applied to the in-memory graph. {@link #compact(AbstractTinkerGraph)} rewrites the snapshot from the
- * current committed state and truncates the log.
- * <p/>
- * On commit the appended record is made durable according to the configured {@link SyncMode}: {@link SyncMode#COMMIT}
- * (default) {@code fsync}s so the commit survives an OS crash or power loss, while {@link SyncMode#OS} only flushes to
- * the operating system.
- * <p/>
- * On-disk compatibility: the store's format version is recorded once in a {@code VERSION} marker file (magic +
- * version), which is the single source of truth even when a store momentarily holds a snapshot and a log written at
- * different times. Individual files carry only the magic for identity/corruption detection. Opening a store whose
- * marker names an unsupported version fails loudly — records are never misread across a format change. The engine
- * keeps this simple by not attempting in-place migration: the supported path across an incompatible format bump is to
- * export via {@code g.io()} before upgrading. Additive changes should prefer new record opcodes; unknown opcodes are
- * a hard error (a durable store must not silently drop records it cannot parse), so a genuinely incompatible change
- * bumps the version.
- * <p/>
- * The in-memory graph remains authoritative (write-through). This engine does not support graphs larger than memory.
+ * The GraphBinary {@link TinkerStorage} codec on top of {@link AbstractLogStorage}. The base owns the durable
+ * log-structured machinery (file layout, {@code VERSION} marker, CRC framing, replay fold, {@link SyncMode}
+ * durability, crash-safe and threshold compaction); this class supplies only how an element is encoded and decoded,
+ * using the GraphBinary serializers ({@link GraphBinaryWriter}/{@link GraphBinaryReader}).
* <p/>
* Known limitation (write amplification): a commit records each changed element in full — a single property change on
* a large element rewrites the whole element to the log. Elements are typically small and automatic compaction bounds
@@ -84,203 +48,53 @@ import java.util.zip.CRC32;
* the {@link TinkerStorageMutation} contract and the replay fold. The snapshot, by contrast, is streamed one element
* at a time so compaction never holds a second full copy of the graph in heap.
*/
-public final class GraphBinaryStorage implements TinkerStorage {
-
- /**
- * Magic bytes ("TGSB" — TinkerGraph Storage Binary) at the start of every storage file, so a file can be
- * identified as one written by this engine (and a foreign or corrupt file rejected) before any record is read.
- */
- static final byte[] MAGIC = { 'T', 'G', 'S', 'B' };
-
- /**
- * On-disk format version of the store. Recorded once per store in the {@link #VERSION_FILE} marker rather than in
- * every file, so a store has a single unambiguous version even when it momentarily holds a snapshot and a log
- * written at different times. A future format bump is detected against this marker so an older store is rejected
- * (never silently misread); the supported migration path is to export via {@code g.io()} before upgrading.
- */
- static final byte FORMAT_VERSION = 1;
-
- /**
- * Store-level version marker file, holding {@link #MAGIC} followed by the one-byte {@link #FORMAT_VERSION}. It is
- * the single source of truth for the store's format version.
- */
- static final String VERSION_FILE = "VERSION";
-
- /**
- * Bytes of the per-file header: just {@link #MAGIC}. The format version lives in the store-level
- * {@link #VERSION_FILE}, not in each file.
- */
- static final int HEADER_SIZE = MAGIC.length;
+public final class GraphBinaryStorage extends AbstractLogStorage {
private static final byte OP_PUT_VERTEX = 1;
private static final byte OP_DEL_VERTEX = 2;
private static final byte OP_PUT_EDGE = 3;
private static final byte OP_DEL_EDGE = 4;
- static final String SNAPSHOT_FILE = "snapshot.gbin";
- static final String LOG_FILE = "log.gbin";
-
private final GraphBinaryWriter writer = new GraphBinaryWriter(TypeSerializerRegistry.INSTANCE);
private final GraphBinaryReader reader = new GraphBinaryReader(TypeSerializerRegistry.INSTANCE);
- private File directory;
- private File snapshotFile;
- private File logFile;
- private File versionFile;
-
/**
- * Default automatic-compaction threshold: 64 MB of appended log since the last compaction.
+ * Serialize a commit record: txVersion, entry count, then each entry as an op byte followed by either the
+ * serialized element (put) or the serialized id (delete).
*/
- static final long DEFAULT_COMPACT_THRESHOLD_BYTES = 64L * 1024 * 1024;
-
- private DataOutputStream logOut;
- private FileOutputStream logFos;
- private SyncMode syncMode = SyncMode.COMMIT;
- private long compactThresholdBytes = DEFAULT_COMPACT_THRESHOLD_BYTES;
- private long logBytesSinceCompaction = 0;
- private boolean closed = false;
-
@Override
- public void open(final AbstractTinkerGraph graph, final Configuration config) {
- final String location = config.getString(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION, null);
- if (null == location)
- throw new IllegalStateException(String.format("%s must be set to use the GraphBinary storage engine",
- TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION));
- this.directory = new File(location);
- this.snapshotFile = new File(directory, SNAPSHOT_FILE);
- this.logFile = new File(directory, LOG_FILE);
- this.versionFile = new File(directory, VERSION_FILE);
- this.syncMode = SyncMode.fromConfigValue(config.getString(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_SYNC, null));
- this.compactThresholdBytes = config.getLong(
- TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, DEFAULT_COMPACT_THRESHOLD_BYTES);
- // seed the counter with any pre-existing log so a graph reopened with a large log still compacts promptly
- this.logBytesSinceCompaction = logFile.exists() ? logFile.length() : 0;
- ensureDirectory();
- establishStoreVersion();
- }
-
- /**
- * Read and validate the store-level version marker, or create it for a new store. This is the single source of
- * truth for the store's format version: a marker naming an unsupported version, or bad magic, fails the open
- * loudly rather than risking a misread. The supported path across an incompatible format bump is to export the
- * graph via {@code g.io()} before upgrading.
- */
- private void establishStoreVersion() {
- // an existing store (has a snapshot or log) written before the marker existed is treated as version 1
- final boolean storeHasData = snapshotFile.exists() || logFile.exists();
- if (!versionFile.exists()) {
- if (storeHasData && FORMAT_VERSION != 1)
- throw new IllegalStateException(String.format(
- "Storage location %s has data but no version marker; cannot confirm it is format version %d",
- directory, FORMAT_VERSION));
- writeStoreVersion();
- return;
- }
- try (final DataInputStream in = new DataInputStream(new java.io.BufferedInputStream(new FileInputStream(versionFile)))) {
- final byte[] magic = new byte[MAGIC.length];
- readFully(in, magic);
- if (!Arrays.equals(magic, MAGIC))
- throw new IOException(String.format("%s is not a TinkerGraph storage version marker (bad magic)", versionFile));
- final byte version = in.readByte();
- if (version != FORMAT_VERSION)
- throw new IOException(String.format(
- "Unsupported storage format version %d at %s (this build writes %d); export via g.io() before upgrading",
- version, directory, FORMAT_VERSION));
- } catch (IOException ex) {
- throw new UncheckedIOException(String.format("Could not read storage version marker %s", versionFile), ex);
- }
- }
-
- /**
- * Write the store-level version marker ({@link #MAGIC} + {@link #FORMAT_VERSION}) durably.
- */
- private void writeStoreVersion() {
- try (final FileOutputStream fos = new FileOutputStream(versionFile);
- final DataOutputStream out = new DataOutputStream(fos)) {
- out.write(MAGIC);
- out.writeByte(FORMAT_VERSION);
- out.flush();
- fos.getFD().sync();
- } catch (IOException ex) {
- throw new UncheckedIOException(String.format("Could not write storage version marker %s", versionFile), ex);
- }
- }
-
- /**
- * Ensure the backing directory exists, creating it if necessary. Called on open and again before writing a
- * snapshot, since {@code close()} may be invoked more than once and the directory may have been removed in between.
- */
- private void ensureDirectory() {
- if (directory.exists()) {
- if (!directory.isDirectory())
- throw new IllegalStateException(String.format("Storage location %s exists but is not a directory", directory));
- } else if (!directory.mkdirs()) {
- throw new IllegalStateException(String.format("Could not create storage directory %s", directory));
+ protected byte[] encodeCommit(final long txVersion,
+ final Collection<TinkerStorageMutation<TinkerVertex>> changedVertices,
+ final Collection<TinkerStorageMutation<TinkerEdge>> changedEdges) throws IOException {
+ final ByteBufferBuffer buffer = new ByteBufferBuffer();
+ buffer.writeLong(txVersion);
+ buffer.writeInt(changedVertices.size() + changedEdges.size());
+ for (final TinkerStorageMutation<TinkerVertex> m : changedVertices) {
+ if (m.isDeleted()) {
+ buffer.writeByte(OP_DEL_VERTEX);
+ writer.write(m.id(), buffer);
+ } else {
+ buffer.writeByte(OP_PUT_VERTEX);
+ // detach to a stable form independent of the transactional element
+ writer.write(DetachedFactory.detach(m.element(), true), buffer);
+ }
}
- }
-
- @Override
- public void replay(final AbstractTinkerGraph graph) {
- // Fold snapshot then log into final state: last write per id wins, deletes remove.
- final Map<Object, DetachedVertex> vertices = new LinkedHashMap<>();
- final Map<Object, DetachedEdge> edges = new LinkedHashMap<>();
-
- if (snapshotFile.exists())
- foldRecords(snapshotFile, vertices, edges);
- if (logFile.exists())
- foldRecords(logFile, vertices, edges);
-
- if (vertices.isEmpty() && edges.isEmpty())
- return;
-
- // Attach vertices first so edges can find their endpoints, then commit once.
- for (final DetachedVertex v : vertices.values())
- v.attach(Attachable.Method.getOrCreate(graph));
- for (final DetachedEdge e : edges.values())
- e.attach(Attachable.Method.getOrCreate(graph));
-
- graph.tx().commit();
- }
-
- /**
- * Read every record in a file, folding puts and deletes into the supplied maps.
- */
- private void foldRecords(final File file, final Map<Object, DetachedVertex> vertices, final Map<Object, DetachedEdge> edges) {
- final long fileLength = file.length();
- try (final DataInputStream in = new DataInputStream(new java.io.BufferedInputStream(new FileInputStream(file)))) {
- long remaining = readAndVerifyHeader(in, file, fileLength);
- while (true) {
- final byte[] record = readFrame(in, remaining);
- if (record == null)
- break;
- // account for the header (length + crc) and payload just consumed
- remaining -= 2L * Integer.BYTES + record.length;
- applyRecord(record, vertices, edges);
+ for (final TinkerStorageMutation<TinkerEdge> m : changedEdges) {
+ if (m.isDeleted()) {
+ buffer.writeByte(OP_DEL_EDGE);
+ writer.write(m.id(), buffer);
+ } else {
+ buffer.writeByte(OP_PUT_EDGE);
+ writer.write(DetachedFactory.detach(m.element(), true), buffer);
}
- } catch (IOException ex) {
- throw new UncheckedIOException(String.format("Could not read storage file %s", file), ex);
}
+ return buffer.toWrittenArray();
}
- /**
- * Read and validate the per-file header (magic only), returning the number of record bytes that follow it. An
- * empty file (freshly created, no header yet) is treated as having no records. The store's format version is
- * validated once against the {@link #VERSION_FILE} marker in {@link #open}, not here.
- */
- private long readAndVerifyHeader(final DataInputStream in, final File file, final long fileLength) throws IOException {
- if (fileLength == 0)
- return 0;
- if (fileLength < HEADER_SIZE)
- throw new IOException(String.format("Corrupt storage file %s: shorter than its %d-byte header", file, HEADER_SIZE));
- final byte[] magic = new byte[MAGIC.length];
- readFully(in, magic);
- if (!Arrays.equals(magic, MAGIC))
- throw new IOException(String.format("%s is not a TinkerGraph storage file (bad magic)", file));
- return fileLength - HEADER_SIZE;
- }
-
- private void applyRecord(final byte[] record, final Map<Object, DetachedVertex> vertices, final Map<Object, DetachedEdge> edges) throws IOException {
- // the format version is validated once per file in readAndVerifyHeader, so records no longer repeat it
+ @Override
+ protected void decodeFrame(final byte[] record,
+ final Map<Object, DetachedVertex> vertices,
+ final Map<Object, DetachedEdge> edges) throws IOException {
final ByteBufferBuffer buffer = new ByteBufferBuffer(record);
buffer.readLong(); // txVersion, retained for diagnostics/future use
final int entryCount = buffer.readInt();
@@ -313,156 +127,12 @@ public final class GraphBinaryStorage implements TinkerStorage {
}
}
- @Override
- public void persist(final long txVersion,
- final Collection<TinkerStorageMutation<TinkerVertex>> changedVertices,
- final Collection<TinkerStorageMutation<TinkerEdge>> changedEdges) {
- ensureLogOpen();
- try {
- final byte[] frame = encodeRecord(txVersion, changedVertices, changedEdges);
- writeFrame(logOut, frame);
- logBytesSinceCompaction += 2L * Integer.BYTES + frame.length; // length + crc prefixes + payload
- } catch (IOException ex) {
- throw new UncheckedIOException("Could not append transaction to storage log", ex);
- }
- }
-
/**
- * Serialize a commit record: txVersion, entry count, then each entry as an op byte followed by either the
- * serialized element (put) or the serialized id (delete). The format version lives in the file header, not the
- * record.
+ * Stream the current committed state as one framed put record per vertex and per edge, so peak memory is bounded
+ * to a single element rather than materializing the whole graph as one byte array.
*/
- private byte[] encodeRecord(final long txVersion,
- final Collection<TinkerStorageMutation<TinkerVertex>> changedVertices,
- final Collection<TinkerStorageMutation<TinkerEdge>> changedEdges) throws IOException {
- final ByteBufferBuffer buffer = new ByteBufferBuffer();
- buffer.writeLong(txVersion);
- buffer.writeInt(changedVertices.size() + changedEdges.size());
- for (final TinkerStorageMutation<TinkerVertex> m : changedVertices) {
- if (m.isDeleted()) {
- buffer.writeByte(OP_DEL_VERTEX);
- writer.write(m.id(), buffer);
- } else {
- buffer.writeByte(OP_PUT_VERTEX);
- // detach to a stable form independent of the transactional element
- writer.write(DetachedFactory.detach(m.element(), true), buffer);
- }
- }
- for (final TinkerStorageMutation<TinkerEdge> m : changedEdges) {
- if (m.isDeleted()) {
- buffer.writeByte(OP_DEL_EDGE);
- writer.write(m.id(), buffer);
- } else {
- buffer.writeByte(OP_PUT_EDGE);
- writer.write(DetachedFactory.detach(m.element(), true), buffer);
- }
- }
- return buffer.toWrittenArray();
- }
-
- @Override
- public void flush() {
- if (closed)
- return;
- if (logOut != null) {
- try {
- // flush the JVM buffer into the OS page cache; durable against a JVM process crash
- logOut.flush();
- // in COMMIT mode also force the OS page cache to the device, so an acknowledged commit is durable
- // against an OS crash or power loss. OS mode stops at the flush above and accepts that weaker guarantee.
- if (syncMode == SyncMode.COMMIT)
- logFos.getFD().sync();
- } catch (IOException ex) {
- throw new UncheckedIOException("Could not flush storage log", ex);
- }
- }
- }
-
- @Override
- public void compact(final AbstractTinkerGraph graph) {
- if (closed)
- return;
- // Write a fresh snapshot of the current committed state, then truncate the log. This must be crash-safe: at
- // no point may a crash leave the store without a readable snapshot-or-log covering the committed state.
- // Ordering is write-tmp -> fsync tmp -> atomically rename tmp over the snapshot -> fsync dir (the rename is
- // now durable) -> delete the log -> fsync dir. The old snapshot is only ever replaced by an atomic rename, so
- // a crash at any step leaves either the old (snapshot + log) or the new (snapshot) intact — never neither.
- closeLog();
- ensureDirectory();
- final File tmp = new File(directory, SNAPSHOT_FILE + ".tmp");
- try (final FileOutputStream fos = new FileOutputStream(tmp);
- final DataOutputStream out = new DataOutputStream(new BufferedOutputStream(fos))) {
- writeHeader(out);
- writeSnapshot(graph, out);
- out.flush();
- // force the snapshot's bytes to the device before it is renamed into place
- fos.getFD().sync();
- } catch (IOException ex) {
- throw new UncheckedIOException("Could not write storage snapshot", ex);
- }
-
- try {
- // atomically replace the snapshot; no delete-then-rename window where the snapshot is briefly absent
- atomicMove(tmp, snapshotFile);
- // fsync the directory so the rename survives a crash before we touch the log
- syncDirectory();
-
- // truncate the log now that the snapshot durably reflects the committed state
- if (logFile.exists() && !logFile.delete())
- throw new IOException("Could not truncate storage log " + logFile);
- // fsync the directory again so the log's removal is durable
- syncDirectory();
- } catch (IOException ex) {
- throw new UncheckedIOException("Could not finalize storage snapshot", ex);
- }
-
- // the log is now empty; the accumulated state lives in the snapshot
- logBytesSinceCompaction = 0;
- }
-
@Override
- public void maybeCompact(final AbstractTinkerGraph graph) {
- if (closed || compactThresholdBytes <= 0)
- return;
- if (logBytesSinceCompaction >= compactThresholdBytes)
- compact(graph);
- }
-
- /**
- * Atomically move {@code source} onto {@code target}, replacing any existing target. Falls back to a non-atomic
- * replacing move on filesystems that do not support atomic moves.
- */
- private static void atomicMove(final File source, final File target) throws IOException {
- try {
- Files.move(source.toPath(), target.toPath(),
- StandardCopyOption.ATOMIC_MOVE, StandardCopyOption.REPLACE_EXISTING);
- } catch (AtomicMoveNotSupportedException anse) {
- Files.move(source.toPath(), target.toPath(), StandardCopyOption.REPLACE_EXISTING);
- }
- }
-
- /**
- * fsync the storage directory so that recent namespace changes (a rename into place, a file deletion) are durable.
- * A directory fsync is required because those operations only update the directory entry, which the earlier file
- * fsync does not cover.
- */
- private void syncDirectory() {
- try (final FileChannel dirChannel = FileChannel.open(directory.toPath(), StandardOpenOption.READ)) {
- dirChannel.force(true);
- } catch (IOException ex) {
- // some platforms (notably Windows) cannot open a directory as a channel; the atomic rename is the
- // durability guarantee there, so treat inability to sync the directory as non-fatal
- }
- }
-
- /**
- * Write the entire current committed state of the graph to {@code out} as a stream of single-element put records,
- * one frame per vertex and per edge. Each frame is an ordinary put record (see {@link #encodeRecord}) with an
- * entry count of one, so {@link #foldRecords} reconstructs the graph from these frames exactly as it would from a
- * commit log. Writing one element at a time keeps peak memory bounded to a single element rather than materializing
- * the whole graph as one byte array, so a snapshot never needs to hold a second full copy of the graph in heap.
- */
- private void writeSnapshot(final AbstractTinkerGraph graph, final DataOutputStream out) throws IOException {
+ protected void writeSnapshot(final AbstractTinkerGraph graph, final DataOutputStream out) throws IOException {
final Iterator<Vertex> vertexIterator = graph.vertices();
while (vertexIterator.hasNext())
writeElementFrame(out, OP_PUT_VERTEX, vertexIterator.next());
@@ -472,8 +142,7 @@ public final class GraphBinaryStorage implements TinkerStorage {
}
/**
- * Encode a single element as a one-entry put record and write it as a framed record to {@code out}. Only one
- * element's bytes are held in memory at a time.
+ * Encode a single element as a one-entry put record and write it as a framed record to {@code out}.
*/
private void writeElementFrame(final DataOutputStream out, final byte op, final Object element) throws IOException {
final ByteBufferBuffer buffer = new ByteBufferBuffer();
@@ -483,109 +152,4 @@ public final class GraphBinaryStorage implements TinkerStorage {
writer.write(DetachedFactory.detach(element, true), buffer);
writeFrame(out, buffer.toWrittenArray());
}
-
- @Override
- public void close() {
- closeLog();
- closed = true;
- }
-
- private void ensureLogOpen() {
- if (logOut == null) {
- try {
- final boolean freshFile = !logFile.exists() || logFile.length() == 0;
- // retain the FileOutputStream so flush() can reach its FileDescriptor for fsync
- logFos = new FileOutputStream(logFile, true);
- logOut = new DataOutputStream(new BufferedOutputStream(logFos));
- if (freshFile)
- writeHeader(logOut);
- } catch (IOException ex) {
- throw new UncheckedIOException("Could not open storage log for append", ex);
- }
- }
- }
-
- /**
- * Write the per-file header ({@link #MAGIC}) at the start of a storage file. The format version is recorded once
- * per store in the {@link #VERSION_FILE} marker, not per file.
- */
- private static void writeHeader(final DataOutputStream out) throws IOException {
- out.write(MAGIC);
- }
-
- private void closeLog() {
- if (logOut != null) {
- try {
- logOut.flush();
- logOut.close();
- } catch (IOException ex) {
- throw new UncheckedIOException("Could not close storage log", ex);
- } finally {
- logOut = null;
- logFos = null;
- }
- }
- }
-
- /**
- * Write a framed record: a 4-byte big-endian payload length, a 4-byte CRC32 of the payload, then the payload.
- * The checksum lets a reader tell a bit-flip inside a complete frame (corruption) from a short final frame left
- * by an interrupted append (truncation).
- */
- private static void writeFrame(final DataOutputStream out, final byte[] payload) throws IOException {
- final CRC32 crc = new CRC32();
- crc.update(payload);
- out.writeInt(payload.length);
- out.writeInt((int) crc.getValue());
- out.write(payload);
- }
-
- /**
- * Read a framed record, or return {@code null} at end of the readable log. A frame that is only partially present
- * — the file ends inside the header or payload — is treated as an interrupted trailing append (truncation) and
- * ends reading so earlier committed records still load. A frame that is fully present but whose stored CRC does
- * not match its payload is genuine corruption and is raised, rather than silently dropping it and everything
- * after it.
- *
- * @param remaining bytes left in the file at the current position; used to distinguish a short trailing frame
- * (truncation) from a complete frame, and to bound the payload allocation against a garbage length
- */
- private static byte[] readFrame(final DataInputStream in, final long remaining) throws IOException {
- if (remaining == 0)
- return null; // clean end of file, exactly on a frame boundary
- if (remaining < 2L * Integer.BYTES)
- return null; // not even a full header left — interrupted append
-
- final int length = in.readInt();
- final int storedCrc = in.readInt();
- if (length < 0)
- throw new IOException("Corrupt storage frame: negative payload length " + length);
- if ((long) length > remaining - 2L * Integer.BYTES)
- return null; // frame claims more bytes than remain — truncated trailing append
-
- final byte[] payload = new byte[length];
- try {
- readFully(in, payload);
- } catch (EOFException eof) {
- return null; // partial trailing payload from an interrupted append
- }
-
- final CRC32 crc = new CRC32();
- crc.update(payload);
- if ((int) crc.getValue() != storedCrc)
- throw new IOException(String.format(
- "Corrupt storage frame: CRC mismatch (stored %08x, computed %08x) in a fully-present %d-byte record",
- storedCrc, (int) crc.getValue(), length));
- return payload;
- }
-
- private static void readFully(final InputStream in, final byte[] dst) throws IOException {
- int off = 0;
- while (off < dst.length) {
- final int read = in.read(dst, off, dst.length - off);
- if (read < 0)
- throw new EOFException();
- off += read;
- }
- }
}
(tinkerpop) 05/09: Broaden TinkerStorage TCK: value types, ids, varints, unicode round-trips
Posted by sp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
spmallette pushed a commit to branch tinkergraph-storage
in repository https://gitbox.apache.org/repos/asf/tinkerpop.git
commit f32a598ea3d247db9bc22f052fe1ac77fe3f7142
Author: Stephen Mallette <st...@amazon.com>
AuthorDate: Wed Aug 19 16:24:13 2026 +0000
Broaden TinkerStorage TCK: value types, ids, varints, unicode round-trips
Add conformance round-trip coverage for the storage codec's breadth: a
value-type matrix (int/long/float/double/boolean/byte/short/char/string/UUID/
BigInteger/BigDecimal/OffsetDateTime/Duration), collection-valued properties
(List/Map/Set), null values, heterogeneous same-key types, non-Long (UUID and
String) element ids, unicode keys and values, and a large-schema graph whose
>127 distinct keys and >127 values under one key exercise the multi-byte varint
path that small graphs never reach.
Assisted-by: Claude Code:claude-opus-4-8
---
.../AbstractTinkerStorageConformanceTest.java | 206 +++++++++++++++++++++
1 file changed, 206 insertions(+)
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java
index 552cd37da4..f5ec5033f4 100644
--- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java
+++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java
@@ -32,15 +32,26 @@ import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.time.Duration;
+import java.time.OffsetDateTime;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.UUID;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.concurrent.TimeUnit;
+import static org.junit.Assert.assertArrayEquals;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertFalse;
import static org.junit.Assert.assertNull;
@@ -284,6 +295,201 @@ public abstract class AbstractTinkerStorageConformanceTest {
}
}
+ @Test
+ public void shouldRoundTripDiverseValueTypes() {
+ final Map<String, Object> values = new LinkedHashMap<>();
+ values.put("int", 42);
+ values.put("long", 42L);
+ values.put("float", 1.5f);
+ values.put("double", 2.5d);
+ values.put("bool", true);
+ values.put("byte", (byte) 7);
+ values.put("short", (short) 9);
+ values.put("char", 'x');
+ values.put("string", "hello");
+ values.put("uuid", new UUID(12L, 34L));
+ values.put("bigint", new BigInteger("123456789012345678901234567890"));
+ values.put("bigdec", new BigDecimal("3.14159265358979"));
+ values.put("datetime", OffsetDateTime.parse("2020-01-02T03:04:05Z"));
+ values.put("duration", Duration.ofSeconds(90));
+
+ TinkerStorageGraph graph = open();
+ try {
+ final Vertex v = graph.addVertex(T.id, 1);
+ values.forEach(v::property);
+ graph.tx().commit();
+ } finally {
+ graph.close();
+ }
+ graph = open();
+ try {
+ final Vertex v = graph.vertices(1).next();
+ values.forEach((k, expected) -> assertEquals(k, expected, v.value(k)));
+ } finally {
+ graph.close();
+ }
+ }
+
+ @Test
+ public void shouldRoundTripCollectionValuedProperties() {
+ final List<Object> list = Arrays.asList(1, "two", 3.0d);
+ final Map<String, Object> map = new LinkedHashMap<>();
+ map.put("a", 1);
+ map.put("b", "two");
+ final Set<Object> set = new LinkedHashSet<>(Arrays.asList("x", "y", "z"));
+
+ TinkerStorageGraph graph = open();
+ try {
+ final Vertex v = graph.addVertex(T.id, 1);
+ v.property("list", list);
+ v.property("map", map);
+ v.property("set", set);
+ graph.tx().commit();
+ } finally {
+ graph.close();
+ }
+ graph = open();
+ try {
+ final Vertex v = graph.vertices(1).next();
+ assertEquals(list, v.value("list"));
+ assertEquals(map, v.value("map"));
+ assertEquals(set, v.value("set"));
+ } finally {
+ graph.close();
+ }
+ }
+
+ @Test
+ public void shouldRoundTripNullPropertyValue() {
+ final Configuration conf = config();
+ conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES, true);
+ TinkerStorageGraph graph = TinkerStorageGraph.open(conf);
+ try {
+ final Vertex v = graph.addVertex(T.id, 1);
+ v.property("maybe", null);
+ graph.tx().commit();
+ } finally {
+ graph.close();
+ }
+ graph = TinkerStorageGraph.open(conf);
+ try {
+ final VertexProperty<Object> vp = graph.vertices(1).next().<Object>properties("maybe").next();
+ assertTrue(vp.isPresent());
+ assertNull(vp.value());
+ } finally {
+ graph.close();
+ }
+ }
+
+ @Test
+ public void shouldRoundTripHeterogeneousSameKeyTypes() {
+ TinkerStorageGraph graph = open();
+ try {
+ graph.addVertex(T.id, 1, "k", 42); // Integer
+ graph.addVertex(T.id, 2, "k", 42L); // Long
+ graph.tx().commit();
+ } finally {
+ graph.close();
+ }
+ graph = open();
+ try {
+ assertEquals(Integer.valueOf(42), graph.vertices(1).next().value("k"));
+ assertEquals(Long.valueOf(42L), graph.vertices(2).next().value("k"));
+ } finally {
+ graph.close();
+ }
+ }
+
+ @Test
+ public void shouldRoundTripUuidElementIds() {
+ roundTripElementIds(new UUID(0L, 1L), new UUID(0L, 2L), new UUID(0L, 10L));
+ }
+
+ @Test
+ public void shouldRoundTripStringElementIds() {
+ roundTripElementIds("v-1", "v-2", "e-10");
+ }
+
+ // uses the default ANY id manager so any id type is accepted verbatim; the point is that the storage codec
+ // round-trips non-Long element ids through its scalar id encoding
+ private void roundTripElementIds(final Object outId, final Object inId, final Object edgeId) {
+ TinkerStorageGraph graph = open();
+ try {
+ final Vertex a = graph.addVertex(T.id, outId, "name", "a");
+ final Vertex b = graph.addVertex(T.id, inId, "name", "b");
+ a.addEdge("knows", b, T.id, edgeId, "weight", 0.5d);
+ graph.tx().commit();
+ } finally {
+ graph.close();
+ }
+ graph = open();
+ try {
+ assertEquals("a", graph.vertices(outId).next().value("name"));
+ assertEquals(outId, graph.vertices(outId).next().id());
+ final Edge e = graph.edges(edgeId).next();
+ assertEquals("knows", e.label());
+ assertEquals(outId, e.outVertex().id());
+ assertEquals(inId, e.inVertex().id());
+ } finally {
+ graph.close();
+ }
+ }
+
+ @Test
+ public void shouldRoundTripLargeSchemaAcrossVarintBoundary() {
+ // >127 distinct keys and >127 values under one key push dictionary refs and counts past the single-byte
+ // LEB128 range, exercising the multi-byte varint path that small graphs never reach
+ final int n = 200;
+ // list cardinality so the >127 values under "multi" survive reopen (reconstruction takes cardinality from
+ // graph config, not the stored record)
+ final Configuration conf = config();
+ conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_DEFAULT_VERTEX_PROPERTY_CARDINALITY, "list");
+ TinkerStorageGraph graph = TinkerStorageGraph.open(conf);
+ try {
+ final Vertex v = graph.addVertex(T.id, 1);
+ for (int i = 0; i < n; i++)
+ v.property("key" + i, i);
+ for (int i = 0; i < n; i++)
+ v.property(VertexProperty.Cardinality.list, "multi", i);
+ graph.tx().commit();
+ graph.compact(); // also exercises a dictionary header with >127 entries
+ } finally {
+ graph.close();
+ }
+ graph = TinkerStorageGraph.open(conf);
+ try {
+ final Vertex v = graph.vertices(1).next();
+ for (int i = 0; i < n; i++)
+ assertEquals("key" + i, Integer.valueOf(i), v.value("key" + i));
+ assertEquals(n, countOf(v.properties("multi")));
+ } finally {
+ graph.close();
+ }
+ }
+
+ @Test
+ public void shouldRoundTripUnicodeKeysAndValues() {
+ TinkerStorageGraph graph = open();
+ try {
+ final Vertex v = graph.addVertex(T.id, 1);
+ v.property("naïve", "café");
+ v.property("日本語", "テスト");
+ v.property("emoji", "party🎉");
+ graph.tx().commit();
+ } finally {
+ graph.close();
+ }
+ graph = open();
+ try {
+ final Vertex v = graph.vertices(1).next();
+ assertEquals("café", v.value("naïve"));
+ assertEquals("テスト", v.value("日本語"));
+ assertEquals("party🎉", v.value("emoji"));
+ } finally {
+ graph.close();
+ }
+ }
+
private static long countOf(final Iterator<?> it) {
long count = 0;
while (it.hasNext()) {
(tinkerpop) 02/09: Encode TinkerStorageGraph elements as dictionary-referenced components
Posted by sp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
spmallette pushed a commit to branch tinkergraph-storage
in repository https://gitbox.apache.org/repos/asf/tinkerpop.git
commit dc72ba579a1245ee774a81924e9a9e792d7aa79a
Author: Stephen Mallette <st...@amazon.com>
AuthorDate: Wed Aug 19 14:31:18 2026 +0000
Encode TinkerStorageGraph elements as dictionary-referenced components
Replace whole-DetachedVertex/DetachedEdge GraphBinary serialization with a
component codec: element ids and property values go through GraphBinary's
scalar serializers (a one-byte DataType tag + raw value), while labels,
property keys, and meta-property keys are dictionary-encoded to small integer
refs. New keys are emitted as OP_DICT_APPEND records within the same frame,
before their first use. Vertex-property ids are auto-generated and no longer
persisted (regenerated on load); element and edge ids are preserved.
Compaction preserves dictionary numbering and writes the whole dictionary as a
self-contained snapshot header, and dict-append decode is idempotent, so a log
surviving the compaction crash window still resolves its refs. This removes the
old whole-object codec (no dead code) and drops per-element key/label
repetition and the per-property envelope, shrinking the on-disk footprint.
Assisted-by: Claude Code:claude-opus-4-8
---
.../structure/storage/GraphBinaryStorage.java | 371 ++++++++++++++++++---
.../structure/storage/GraphBinaryStorageTest.java | 14 +-
.../storage/StorageCrashConsistencyTest.java | 15 +-
3 files changed, 334 insertions(+), 66 deletions(-)
diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java
index a5f3fd1c5c..1266dee705 100644
--- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java
+++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java
@@ -19,34 +19,51 @@
package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage;
import org.apache.tinkerpop.gremlin.structure.Edge;
+import org.apache.tinkerpop.gremlin.structure.Property;
import org.apache.tinkerpop.gremlin.structure.Vertex;
+import org.apache.tinkerpop.gremlin.structure.VertexProperty;
+import org.apache.tinkerpop.gremlin.structure.io.binary.DataType;
import org.apache.tinkerpop.gremlin.structure.io.binary.GraphBinaryReader;
import org.apache.tinkerpop.gremlin.structure.io.binary.GraphBinaryWriter;
+import org.apache.tinkerpop.gremlin.structure.io.binary.TypeSerializer;
import org.apache.tinkerpop.gremlin.structure.io.binary.TypeSerializerRegistry;
import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedEdge;
-import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedFactory;
+import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedProperty;
import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex;
+import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertexProperty;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.AbstractTinkerGraph;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerEdge;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex;
import java.io.DataOutputStream;
import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
import java.util.Collection;
+import java.util.HashMap;
import java.util.Iterator;
+import java.util.LinkedHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
import java.util.Map;
+import java.util.Set;
/**
* The GraphBinary {@link TinkerStorage} codec on top of {@link AbstractLogStorage}. The base owns the durable
* log-structured machinery (file layout, {@code VERSION} marker, CRC framing, replay fold, {@link SyncMode}
- * durability, crash-safe and threshold compaction); this class supplies only how an element is encoded and decoded,
- * using the GraphBinary serializers ({@link GraphBinaryWriter}/{@link GraphBinaryReader}).
+ * durability, crash-safe and threshold compaction); this class supplies only how an element is encoded and decoded.
* <p/>
- * Known limitation (write amplification): a commit records each changed element in full — a single property change on
- * a large element rewrites the whole element to the log. Elements are typically small and automatic compaction bounds
- * the resulting log growth, so this is accepted rather than mitigated with per-property deltas, which would complicate
- * the {@link TinkerStorageMutation} contract and the replay fold. The snapshot, by contrast, is streamed one element
- * at a time so compaction never holds a second full copy of the graph in heap.
+ * Rather than serialize a whole {@code DetachedVertex}/{@code DetachedEdge} (which repeats every property key and
+ * label as a full string on every element and wraps each property in a {@code VertexProperty} envelope), this codec
+ * writes element <em>components</em> directly: element ids and property values go through GraphBinary's scalar
+ * serializers (a one-byte {@link DataType} tag plus the raw value), while labels, property keys, and meta-property
+ * keys are dictionary-encoded to small integer refs. The dictionary is a single dense namespace built by first
+ * appearance; new entries are emitted as {@code OP_DICT_APPEND} records inside the same frame, before the entries
+ * that reference them, so a torn trailing frame drops a ref and its user atomically.
+ * <p/>
+ * Vertex-property ids are auto-generated and, by default, not persisted (they are regenerated on load); element and
+ * edge ids are always preserved. The snapshot streams one element per frame, so compaction never holds a second full
+ * copy of the graph in heap.
*/
public final class GraphBinaryStorage extends AbstractLogStorage {
@@ -54,71 +71,230 @@ public final class GraphBinaryStorage extends AbstractLogStorage {
private static final byte OP_DEL_VERTEX = 2;
private static final byte OP_PUT_EDGE = 3;
private static final byte OP_DEL_EDGE = 4;
+ private static final byte OP_DICT_APPEND = 5;
- private final GraphBinaryWriter writer = new GraphBinaryWriter(TypeSerializerRegistry.INSTANCE);
- private final GraphBinaryReader reader = new GraphBinaryReader(TypeSerializerRegistry.INSTANCE);
+ private final TypeSerializerRegistry registry = TypeSerializerRegistry.INSTANCE;
+ private final GraphBinaryWriter writer = new GraphBinaryWriter(registry);
+ private final GraphBinaryReader reader = new GraphBinaryReader(registry);
/**
- * Serialize a commit record: txVersion, entry count, then each entry as an op byte followed by either the
- * serialized element (put) or the serialized id (delete).
+ * Shared string dictionary for labels, property keys, and meta-property keys. Dense ids assigned by first
+ * appearance; grows monotonically within a write session and is rebuilt fresh on compaction and on replay.
*/
+ private final Map<String, Integer> keyToId = new HashMap<>();
+ private final List<String> idToKey = new ArrayList<>();
+
+ @Override
+ protected void beginReplay() {
+ keyToId.clear();
+ idToKey.clear();
+ }
+
+ // --------------------------------------------------------------------------------------------- encode
+
@Override
protected byte[] encodeCommit(final long txVersion,
final Collection<TinkerStorageMutation<TinkerVertex>> changedVertices,
final Collection<TinkerStorageMutation<TinkerEdge>> changedEdges) throws IOException {
- final ByteBufferBuffer buffer = new ByteBufferBuffer();
- buffer.writeLong(txVersion);
- buffer.writeInt(changedVertices.size() + changedEdges.size());
+ final ByteBufferBuffer buf = new ByteBufferBuffer();
+ // register the strings introduced by this commit (deletes carry only an id, no strings)
+ final List<String> appends = new ArrayList<>();
+ for (final TinkerStorageMutation<TinkerVertex> m : changedVertices)
+ if (!m.isDeleted()) registerVertexStrings(m.element(), appends);
+ for (final TinkerStorageMutation<TinkerEdge> m : changedEdges)
+ if (!m.isDeleted()) registerEdgeStrings(m.element(), appends);
+
+ writeVarInt(buf, appends.size() + changedVertices.size() + changedEdges.size());
+ // dictionary appends first, so every ref below resolves during the fold
+ for (final String s : appends) {
+ buf.writeByte(OP_DICT_APPEND);
+ writeVarInt(buf, keyToId.get(s));
+ writeString(buf, s);
+ }
for (final TinkerStorageMutation<TinkerVertex> m : changedVertices) {
if (m.isDeleted()) {
- buffer.writeByte(OP_DEL_VERTEX);
- writer.write(m.id(), buffer);
+ buf.writeByte(OP_DEL_VERTEX);
+ writeScalar(buf, m.id());
} else {
- buffer.writeByte(OP_PUT_VERTEX);
- // detach to a stable form independent of the transactional element
- writer.write(DetachedFactory.detach(m.element(), true), buffer);
+ buf.writeByte(OP_PUT_VERTEX);
+ writeVertexRecord(buf, m.element());
}
}
for (final TinkerStorageMutation<TinkerEdge> m : changedEdges) {
if (m.isDeleted()) {
- buffer.writeByte(OP_DEL_EDGE);
- writer.write(m.id(), buffer);
+ buf.writeByte(OP_DEL_EDGE);
+ writeScalar(buf, m.id());
} else {
- buffer.writeByte(OP_PUT_EDGE);
- writer.write(DetachedFactory.detach(m.element(), true), buffer);
+ buf.writeByte(OP_PUT_EDGE);
+ writeEdgeRecord(buf, m.element());
+ }
+ }
+ return buf.toWrittenArray();
+ }
+
+ @Override
+ protected void writeSnapshot(final AbstractTinkerGraph graph, final DataOutputStream out) throws IOException {
+ // Preserve the existing dictionary numbering rather than renumbering: the compaction crash window can leave
+ // the new snapshot in place with the old log not yet truncated, and that log's refs use the current
+ // numbering. Register any not-yet-seen live strings (this only extends the dictionary, never renumbers), then
+ // emit the whole dictionary as a self-contained header frame so a snapshot-only replay resolves every ref.
+ final List<String> ignored = new ArrayList<>();
+ Iterator<Vertex> vertices = graph.vertices();
+ while (vertices.hasNext())
+ registerVertexStrings(vertices.next(), ignored);
+ Iterator<Edge> edges = graph.edges();
+ while (edges.hasNext())
+ registerEdgeStrings(edges.next(), ignored);
+
+ final ByteBufferBuffer dictBuf = new ByteBufferBuffer();
+ writeVarInt(dictBuf, idToKey.size());
+ for (int id = 0; id < idToKey.size(); id++) {
+ dictBuf.writeByte(OP_DICT_APPEND);
+ writeVarInt(dictBuf, id);
+ writeString(dictBuf, idToKey.get(id));
+ }
+ writeFrame(out, dictBuf.toWrittenArray());
+
+ // one element per frame; all keys are already in the dictionary header, so no per-frame appends
+ vertices = graph.vertices();
+ while (vertices.hasNext())
+ writeElementFrame(out, OP_PUT_VERTEX, vertices.next());
+ edges = graph.edges();
+ while (edges.hasNext())
+ writeElementFrame(out, OP_PUT_EDGE, edges.next());
+ }
+
+ /**
+ * Write one element as its own single-entry put frame. One element is held in memory at a time, so a large graph
+ * is never buffered whole.
+ */
+ private void writeElementFrame(final DataOutputStream out, final byte op, final Object element) throws IOException {
+ final ByteBufferBuffer buf = new ByteBufferBuffer();
+ writeVarInt(buf, 1);
+ buf.writeByte(op);
+ if (op == OP_PUT_VERTEX) writeVertexRecord(buf, (Vertex) element);
+ else writeEdgeRecord(buf, (Edge) element);
+ writeFrame(out, buf.toWrittenArray());
+ }
+
+ private void registerVertexStrings(final Vertex v, final List<String> appends) {
+ for (final String label : v.labels())
+ register(label, appends);
+ final Iterator<VertexProperty<Object>> vps = v.properties();
+ while (vps.hasNext()) {
+ final VertexProperty<Object> vp = vps.next();
+ register(vp.key(), appends);
+ final Iterator<Property<Object>> metas = vp.properties();
+ while (metas.hasNext())
+ register(metas.next().key(), appends);
+ }
+ }
+
+ private void registerEdgeStrings(final Edge e, final List<String> appends) {
+ register(e.label(), appends);
+ final Iterator<Property<Object>> props = e.properties();
+ while (props.hasNext())
+ register(props.next().key(), appends);
+ }
+
+ private void register(final String s, final List<String> appends) {
+ if (!keyToId.containsKey(s)) {
+ final int id = idToKey.size();
+ keyToId.put(s, id);
+ idToKey.add(s);
+ appends.add(s);
+ }
+ }
+
+ private void writeVertexRecord(final ByteBufferBuffer buf, final Vertex v) throws IOException {
+ writeScalar(buf, v.id());
+ final Set<String> labels = v.labels();
+ writeVarInt(buf, labels.size());
+ for (final String label : labels)
+ writeVarInt(buf, keyToId.get(label));
+
+ // group vertex properties by key so multi-properties (list/set) round-trip
+ final Map<String, List<VertexProperty<Object>>> groups = new LinkedHashMap<>();
+ final Iterator<VertexProperty<Object>> vps = v.properties();
+ while (vps.hasNext()) {
+ final VertexProperty<Object> vp = vps.next();
+ groups.computeIfAbsent(vp.key(), k -> new ArrayList<>()).add(vp);
+ }
+ writeVarInt(buf, groups.size());
+ for (final Map.Entry<String, List<VertexProperty<Object>>> group : groups.entrySet()) {
+ writeVarInt(buf, keyToId.get(group.getKey()));
+ final List<VertexProperty<Object>> values = group.getValue();
+ writeVarInt(buf, values.size());
+ for (final VertexProperty<Object> vp : values) {
+ writeScalar(buf, vp.value());
+ final List<Property<Object>> metas = new ArrayList<>();
+ vp.properties().forEachRemaining(metas::add);
+ writeVarInt(buf, metas.size());
+ for (final Property<Object> meta : metas) {
+ writeVarInt(buf, keyToId.get(meta.key()));
+ writeScalar(buf, meta.value());
+ }
}
}
- return buffer.toWrittenArray();
}
+ private void writeEdgeRecord(final ByteBufferBuffer buf, final Edge e) throws IOException {
+ writeScalar(buf, e.id());
+ writeVarInt(buf, keyToId.get(e.label()));
+ writeScalar(buf, e.outVertex().id());
+ writeScalar(buf, e.inVertex().id());
+ final List<Property<Object>> props = new ArrayList<>();
+ e.properties().forEachRemaining(props::add);
+ writeVarInt(buf, props.size());
+ for (final Property<Object> p : props) {
+ writeVarInt(buf, keyToId.get(p.key()));
+ writeScalar(buf, p.value());
+ }
+ }
+
+ // --------------------------------------------------------------------------------------------- decode
+
@Override
protected void decodeFrame(final byte[] record,
final Map<Object, DetachedVertex> vertices,
final Map<Object, DetachedEdge> edges) throws IOException {
- final ByteBufferBuffer buffer = new ByteBufferBuffer(record);
- buffer.readLong(); // txVersion, retained for diagnostics/future use
- final int entryCount = buffer.readInt();
+ final ByteBufferBuffer buf = new ByteBufferBuffer(record);
+ final int entryCount = readVarInt(buf);
for (int i = 0; i < entryCount; i++) {
- final byte op = buffer.readByte();
+ final byte op = buf.readByte();
switch (op) {
+ case OP_DICT_APPEND: {
+ final int id = readVarInt(buf);
+ final String s = readString(buf);
+ // idempotent: a snapshot header defines the whole dictionary, and a log surviving the compaction
+ // crash window may re-append entries the snapshot already established. Re-appending an existing
+ // id with the same string is a no-op; a mismatch or a gap is corruption.
+ if (id < idToKey.size()) {
+ if (!idToKey.get(id).equals(s))
+ throw new IOException(String.format("Corrupt storage: dictionary id %d redefined ('%s' vs '%s')", id, idToKey.get(id), s));
+ } else if (id == idToKey.size()) {
+ idToKey.add(s);
+ } else {
+ throw new IOException(String.format("Corrupt storage: dictionary append gap (got %d, expected <= %d)", id, idToKey.size()));
+ }
+ break;
+ }
case OP_PUT_VERTEX: {
- final Vertex v = reader.read(buffer);
- vertices.put(v.id(), (DetachedVertex) v);
+ final DetachedVertex v = readVertexRecord(buf);
+ vertices.put(v.id(), v);
break;
}
case OP_DEL_VERTEX: {
- final Object id = reader.read(buffer);
- vertices.remove(id);
+ vertices.remove(readScalar(buf));
break;
}
case OP_PUT_EDGE: {
- final Edge e = reader.read(buffer);
- edges.put(e.id(), (DetachedEdge) e);
+ final DetachedEdge e = readEdgeRecord(buf);
+ edges.put(e.id(), e);
break;
}
case OP_DEL_EDGE: {
- final Object id = reader.read(buffer);
- edges.remove(id);
+ edges.remove(readScalar(buf));
break;
}
default:
@@ -127,29 +303,114 @@ public final class GraphBinaryStorage extends AbstractLogStorage {
}
}
+ private DetachedVertex readVertexRecord(final ByteBufferBuffer buf) throws IOException {
+ final Object id = readScalar(buf);
+ final DetachedVertex.Builder b = DetachedVertex.build().setId(id);
+ final int labelCount = readVarInt(buf);
+ if (labelCount == 1) {
+ b.setLabel(idToKey.get(readVarInt(buf)));
+ } else if (labelCount > 1) {
+ final Set<String> labels = new LinkedHashSet<>();
+ for (int i = 0; i < labelCount; i++)
+ labels.add(idToKey.get(readVarInt(buf)));
+ b.setLabels(labels);
+ }
+ final int keyGroupCount = readVarInt(buf);
+ for (int g = 0; g < keyGroupCount; g++) {
+ final String key = idToKey.get(readVarInt(buf));
+ final int valueCount = readVarInt(buf);
+ for (int j = 0; j < valueCount; j++) {
+ final Object value = readScalar(buf);
+ final DetachedVertexProperty.Builder vpb = DetachedVertexProperty.build().setLabel(key).setValue(value);
+ final int metaCount = readVarInt(buf);
+ for (int m = 0; m < metaCount; m++) {
+ final String metaKey = idToKey.get(readVarInt(buf));
+ final Object metaValue = readScalar(buf);
+ vpb.addProperty(new DetachedProperty<>(metaKey, metaValue));
+ }
+ b.addProperty(vpb.create());
+ }
+ }
+ return b.create();
+ }
+
+ private DetachedEdge readEdgeRecord(final ByteBufferBuffer buf) throws IOException {
+ final Object id = readScalar(buf);
+ final String label = idToKey.get(readVarInt(buf));
+ final Object outVId = readScalar(buf);
+ final Object inVId = readScalar(buf);
+ final DetachedEdge.Builder b = DetachedEdge.build().setId(id).setLabel(label)
+ .setOutV(DetachedVertex.build().setId(outVId).create())
+ .setInV(DetachedVertex.build().setId(inVId).create());
+ final int propCount = readVarInt(buf);
+ for (int i = 0; i < propCount; i++) {
+ final String key = idToKey.get(readVarInt(buf));
+ final Object value = readScalar(buf);
+ b.addProperty(new DetachedProperty<>(key, value));
+ }
+ return b.create();
+ }
+
+ // --------------------------------------------------------------------------------------------- primitives
+
/**
- * Stream the current committed state as one framed put record per vertex and per edge, so peak memory is bounded
- * to a single element rather than materializing the whole graph as one byte array.
+ * Write a value as a one-byte {@link DataType} tag followed by the raw value (no value-flag byte). A {@code null}
+ * is a single {@link DataType#UNSPECIFIED_NULL} tag.
*/
- @Override
- protected void writeSnapshot(final AbstractTinkerGraph graph, final DataOutputStream out) throws IOException {
- final Iterator<Vertex> vertexIterator = graph.vertices();
- while (vertexIterator.hasNext())
- writeElementFrame(out, OP_PUT_VERTEX, vertexIterator.next());
- final Iterator<Edge> edgeIterator = graph.edges();
- while (edgeIterator.hasNext())
- writeElementFrame(out, OP_PUT_EDGE, edgeIterator.next());
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private void writeScalar(final ByteBufferBuffer buf, final Object value) throws IOException {
+ if (value == null) {
+ buf.writeByte(DataType.UNSPECIFIED_NULL.getCodeByte());
+ return;
+ }
+ final TypeSerializer serializer = registry.getSerializer(value.getClass());
+ buf.writeByte(serializer.getDataType().getCodeByte());
+ serializer.writeValue(value, buf, writer, false);
+ }
+
+ @SuppressWarnings({"unchecked", "rawtypes"})
+ private Object readScalar(final ByteBufferBuffer buf) throws IOException {
+ final DataType dataType = DataType.get(Byte.toUnsignedInt(buf.readByte()));
+ if (dataType == DataType.UNSPECIFIED_NULL)
+ return null;
+ final TypeSerializer serializer = registry.getSerializer(dataType);
+ return serializer.readValue(buf, reader, false);
+ }
+
+ private static void writeString(final ByteBufferBuffer buf, final String s) {
+ final byte[] bytes = s.getBytes(StandardCharsets.UTF_8);
+ writeVarInt(buf, bytes.length);
+ buf.writeBytes(bytes);
+ }
+
+ private static String readString(final ByteBufferBuffer buf) {
+ final byte[] bytes = new byte[readVarInt(buf)];
+ buf.readBytes(bytes);
+ return new String(bytes, StandardCharsets.UTF_8);
}
/**
- * Encode a single element as a one-entry put record and write it as a framed record to {@code out}.
+ * Unsigned LEB128 varint. Counts and dictionary refs are small and non-negative, so they cost one byte in the
+ * common case.
*/
- private void writeElementFrame(final DataOutputStream out, final byte op, final Object element) throws IOException {
- final ByteBufferBuffer buffer = new ByteBufferBuffer();
- buffer.writeLong(0L); // snapshot records have no single tx version
- buffer.writeInt(1);
- buffer.writeByte(op);
- writer.write(DetachedFactory.detach(element, true), buffer);
- writeFrame(out, buffer.toWrittenArray());
+ private static void writeVarInt(final ByteBufferBuffer buf, final int value) {
+ int v = value;
+ while ((v & ~0x7F) != 0) {
+ buf.writeByte((v & 0x7F) | 0x80);
+ v >>>= 7;
+ }
+ buf.writeByte(v & 0x7F);
+ }
+
+ private static int readVarInt(final ByteBufferBuffer buf) {
+ int result = 0;
+ int shift = 0;
+ byte b;
+ do {
+ b = buf.readByte();
+ result |= (b & 0x7F) << shift;
+ shift += 7;
+ } while ((b & 0x80) != 0);
+ return result;
}
}
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java
index c113511176..cbde701a22 100644
--- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java
+++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java
@@ -127,10 +127,10 @@ public class GraphBinaryStorageTest extends AbstractTinkerStorageConformanceTest
graph.tx().commit();
graph.compact();
- // the snapshot must be written as one framed record per element (3 vertices + 2 edges = 5), rather than a
- // single whole-graph frame, so compaction never buffers the entire graph in one array
+ // the snapshot must be streamed: a dictionary header frame plus one framed record per element (3 vertices +
+ // 2 edges = 5), rather than a single whole-graph frame, so compaction never buffers the entire graph at once
final File snapshotFile = new File(location, GraphBinaryStorage.SNAPSHOT_FILE);
- assertEquals(5, countFrames(snapshotFile));
+ assertEquals(1 + 5, countFrames(snapshotFile));
graph.close();
// and the streamed snapshot must reopen to exactly the same graph
@@ -145,9 +145,9 @@ public class GraphBinaryStorageTest extends AbstractTinkerStorageConformanceTest
@Test
public void shouldStreamSnapshotFrameByFrameAtScale() {
// Bounded-memory proxy for the streaming snapshot path: rather than measure heap (flaky, JVM-dependent), assert
- // the observable streaming property holds at scale — a large graph is written as exactly one frame per element,
- // never one whole-graph frame — and round-trips intact. This is the property that keeps compaction from
- // materializing a second full copy of the graph in memory; it is not a hard OOM assertion.
+ // the observable streaming property holds at scale — a large graph is written as a dictionary header frame plus
+ // one frame per element, never one whole-graph frame — and round-trips intact. This is the property that keeps
+ // compaction from materializing a second full copy of the graph in memory; it is not a hard OOM assertion.
final int vertexCount = 500;
final int edgeCount = 499;
final TinkerStorageGraph graph = open();
@@ -161,7 +161,7 @@ public class GraphBinaryStorageTest extends AbstractTinkerStorageConformanceTest
graph.compact();
try {
- assertEquals(vertexCount + edgeCount, countFrames(new File(location, GraphBinaryStorage.SNAPSHOT_FILE)));
+ assertEquals(1 + vertexCount + edgeCount, countFrames(new File(location, GraphBinaryStorage.SNAPSHOT_FILE)));
} catch (Exception ex) {
throw new RuntimeException(ex);
}
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCrashConsistencyTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCrashConsistencyTest.java
index 614cfdb0ba..8f3c475f13 100644
--- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCrashConsistencyTest.java
+++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCrashConsistencyTest.java
@@ -123,10 +123,17 @@ public class StorageCrashConsistencyTest {
@Test
public void shouldRecoverDurableCommitWithNoSnapshot() throws Exception {
- // WAL guarantee: a commit whose frame was durably written to the log, with no compaction having run, must
- // recover on reopen even though no snapshot exists.
- captureBuildingBlocks();
- Files.write(logFile.toPath(), logV2);
+ // WAL guarantee: a commit durably written to the log with no compaction (no snapshot) recovers on reopen. On
+ // a fresh store the dictionary starts empty, so the commit frame is self-contained (it carries its own
+ // dictionary appends), which is exactly the real "log only, never compacted" case.
+ final TinkerStorageGraph g = open();
+ g.addVertex(T.id, 2, "value", 2);
+ g.tx().commit();
+ g.tx().close();
+ final byte[] selfContainedLog = Files.readAllBytes(logFile.toPath());
+ g.close(); // compaction on close would fold the log away; the raw log was captured above
+ resetFiles();
+ Files.write(logFile.toPath(), selfContainedLog);
assertReopensTo(2);
}
(tinkerpop) 06/09: Test TinkerStorage durability, lifecycle, and dictionary crash window
Posted by sp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
spmallette pushed a commit to branch tinkergraph-storage
in repository https://gitbox.apache.org/repos/asf/tinkerpop.git
commit 379deede9256845a4fc9fadedae2cf6c2a18da79
Author: Stephen Mallette <st...@amazon.com>
AuthorDate: Wed Aug 19 16:34:36 2026 +0000
Test TinkerStorage durability, lifecycle, and dictionary crash window
Add TCK cases for repeated compaction cycles, multiple open/close sessions,
and concurrent commits with distinct keys. Add a crash-consistency test for the
compaction crash window where a dead dictionary key forces the surviving log to
diverge from the new snapshot's dictionary — guarding the decision to preserve
dictionary numbering across compaction rather than renumber.
Assisted-by: Claude Code:claude-opus-4-8
---
.../AbstractTinkerStorageConformanceTest.java | 112 +++++++++++++++++++++
.../storage/StorageCrashConsistencyTest.java | 43 ++++++++
2 files changed, 155 insertions(+)
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java
index f5ec5033f4..34afede360 100644
--- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java
+++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractTinkerStorageConformanceTest.java
@@ -490,6 +490,118 @@ public abstract class AbstractTinkerStorageConformanceTest {
}
}
+ @Test
+ public void shouldSurviveRepeatedCompactionCycles() {
+ TinkerStorageGraph graph = open();
+ try {
+ graph.addVertex(T.id, 1, "a", 1);
+ graph.tx().commit();
+ graph.compact();
+ graph.addVertex(T.id, 2, "b", 2);
+ graph.tx().commit();
+ graph.compact();
+ graph.addVertex(T.id, 3, "c", 3);
+ graph.tx().commit();
+ graph.compact();
+ } finally {
+ graph.close();
+ }
+ graph = open();
+ try {
+ assertEquals(3, countOf(graph.vertices()));
+ assertEquals(Integer.valueOf(1), graph.vertices(1).next().value("a"));
+ assertEquals(Integer.valueOf(2), graph.vertices(2).next().value("b"));
+ assertEquals(Integer.valueOf(3), graph.vertices(3).next().value("c"));
+ } finally {
+ graph.close();
+ }
+ }
+
+ @Test
+ public void shouldSurviveMultipleOpenCloseSessions() {
+ TinkerStorageGraph graph = open();
+ try {
+ graph.addVertex(T.id, 1, "n", "one");
+ graph.tx().commit();
+ } finally {
+ graph.close();
+ }
+ graph = open();
+ try {
+ graph.addVertex(T.id, 2, "n", "two");
+ graph.tx().commit();
+ } finally {
+ graph.close();
+ }
+ graph = open();
+ try {
+ graph.addVertex(T.id, 3, "n", "three");
+ graph.tx().commit();
+ graph.compact();
+ } finally {
+ graph.close();
+ }
+ graph = open();
+ try {
+ assertEquals(3, countOf(graph.vertices()));
+ assertEquals("one", graph.vertices(1).next().value("n"));
+ assertEquals("two", graph.vertices(2).next().value("n"));
+ assertEquals("three", graph.vertices(3).next().value("n"));
+ } finally {
+ graph.close();
+ }
+ }
+
+ @Test
+ public void shouldRoundTripConcurrentCommitsWithDistinctKeys() throws Exception {
+ // concurrent commits that each introduce a distinct property key stress dictionary growth under the
+ // commit-write lock; on reopen every distinct key must resolve
+ final int threads = 8;
+ final int perThread = 25;
+ final TinkerStorageGraph writeGraph = open();
+ try {
+ final ExecutorService pool = Executors.newFixedThreadPool(threads);
+ final CountDownLatch start = new CountDownLatch(1);
+ final List<Future<?>> futures = new ArrayList<>();
+ for (int t = 0; t < threads; t++) {
+ final int threadId = t;
+ futures.add(pool.submit(() -> {
+ start.await();
+ for (int i = 0; i < perThread; i++) {
+ final int id = threadId * perThread + i;
+ writeGraph.addVertex(T.id, id, "k_" + threadId + "_" + i, id);
+ writeGraph.tx().commit();
+ }
+ return null;
+ }));
+ }
+ start.countDown();
+ for (final Future<?> f : futures)
+ f.get(60, TimeUnit.SECONDS);
+ pool.shutdown();
+ assertTrue(pool.awaitTermination(60, TimeUnit.SECONDS));
+ } finally {
+ writeGraph.close();
+ }
+ final TinkerStorageGraph reopened = open();
+ try {
+ final int expected = threads * perThread;
+ assertEquals(expected, countOf(reopened.vertices()));
+ for (int t = 0; t < threads; t++) {
+ for (int i = 0; i < perThread; i++) {
+ final int id = threadId(t, i, perThread);
+ assertEquals(Integer.valueOf(id), reopened.vertices(id).next().value("k_" + t + "_" + i));
+ }
+ }
+ } finally {
+ reopened.close();
+ }
+ }
+
+ private static int threadId(final int t, final int i, final int perThread) {
+ return t * perThread + i;
+ }
+
private static long countOf(final Iterator<?> it) {
long count = 0;
while (it.hasNext()) {
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCrashConsistencyTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCrashConsistencyTest.java
index 8f3c475f13..4bbb3854ce 100644
--- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCrashConsistencyTest.java
+++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/StorageCrashConsistencyTest.java
@@ -22,6 +22,7 @@ import org.apache.commons.configuration2.BaseConfiguration;
import org.apache.commons.configuration2.Configuration;
import org.apache.tinkerpop.gremlin.structure.Graph;
import org.apache.tinkerpop.gremlin.structure.T;
+import org.apache.tinkerpop.gremlin.structure.Vertex;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph;
import org.junit.Before;
@@ -169,6 +170,48 @@ public class StorageCrashConsistencyTest {
assertReopensTo(1, 2);
}
+ @Test
+ public void shouldRecoverCrashWindowWithADeadKeyForcingDictionaryDivergence() throws Exception {
+ // Guards the preserve-dictionary-numbering decision. A key ("alpha") is deleted from the live graph after it
+ // is in the dictionary, then a new key ("gamma") is added. Preserving numbering keeps alpha's id forever, so
+ // the surviving log's gamma ref still matches the new snapshot's dictionary. Renumbering on compaction would
+ // instead drop the now-dead alpha and shift gamma to a lower id, so the log's higher-numbered gamma ref would
+ // no longer resolve. A single-key (or no-delete) state cannot tell the two apart.
+ final TinkerStorageGraph g = open();
+ g.addVertex(T.id, 1, "alpha", 1);
+ g.tx().commit();
+ g.addVertex(T.id, 2, "beta", 2);
+ g.tx().commit();
+ g.compact(); // snapshot holds alpha and beta in the dictionary at stable ids
+
+ g.vertices(1).next().remove(); // alpha becomes a dead key: retained only under preserve-numbering
+ g.tx().commit();
+ // re-write the surviving vertex: its record now carries a bare reference to the pre-existing key "beta"
+ // (not re-appended) plus a new key "gamma". If compaction renumbered, "beta"'s id would shift and this bare
+ // reference would resolve to the wrong key on replay.
+ g.vertices(2).next().property("gamma", "g");
+ g.tx().commit();
+ final byte[] logNotYetDeleted = Files.readAllBytes(logFile.toPath());
+ g.compact(); // new full-dictionary snapshot (preserved numbering), then log truncated
+ final byte[] newSnapshot = Files.readAllBytes(snapshotFile.toPath());
+ g.close();
+
+ // reconstruct the crash window: new snapshot in place, old log not yet deleted
+ Files.write(snapshotFile.toPath(), newSnapshot);
+ Files.write(logFile.toPath(), logNotYetDeleted);
+
+ final TinkerStorageGraph reopened = open();
+ try {
+ assertEquals(1, countOf(reopened.vertices())); // only the surviving vertex 2
+ assertEquals(0, countOf(reopened.vertices(1))); // alpha's vertex was deleted
+ final Vertex v = reopened.vertices(2).next();
+ assertEquals(Integer.valueOf(2), v.value("beta"));
+ assertEquals("g", v.value("gamma"));
+ } finally {
+ reopened.close();
+ }
+ }
+
private static long countOf(final Iterator<?> it) {
long count = 0;
while (it.hasNext()) {
(tinkerpop) 03/09: Add opt-in vertex-property id persistence to TinkerStorageGraph storage
Posted by sp...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
spmallette pushed a commit to branch tinkergraph-storage
in repository https://gitbox.apache.org/repos/asf/tinkerpop.git
commit 7832c81a9c53879a8e27c0d544dba0d28baf00b1
Author: Stephen Mallette <st...@amazon.com>
AuthorDate: Wed Aug 19 14:37:00 2026 +0000
Add opt-in vertex-property id persistence to TinkerStorageGraph storage
Add gremlin.tinkergraph.storage.preserveVertexPropertyIds (default false). When
enabled, the GraphBinary codec persists each vertex-property id so it is stable
across reopen; by default those ids are regenerated on load to keep the store
smaller. A per-vertex-record flag makes each record self-describing, so a store
written with the option reopens correctly even if the reader's setting differs.
A new configureCodec hook lets the codec read its own config at open.
Also adds codec tests: dictionary growth across log commits, dictionary rewrite
on compaction, preserve-ids on/off, and a bytes/element size-regression guard
(well under the ~168 bytes/element the old whole-object format cost).
Assisted-by: Claude Code:claude-opus-4-8
---
.../gremlin/tinkergraph/structure/TinkerGraph.java | 8 ++
.../structure/storage/AbstractLogStorage.java | 8 ++
.../structure/storage/GraphBinaryStorage.java | 21 ++++
.../structure/storage/GraphBinaryStorageTest.java | 119 +++++++++++++++++++++
4 files changed, 156 insertions(+)
diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java
index 65a02537c6..f011cd5202 100644
--- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java
+++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/TinkerGraph.java
@@ -75,6 +75,14 @@ public interface TinkerGraph extends Graph {
* set.
*/
String GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD = "gremlin.tinkergraph.storage.compactThreshold";
+ /**
+ * Whether a {@link TinkerStorageGraph} storage engine persists auto-generated vertex-property ids so they are
+ * stable across a close and reopen. Defaults to {@code false}: vertex-property ids are regenerated on load, which
+ * keeps the store smaller. Element and edge ids are always preserved regardless of this setting. Each record is
+ * self-describing, so a store written with this enabled reopens correctly even if the setting later differs. Only
+ * meaningful when {@link #GREMLIN_TINKERGRAPH_STORAGE} is set.
+ */
+ String GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS = "gremlin.tinkergraph.storage.preserveVertexPropertyIds";
String GREMLIN_TINKERGRAPH_ALLOW_NULL_PROPERTY_VALUES = "gremlin.tinkergraph.allowNullPropertyValues";
String GREMLIN_TINKERGRAPH_SERVICE = "gremlin.tinkergraph.service";
String GREMLIN_TINKERGRAPH_VERTEX_LABEL_CARDINALITY = "gremlin.tinkergraph.vertexLabelCardinality";
diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractLogStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractLogStorage.java
index b2bdf249f5..4c8eefeb08 100644
--- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractLogStorage.java
+++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/AbstractLogStorage.java
@@ -155,10 +155,18 @@ public abstract class AbstractLogStorage implements TinkerStorage {
TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, DEFAULT_COMPACT_THRESHOLD_BYTES);
// seed the counter with any pre-existing log so a graph reopened with a large log still compacts promptly
this.logBytesSinceCompaction = logFile.exists() ? logFile.length() : 0;
+ configureCodec(config);
ensureDirectory();
establishStoreVersion();
}
+ /**
+ * Read any codec-specific configuration. Called once during {@link #open}. Default is a no-op.
+ */
+ protected void configureCodec(final Configuration config) {
+ // no-op by default
+ }
+
@Override
public void replay(final AbstractTinkerGraph graph) {
beginReplay();
diff --git a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java
index 1266dee705..7c6d0cff77 100644
--- a/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java
+++ b/tinkergraph-gremlin/src/main/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorage.java
@@ -18,6 +18,7 @@
*/
package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage;
+import org.apache.commons.configuration2.Configuration;
import org.apache.tinkerpop.gremlin.structure.Edge;
import org.apache.tinkerpop.gremlin.structure.Property;
import org.apache.tinkerpop.gremlin.structure.Vertex;
@@ -33,6 +34,7 @@ import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertex;
import org.apache.tinkerpop.gremlin.structure.util.detached.DetachedVertexProperty;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.AbstractTinkerGraph;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerEdge;
+import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerVertex;
import java.io.DataOutputStream;
@@ -84,6 +86,17 @@ public final class GraphBinaryStorage extends AbstractLogStorage {
private final Map<String, Integer> keyToId = new HashMap<>();
private final List<String> idToKey = new ArrayList<>();
+ /**
+ * When true, persist auto-generated vertex-property ids so they are stable across reopen. Written per vertex
+ * record so a store reopens correctly regardless of the reader's setting.
+ */
+ private boolean preserveVertexPropertyIds = false;
+
+ @Override
+ protected void configureCodec(final Configuration config) {
+ this.preserveVertexPropertyIds = config.getBoolean(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS, false);
+ }
+
@Override
protected void beginReplay() {
keyToId.clear();
@@ -213,6 +226,9 @@ public final class GraphBinaryStorage extends AbstractLogStorage {
for (final String label : labels)
writeVarInt(buf, keyToId.get(label));
+ // self-describing flag: whether each value below carries a persisted vertex-property id
+ buf.writeByte(preserveVertexPropertyIds ? 1 : 0);
+
// group vertex properties by key so multi-properties (list/set) round-trip
final Map<String, List<VertexProperty<Object>>> groups = new LinkedHashMap<>();
final Iterator<VertexProperty<Object>> vps = v.properties();
@@ -227,6 +243,8 @@ public final class GraphBinaryStorage extends AbstractLogStorage {
writeVarInt(buf, values.size());
for (final VertexProperty<Object> vp : values) {
writeScalar(buf, vp.value());
+ if (preserveVertexPropertyIds)
+ writeScalar(buf, vp.id());
final List<Property<Object>> metas = new ArrayList<>();
vp.properties().forEachRemaining(metas::add);
writeVarInt(buf, metas.size());
@@ -315,6 +333,7 @@ public final class GraphBinaryStorage extends AbstractLogStorage {
labels.add(idToKey.get(readVarInt(buf)));
b.setLabels(labels);
}
+ final boolean hasVpIds = buf.readByte() != 0;
final int keyGroupCount = readVarInt(buf);
for (int g = 0; g < keyGroupCount; g++) {
final String key = idToKey.get(readVarInt(buf));
@@ -322,6 +341,8 @@ public final class GraphBinaryStorage extends AbstractLogStorage {
for (int j = 0; j < valueCount; j++) {
final Object value = readScalar(buf);
final DetachedVertexProperty.Builder vpb = DetachedVertexProperty.build().setLabel(key).setValue(value);
+ if (hasVpIds)
+ vpb.setId(readScalar(buf));
final int metaCount = readVarInt(buf);
for (int m = 0; m < metaCount; m++) {
final String metaKey = idToKey.get(readVarInt(buf));
diff --git a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java
index cbde701a22..f5347f0920 100644
--- a/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java
+++ b/tinkergraph-gremlin/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/storage/GraphBinaryStorageTest.java
@@ -21,6 +21,7 @@ package org.apache.tinkerpop.gremlin.tinkergraph.structure.storage;
import org.apache.commons.configuration2.Configuration;
import org.apache.tinkerpop.gremlin.structure.T;
import org.apache.tinkerpop.gremlin.structure.Vertex;
+import org.apache.tinkerpop.gremlin.structure.VertexProperty;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerGraph;
import org.apache.tinkerpop.gremlin.tinkergraph.structure.TinkerStorageGraph;
import org.junit.Test;
@@ -384,6 +385,124 @@ public class GraphBinaryStorageTest extends AbstractTinkerStorageConformanceTest
}
}
+ @Test
+ public void shouldRegenerateVertexPropertyIdsByDefault() {
+ // default: vertex-property ids are not persisted; the property still round-trips (value + meta), the id is
+ // simply reassigned on load
+ TinkerStorageGraph graph = open();
+ try {
+ final VertexProperty<Object> vp = graph.addVertex(T.id, 1).property(VertexProperty.Cardinality.list, "name", "marko");
+ vp.property("since", 2010);
+ graph.tx().commit();
+ } finally {
+ graph.close();
+ }
+ graph = open();
+ try {
+ final VertexProperty<Object> vp = graph.vertices(1).next().<Object>properties("name").next();
+ assertEquals("marko", vp.value());
+ assertEquals(Integer.valueOf(2010), vp.<Integer>property("since").value());
+ } finally {
+ graph.close();
+ }
+ }
+
+ @Test
+ public void shouldPreserveVertexPropertyIdsWhenConfigured() {
+ final Configuration conf = config();
+ conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_PRESERVE_VP_IDS, true);
+ final Object vpId;
+ TinkerStorageGraph graph = TinkerStorageGraph.open(conf);
+ try {
+ final VertexProperty<Object> vp = graph.addVertex(T.id, 1).property(VertexProperty.Cardinality.list, "name", "marko");
+ vpId = vp.id();
+ graph.tx().commit();
+ } finally {
+ graph.close();
+ }
+ graph = TinkerStorageGraph.open(conf);
+ try {
+ final VertexProperty<Object> vp = graph.vertices(1).next().<Object>properties("name").next();
+ assertEquals("marko", vp.value());
+ assertEquals("vertex-property id should be preserved across reopen", vpId, vp.id());
+ } finally {
+ graph.close();
+ }
+ }
+
+ @Test
+ public void shouldReplayDictionaryGrowthAcrossLogCommits() throws Exception {
+ // each commit introduces a new property key, so the dictionary grows via OP_DICT_APPEND across successive log
+ // frames. Reopening from a log with no snapshot must rebuild the dictionary incrementally and resolve all refs.
+ final Configuration conf = config();
+ conf.setProperty(TinkerGraph.GREMLIN_TINKERGRAPH_STORAGE_COMPACT_THRESHOLD, 0L); // keep the log, no auto-compaction
+ final String location = conf.getString(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION);
+ TinkerStorageGraph graph = TinkerStorageGraph.open(conf);
+ for (int i = 0; i < 10; i++) {
+ graph.addVertex(T.id, i, "key" + i, i);
+ graph.tx().commit();
+ }
+ graph.tx().close();
+ final byte[] log = Files.readAllBytes(new File(location, GraphBinaryStorage.LOG_FILE).toPath());
+ graph.close();
+
+ // restore a log-only store (no snapshot) and reopen
+ Files.deleteIfExists(new File(location, GraphBinaryStorage.SNAPSHOT_FILE).toPath());
+ Files.write(new File(location, GraphBinaryStorage.LOG_FILE).toPath(), log);
+ graph = TinkerStorageGraph.open(conf);
+ try {
+ assertEquals(10, countOf(graph.vertices()));
+ for (int i = 0; i < 10; i++)
+ assertEquals(Integer.valueOf(i), graph.vertices(i).next().value("key" + i));
+ } finally {
+ graph.close();
+ }
+ }
+
+ @Test
+ public void shouldRewriteDictionaryOnCompactionAndReopen() {
+ TinkerStorageGraph graph = open();
+ try {
+ for (int i = 0; i < 10; i++)
+ graph.addVertex(T.id, i, "key" + i, i);
+ graph.tx().commit();
+ graph.compact(); // writes a fresh full-dictionary snapshot header, then element frames
+ } finally {
+ graph.close();
+ }
+ graph = open();
+ try {
+ assertEquals(10, countOf(graph.vertices()));
+ for (int i = 0; i < 10; i++)
+ assertEquals(Integer.valueOf(i), graph.vertices(i).next().value("key" + i));
+ } finally {
+ graph.close();
+ }
+ }
+
+ @Test
+ public void shouldStoreFewBytesPerElement() {
+ // regression guard: the dictionary-encoded format must stay well under the ~168 bytes/element the old
+ // whole-object format cost for a comparable graph (3 vertex props, 2 edge props, E=V).
+ final int vertexCount = 200;
+ final TinkerStorageGraph graph = open();
+ final String location = graph.configuration().getString(TinkerGraph.GREMLIN_TINKERGRAPH_GRAPH_LOCATION);
+ try {
+ for (int i = 0; i < vertexCount; i++)
+ graph.addVertex(T.id, i, "name", "v" + i, "age", i % 100, "score", i * 1.5d);
+ for (int i = 0; i < vertexCount; i++)
+ graph.vertices(i).next().addEdge("knows", graph.vertices((i + 1) % vertexCount).next(),
+ T.id, 1_000_000 + i, "weight", i * 0.5d, "count", i % 7);
+ graph.tx().commit();
+ graph.compact();
+ final long bytes = new File(location, GraphBinaryStorage.SNAPSHOT_FILE).length();
+ final double perElement = (double) bytes / (2 * vertexCount);
+ assertTrue("expected well under 168 bytes/element (whole-object baseline), got " + perElement, perElement < 100.0);
+ } finally {
+ graph.close();
+ }
+ }
+
private static String rootMessage(final Throwable t) {
Throwable cur = t;
while (cur.getCause() != null && cur.getCause() != cur)