You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by be...@apache.org on 2024/11/11 12:33:55 UTC
(cassandra) branch cep-15-accord updated (86ebf82c78 -> 0df9ed3943)
This is an automated email from the ASF dual-hosted git repository.
benedict pushed a change to branch cep-15-accord
in repository https://gitbox.apache.org/repos/asf/cassandra.git
discard 86ebf82c78 Improvements to queueing, execution and performance
new 0df9ed3943 Improvements to queueing, execution and performance
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (86ebf82c78)
\
N -- N -- N refs/heads/cep-15-accord (0df9ed3943)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../cassandra/simulator/test/AccordJournalSimulationTest.java | 6 +++++-
.../cassandra/db/compaction/CompactionAccordIteratorsTest.java | 2 ++
test/unit/org/apache/cassandra/service/accord/AccordTestUtils.java | 5 ++++-
3 files changed, 11 insertions(+), 2 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
(cassandra) 01/01: Improvements to queueing, execution and performance
Posted by be...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
benedict pushed a commit to branch cep-15-accord
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 0df9ed394383d698fece28f70e93b3e01ae60a4c
Author: Benedict Elliott Smith <be...@apache.org>
AuthorDate: Thu Oct 17 22:04:10 2024 +0100
Improvements to queueing, execution and performance
Work can now be cancelled, and commands serving remote requests now cancel queued work if they time out. This prevents run-away work growth, as work does not outlive its useful lifespan.
CommandStores, threads and caches are now only loosely coupled, with it possible to independently tune:
- the total number of threads
- the number of work queue/cache units we distribute the threads amongst
- the number of CommandStores per table/shard (which are distributed between queue/cache units, the threads of which will execute CommandStore work)
Mutual exclusivity is managed separately for the queue/cache unit and each CommandStore, and the locks are held only for as long as necessary so we can have multiple threads servicing the same CommandStore(s). Given this threading model, it is now possible for Accord threads to perform all of the loading/saving work, reducing the queueing delay and context switching costs - and this is also configurable. The default configuration is now to do this work on the Accord work pool (as alre [...]
Misc perf improvements:
- Write directly to Memtable for cache evictions, using putIfAbsent immediately
- Use LCS on CommandsForKey table for faster reads
- Flatten UUID fields into TableId to reduce indirection for comparisons
- Introduce asymmetric comparisons to BTreeMap for faster schema lookups
- Read TimestampsForKey directly to avoid parsing CQL
- Save summary information in RedundantBefore to short-circuit executions
- Send Stable message only as necessary on Execute, to reduce load on replicas
- Ensure journal entries are immediately visible to replay without handing over to another thread, so on normal path can avoid context switch and listener overheads
- Journal periodic mode should fsync only as necessary
- Use OpOrder to guard Journal Segment read access (avoiding taking individual references, which can be costly)
- AccordCache can 'shrink' (serialise) entries instead of evicting, to increase effective capacity (evicting any already-shrunk entries that are encountered)
- EphemeralRead cache entries are evicted only on timeout of the remote request, and are not otherwise persisted
- Work is scheduled by first arrival time, not by read completion time - work that is slow to read jumps the queue once data is in memory to serve it, to reduce latency variability
- Some operations may now partially execute without waiting for all state to be brought into memory
---
build.xml | 26 +
conf/cassandra.yaml | 5 +-
modules/accord | 2 +-
.../cassandra/concurrent/InfiniteLoopExecutor.java | 6 +-
.../org/apache/cassandra/concurrent/Stage.java | 3 +-
.../org/apache/cassandra/config/AccordSpec.java | 133 +-
src/java/org/apache/cassandra/config/Config.java | 1 -
.../cassandra/config/DatabaseDescriptor.java | 179 ++-
.../db/compaction/CompactionIterator.java | 13 +-
.../apache/cassandra/db/marshal/AbstractType.java | 35 +
.../apache/cassandra/db/marshal/CompositeType.java | 67 +-
.../cassandra/db/memtable/AbstractMemtable.java | 9 +
.../org/apache/cassandra/db/memtable/Memtable.java | 11 +-
.../db/memtable/ShardedSkipListMemtable.java | 12 +-
.../cassandra/db/memtable/SkipListMemtable.java | 9 +-
.../apache/cassandra/db/memtable/TrieMemtable.java | 2 +-
.../cassandra/db/virtual/AccordVirtualTables.java | 48 +-
.../org/apache/cassandra/dht/LocalPartitioner.java | 1 -
.../org/apache/cassandra/hints/HintsBuffer.java | 14 +-
.../cassandra/index/accord/RoutesSearcher.java | 14 +-
.../apache/cassandra/io/util/DataOutputBuffer.java | 2 +-
.../apache/cassandra/journal/ActiveSegment.java | 212 ++--
.../org/apache/cassandra/journal/Compactor.java | 4 +-
.../org/apache/cassandra/journal/Component.java | 8 +-
.../apache/cassandra/journal/EntrySerializer.java | 217 ++--
src/java/org/apache/cassandra/journal/Flusher.java | 224 ++--
.../apache/cassandra/journal/InMemoryIndex.java | 5 +-
src/java/org/apache/cassandra/journal/Index.java | 12 +-
src/java/org/apache/cassandra/journal/Journal.java | 158 +--
.../org/apache/cassandra/journal/KeySupport.java | 2 +
.../org/apache/cassandra/journal/Metadata.java | 21 +-
src/java/org/apache/cassandra/journal/Params.java | 17 +-
.../apache/cassandra/journal/RecordPointer.java | 12 +
src/java/org/apache/cassandra/journal/Segment.java | 53 +-
.../apache/cassandra/journal/SegmentWriter.java | 114 --
.../org/apache/cassandra/journal/Segments.java | 100 +-
.../apache/cassandra/journal/StaticSegment.java | 100 +-
.../apache/cassandra/journal/SyncedOffsets.java | 256 ----
...teCacheMetrics.java => AccordCacheMetrics.java} | 4 +-
.../apache/cassandra/metrics/AccordMetrics.java | 1 +
src/java/org/apache/cassandra/schema/TableId.java | 52 +-
.../org/apache/cassandra/service/StorageProxy.java | 4 +-
.../cassandra/service/accord/AccordCache.java | 1290 ++++++++++++++++++++
.../cassandra/service/accord/AccordCacheEntry.java | 664 ++++++++++
.../service/accord/AccordCachingState.java | 651 ----------
.../service/accord/AccordCommandStore.java | 499 +++-----
.../service/accord/AccordCommandStores.java | 139 ++-
.../cassandra/service/accord/AccordDataStore.java | 7 +-
.../cassandra/service/accord/AccordExecutor.java | 1025 ++++++++++++++++
.../accord/AccordExecutorAbstractLockLoop.java | 267 ++++
.../AccordExecutorAbstractSemiSyncSubmit.java | 64 +
.../service/accord/AccordExecutorAsyncSubmit.java | 101 ++
.../accord/AccordExecutorInfiniteLoops.java | 101 ++
.../accord/AccordExecutorSemiSyncSubmit.java | 115 ++
.../service/accord/AccordExecutorSimple.java | 160 +++
.../service/accord/AccordExecutorSyncSubmit.java | 121 ++
.../service/accord/AccordFetchCoordinator.java | 2 +-
.../cassandra/service/accord/AccordJournal.java | 23 +-
.../accord/AccordJournalValueSerializers.java | 4 +-
.../cassandra/service/accord/AccordKeyspace.java | 200 +--
.../service/accord/AccordMessageSink.java | 2 +-
.../service/accord/AccordObjectSizes.java | 1 +
.../service/accord/AccordSafeCommand.java | 24 +-
.../service/accord/AccordSafeCommandStore.java | 262 +++-
.../service/accord/AccordSafeCommandsForKey.java | 10 +-
.../accord/AccordSafeCommandsForRanges.java | 85 --
.../cassandra/service/accord/AccordSafeState.java | 17 +-
.../service/accord/AccordSafeTimestampsForKey.java | 10 +-
.../service/accord/AccordSegmentCompactor.java | 3 +-
.../cassandra/service/accord/AccordService.java | 46 +-
.../cassandra/service/accord/AccordStateCache.java | 787 ------------
.../cassandra/service/accord/AccordTask.java | 1114 +++++++++++++++++
.../service/accord/CommandsForRanges.java | 4 +-
.../service/accord/CommandsForRangesLoader.java | 418 +++----
.../cassandra/service/accord/IAccordService.java | 4 +
.../apache/cassandra/service/accord/IJournal.java | 2 +-
.../cassandra/service/accord/JournalKey.java | 32 +
.../cassandra/service/accord/SavedCommand.java | 73 +-
.../cassandra/service/accord/api/AccordAgent.java | 22 +-
.../service/accord/api/AccordRoutableKey.java | 3 +-
.../service/accord/api/AccordRoutingKey.java | 9 +-
.../service/accord/api/AccordScheduler.java | 2 +-
.../service/accord/async/AsyncLoader.java | 321 -----
.../service/accord/async/AsyncOperation.java | 423 -------
.../service/accord/events/CacheEvents.java | 2 +-
.../service/accord/interop/AccordInteropApply.java | 58 +-
.../serializers/CommandStoreSerializers.java | 11 +-
.../accord/serializers/ReadDataSerializers.java | 12 +-
.../service/accord/txn/AbstractKeySorted.java | 9 +-
.../service/accord/txn/AbstractSerialized.java | 12 +-
.../cassandra/service/accord/txn/TxnRead.java | 6 +
.../cassandra/service/accord/txn/TxnWrite.java | 10 +-
.../org/apache/cassandra/utils/ByteBufferUtil.java | 2 +-
src/java/org/apache/cassandra/utils/UUIDGen.java | 11 +-
.../cassandra/utils/btree/AbstractBTreeMap.java | 22 +-
.../apache/cassandra/utils/btree/BTreeBiMap.java | 30 +-
.../org/apache/cassandra/utils/btree/BTreeMap.java | 27 +-
.../utils/concurrent/ConcurrentLinkedStack.java | 68 ++
.../cassandra/utils/concurrent/IntrusiveStack.java | 18 +-
.../utils/concurrent/LockWithAsyncSignal.java | 258 ++++
.../org/apache/cassandra/utils/concurrent/Ref.java | 5 +
.../service/accord/AccordExecutorBurnTest.java | 91 ++
test/conf/cassandra.yaml | 3 +-
.../org/apache/cassandra/distributed/Cluster.java | 5 +
.../distributed/impl/AbstractCluster.java | 74 +-
.../cassandra/distributed/impl/Instance.java | 21 +-
.../cassandra/distributed/impl/InstanceConfig.java | 6 +-
.../cassandra/distributed/test/TestBaseImpl.java | 4 +-
.../test/accord/AccordBootstrapTest.java | 15 +-
.../test/accord/AccordDropTableBase.java | 4 +-
.../test/accord/AccordHostReplacementTest.java | 4 +-
.../test/accord/AccordIncrementalRepairTest.java | 6 +-
.../test/accord/AccordIntegrationTest.java | 14 +-
.../test/accord/AccordJournalIntegrationTest.java | 1 +
.../distributed/test/accord/AccordLoadTest.java | 341 +++---
.../distributed/test/accord/AccordMetricsTest.java | 39 +-
.../test/accord/AccordMigrationRaceTestBase.java | 3 +
.../test/accord/AccordProgressLogTest.java | 24 +-
.../distributed/test/accord/AccordTestBase.java | 6 +-
.../distributed/test/log/BootWithMetadataTest.java | 3 +-
.../fuzz/topology/AccordTopologyMixupTest.java | 3 +-
.../accord/AccordJournalCompactionTest.java | 3 +-
.../cassandra/harry/harry2/RectangleToSquares.java | 87 ++
.../cassandra/simulator/asm/InterceptClasses.java | 4 +-
.../cassandra/simulator/ClusterSimulation.java | 4 +-
.../test/AccordJournalSimulationTest.java | 39 +-
.../unit/org/apache/cassandra/ServerTestUtils.java | 4 +-
.../concurrent/ForwardingExecutorPlus.java | 2 +-
.../config/DatabaseDescriptorRefTest.java | 2 +
.../config/YamlConfigurationLoaderTest.java | 8 +-
test/unit/org/apache/cassandra/cql3/CQLTester.java | 4 +-
.../apache/cassandra/db/ColumnFamilyStoreTest.java | 2 +-
.../compaction/CompactionAccordIteratorsTest.java | 30 +-
.../db/virtual/AccordVirtualTablesTest.java | 6 +-
.../index/accord/AccordIndexStressTest.java | 4 +-
.../cassandra/index/accord/RouteIndexTest.java | 5 +-
.../org/apache/cassandra/journal/SegmentTest.java | 11 +-
.../cassandra/journal/SyncedOffsetsTest.java | 70 --
.../org/apache/cassandra/journal/TestParams.java | 14 +-
.../cassandra/journal/TimeUUIDKeySupport.java | 15 +
.../service/accord/AccordCacheEntryTest.java | 117 ++
...ordStateCacheTest.java => AccordCacheTest.java} | 287 ++---
.../service/accord/AccordCachingStateTest.java | 185 ---
.../service/accord/AccordCommandStoreTest.java | 8 +-
.../service/accord/AccordCommandTest.java | 6 +-
.../service/accord/AccordJournalOrderTest.java | 12 +-
.../service/accord/AccordKeyspaceTest.java | 9 +-
.../service/accord/AccordMessageSinkTest.java | 3 +-
...AsyncOperationTest.java => AccordTaskTest.java} | 226 ++--
.../cassandra/service/accord/AccordTestUtils.java | 79 +-
.../cassandra/service/accord/AsyncLoaderTest.java | 455 +++++++
.../service/accord/CommandsForRangesTest.java | 8 +-
.../cassandra/service/accord/EpochSyncTest.java | 6 +-
.../cassandra/service/accord/MockJournal.java | 4 +-
.../cassandra/service/accord/SavedCommandTest.java | 2 +-
.../accord/SimulatedAccordCommandStore.java | 183 +--
....java => SimulatedAccordCommandStore.java.orig} | 28 +-
...ationTest.java => SimulatedAccordTaskTest.java} | 105 +-
.../service/accord/SimulatedDepsTest.java | 4 +-
.../accord/SimulatedMultiKeyAndRangeTest.java | 2 +-
.../SimulatedRandomKeysWithRangeConflictTest.java | 3 +-
.../service/accord/async/AsyncLoaderTest.java | 455 -------
.../apache/cassandra/utils/AccordGenerators.java | 3 +-
163 files changed, 9201 insertions(+), 5806 deletions(-)
diff --git a/build.xml b/build.xml
index 5f9933fdc2..3909e2c4b8 100644
--- a/build.xml
+++ b/build.xml
@@ -1028,6 +1028,32 @@
</checksum>
</target>
+ <target name="bintar" depends="_artifacts-init,check"
+ description="Create Cassandra tarball and maven artifacts">
+ <tar compression="gzip" longfile="gnu"
+ destfile="${build.dir}/${final.name}-bin.tar.gz">
+
+ <!-- Everything but bin/ (default mode) -->
+ <tarfileset dir="${dist.dir}" prefix="${final.name}">
+ <include name="**"/>
+ <exclude name="bin/*" />
+ <exclude name="tools/bin/*"/>
+ </tarfileset>
+ <!-- Shell includes in bin/ (default mode) -->
+ <tarfileset dir="${dist.dir}" prefix="${final.name}">
+ <include name="bin/*.in.sh" />
+ <include name="tools/bin/*.in.sh" />
+ </tarfileset>
+ <!-- Executable scripts in bin/ -->
+ <tarfileset dir="${dist.dir}" prefix="${final.name}" mode="755">
+ <include name="bin/*"/>
+ <include name="tools/bin/*"/>
+ <exclude name="bin/*.in.sh" />
+ <exclude name="tools/bin/*.in.sh" />
+ </tarfileset>
+ </tar>
+ </target>
+
<!-- Wrapper of build-test without dependencies, so both that target and its dependencies are skipped if the property
no-build-test is true. This is meant to be used to run tests without actually building them, provided that they have
been built before. All test targets depend on this, so one can run them using the no-build-test property.
diff --git a/conf/cassandra.yaml b/conf/cassandra.yaml
index 5d925c998c..19f9c0817a 100644
--- a/conf/cassandra.yaml
+++ b/conf/cassandra.yaml
@@ -2320,7 +2320,10 @@ storage_compatibility_mode: NONE
# journal_directory:
#
# # The number of Accord shards on this node; -1 means use the number of cores
-# shard_count: -1
+# queue_shard_count: -1
+#
+# # The number of Accord shards on this node; -1 means use the number of cores
+# command_store_shard_count: -1
#
# # Recover delay: the time between a transaction being initiated and a remote replica being willing to interrupt it to complete it
# recover_delay: 1s
diff --git a/modules/accord b/modules/accord
index 4ec8d262a7..8bd9d69803 160000
--- a/modules/accord
+++ b/modules/accord
@@ -1 +1 @@
-Subproject commit 4ec8d262a750a76744b7f6991b711f85fa41a89a
+Subproject commit 8bd9d6980350fa68a1db676a7b10940cf0541fb5
diff --git a/src/java/org/apache/cassandra/concurrent/InfiniteLoopExecutor.java b/src/java/org/apache/cassandra/concurrent/InfiniteLoopExecutor.java
index b576551ac0..c9487e4164 100644
--- a/src/java/org/apache/cassandra/concurrent/InfiniteLoopExecutor.java
+++ b/src/java/org/apache/cassandra/concurrent/InfiniteLoopExecutor.java
@@ -108,7 +108,6 @@ public class InfiniteLoopExecutor implements Interruptible
};
}
-
private void loop()
{
boolean interrupted = false;
@@ -194,6 +193,11 @@ public class InfiniteLoopExecutor implements Interruptible
return isTerminated();
}
+ public long threadId()
+ {
+ return thread.getId();
+ }
+
@VisibleForTesting
public boolean isAlive()
{
diff --git a/src/java/org/apache/cassandra/concurrent/Stage.java b/src/java/org/apache/cassandra/concurrent/Stage.java
index dcd8b1dc42..425cab6e59 100644
--- a/src/java/org/apache/cassandra/concurrent/Stage.java
+++ b/src/java/org/apache/cassandra/concurrent/Stage.java
@@ -46,7 +46,7 @@ public enum Stage
MUTATION (true, "MutationStage", "request", DatabaseDescriptor::getConcurrentWriters, DatabaseDescriptor::setConcurrentWriters, Stage::multiThreadedLowSignalStage),
COUNTER_MUTATION (true, "CounterMutationStage", "request", DatabaseDescriptor::getConcurrentCounterWriters, DatabaseDescriptor::setConcurrentCounterWriters, Stage::multiThreadedLowSignalStage),
VIEW_MUTATION (true, "ViewMutationStage", "request", DatabaseDescriptor::getConcurrentViewWriters, DatabaseDescriptor::setConcurrentViewWriters, Stage::multiThreadedLowSignalStage),
- ACCORD_MIGRATION (false, "AccordMigrationStage", "request", DatabaseDescriptor::getConcurrentAccordOps, DatabaseDescriptor::setConcurrentAccordOps, Stage::multiThreadedLowSignalStage),
+ ACCORD_MIGRATION (false, "AccordMigrationStage", "request", DatabaseDescriptor::getAccordConcurrentOps, DatabaseDescriptor::setConcurrentAccordOps, Stage::multiThreadedLowSignalStage),
GOSSIP (true, "GossipStage", "internal", () -> 1, null, Stage::singleThreadedStage),
REQUEST_RESPONSE (false, "RequestResponseStage", "request", FBUtilities::getAvailableProcessors, null, Stage::multiThreadedLowSignalStage),
ANTI_ENTROPY (false, "AntiEntropyStage", "internal", () -> 1, null, Stage::singleThreadedStage),
@@ -58,7 +58,6 @@ public enum Stage
PAXOS_REPAIR (false, "PaxosRepairStage", "internal", FBUtilities::getAvailableProcessors, null, Stage::multiThreadedStage),
INTERNAL_METADATA (false, "InternalMetadataStage", "internal", FBUtilities::getAvailableProcessors, null, Stage::multiThreadedStage),
FETCH_LOG (false, "MetadataFetchLogStage", "internal", () -> 1, null, Stage::singleThreadedStage),
- ACCORD_RANGE_LOADER(false, "AccordRangeLoader", "internal", () -> 4, null, Stage::multiThreadedStage),
;
public final String jmxName;
private final Supplier<ExecutorPlus> executorSupplier;
diff --git a/src/java/org/apache/cassandra/config/AccordSpec.java b/src/java/org/apache/cassandra/config/AccordSpec.java
index 74504326ab..b445e4492e 100644
--- a/src/java/org/apache/cassandra/config/AccordSpec.java
+++ b/src/java/org/apache/cassandra/config/AccordSpec.java
@@ -21,11 +21,14 @@ package org.apache.cassandra.config;
import java.util.concurrent.TimeUnit;
import accord.primitives.TxnId;
+import accord.utils.Invariants;
import com.fasterxml.jackson.annotation.JsonIgnore;
import org.apache.cassandra.journal.Params;
import org.apache.cassandra.service.consensus.TransactionalMode;
import static accord.primitives.Routable.Domain.Range;
+import static org.apache.cassandra.config.AccordSpec.QueueShardModel.THREAD_POOL_PER_SHARD;
+import static org.apache.cassandra.config.AccordSpec.QueueSubmissionModel.SYNC;
public class AccordSpec
{
@@ -35,18 +38,99 @@ public class AccordSpec
public volatile boolean enable_journal_compaction = true;
- public volatile OptionaldPositiveInt shard_count = OptionaldPositiveInt.UNDEFINED;
+ public enum QueueShardModel
+ {
+ /**
+ * Same number of threads as queue shards, but the shard lock is held only while managing the queue,
+ * so that submitting threads may queue load/save work.
+ *
+ * The global READ and WRITE stages are used for IO.
+ */
+ THREAD_PER_SHARD,
+
+ /**
+ * Same number of threads as shards, and the shard lock is held for the duration of serving requests.
+ * The global READ and WRITE stages are used for IO.
+ */
+ THREAD_PER_SHARD_SYNC_QUEUE,
+
+ /**
+ * More threads than shards. Threads update transaction state as well as performing IO, minimising context switching.
+ * Fewer shards is generally better, until queue-contention is encountered.
+ */
+ THREAD_POOL_PER_SHARD,
+
+ /**
+ * More threads than shards. Threads update transaction state only, relying on READ and WRITE stages for IO.
+ * Fewer shards is generally better, until queue-contention is encountered.
+ */
+ THREAD_POOL_PER_SHARD_EXCLUDES_IO,
+ }
+
+ public enum QueueSubmissionModel
+ {
+ /**
+ * The queue workers and all submissions require ownership of the lock.
+ */
+ SYNC,
+
+ /**
+ * The queue workers and some submissions require ownership of the lock.
+ * That is, if the lock is available on submission we take it; if it is not we try to guarantee that
+ * another thread will witness the work submission promptly, but if we cannot we wait for the lock
+ * to ensure work is scheduled.
+ */
+ SEMI_SYNC,
+
+ /**
+ * The queue workers only require ownership of the lock, submissions happens fully asynchronously.
+ */
+ ASYNC,
+
+ /**
+ * The queue is backed by submission to a single-threaded plain executor.
+ * This implementation does not honur the sharding model option.
+ *
+ * Note: this isn't intended to be used by real clusters.
+ */
+ EXEC_ST
+ }
+
+ public QueueShardModel queue_shard_model = THREAD_POOL_PER_SHARD;
+ public QueueSubmissionModel queue_submission_model = SYNC;
+
+ /**
+ * The number of queue (and cache) shards.
+ */
+ public volatile OptionaldPositiveInt queue_shard_count = OptionaldPositiveInt.UNDEFINED;
+
+ /**
+ * The target number of command stores to create per topology shard.
+ * This determines the amount of execution parallelism possible for a given table/shard on the host.
+ * More shards means more parallelism, but more state.
+ *
+ * TODO (expected): make this a table property
+ * TODO (expected): adjust this by proportion of ring
+ */
+ public volatile OptionaldPositiveInt command_store_shard_count = OptionaldPositiveInt.UNDEFINED;
+
+ public volatile OptionaldPositiveInt max_queued_loads = OptionaldPositiveInt.UNDEFINED;
+ public volatile OptionaldPositiveInt max_queued_range_loads = OptionaldPositiveInt.UNDEFINED;
+
+ public DataStorageSpec.LongMebibytesBound cache_size = null;
+ public DataStorageSpec.LongMebibytesBound working_set_size = null;
+ public boolean shrink_cache_entries_before_eviction = true;
// TODO (expected): we should be able to support lower recover delays, at least for txns
public volatile DurationSpec.IntMillisecondsBound recover_delay = new DurationSpec.IntMillisecondsBound(5000);
- public volatile DurationSpec.IntMillisecondsBound range_sync_recover_delay = new DurationSpec.IntMillisecondsBound("5m");
+ public volatile DurationSpec.IntMillisecondsBound range_syncpoint_recover_delay = new DurationSpec.IntMillisecondsBound("5m");
public String slowPreAccept = "30ms <= p50*2 <= 100ms";
public String slowRead = "30ms <= p50*2 <= 100ms";
public long recoveryDelayFor(TxnId txnId, TimeUnit unit)
{
if (txnId.isSyncPoint() && txnId.is(Range))
- return range_sync_recover_delay.to(unit);
+ return range_syncpoint_recover_delay.to(unit);
return recover_delay.to(unit);
}
@@ -63,7 +147,7 @@ public class AccordSpec
public DurationSpec.IntMillisecondsBound barrier_max_backoff = new DurationSpec.IntMillisecondsBound("10m");
- public DurationSpec.IntMillisecondsBound range_barrier_timeout = new DurationSpec.IntMillisecondsBound("2m");
+ public DurationSpec.IntMillisecondsBound range_syncpoint_timeout = new DurationSpec.IntMillisecondsBound("2m");
public volatile DurationSpec.IntSecondsBound fast_path_update_delay = new DurationSpec.IntSecondsBound("60m");
@@ -92,7 +176,7 @@ public class AccordSpec
* default transactional mode for tables created by this node when no transactional mode has been specified in the DDL
*/
public TransactionalMode default_transactional_mode = TransactionalMode.off;
- public boolean ephemeralReadEnabled = false;
+ public boolean ephemeralReadEnabled = true;
public boolean state_cache_listener_jfr_enabled = true;
public final JournalSpec journal = new JournalSpec();
public final MinEpochRetrySpec minEpochSyncRetry = new MinEpochRetrySpec();
@@ -110,9 +194,22 @@ public class AccordSpec
public int segmentSize = 32 << 20;
public FailurePolicy failurePolicy = FailurePolicy.STOP;
public FlushMode flushMode = FlushMode.PERIODIC;
- public DurationSpec.IntMillisecondsBound flushPeriod; // pulls default from 'commitlog_sync_period'
- public DurationSpec.IntMillisecondsBound periodicFlushLagBlock = new DurationSpec.IntMillisecondsBound("1500ms");
+ public volatile DurationSpec flushPeriod; // pulls default from 'commitlog_sync_period'
+ public DurationSpec periodicFlushLagBlock = new DurationSpec.IntMillisecondsBound("1500ms");
public DurationSpec.IntMillisecondsBound compactionPeriod = new DurationSpec.IntMillisecondsBound("60000ms");
+ private volatile long flushCombinedBlockPeriod = Long.MIN_VALUE;
+
+ public void setFlushPeriod(DurationSpec newFlushPeriod)
+ {
+ flushPeriod = newFlushPeriod;
+ flushCombinedBlockPeriod = Long.MIN_VALUE;
+ }
+
+ public void setPeriodicFlushLagBlock(DurationSpec newPeriodicFlushLagBlock)
+ {
+ periodicFlushLagBlock = newPeriodicFlushLagBlock;
+ flushCombinedBlockPeriod = Long.MIN_VALUE;
+ }
@Override
public int segmentSize()
@@ -139,24 +236,32 @@ public class AccordSpec
}
@Override
- public int compactionPeriodMillis()
+ public long compactionPeriod(TimeUnit unit)
{
- return compactionPeriod.toMilliseconds();
+ return compactionPeriod.to(unit);
}
@JsonIgnore
@Override
- public int flushPeriodMillis()
+ public long flushPeriod(TimeUnit units)
{
- return flushPeriod == null ? DatabaseDescriptor.getCommitLogSyncPeriod()
- : flushPeriod.toMilliseconds();
+ return flushPeriod.to(units);
}
@JsonIgnore
@Override
- public int periodicFlushLagBlock()
+ public long periodicBlockPeriod(TimeUnit units)
{
- return periodicFlushLagBlock.toMilliseconds();
+ long nanos = flushCombinedBlockPeriod;
+ if (nanos >= 0)
+ return units.convert(nanos, TimeUnit.NANOSECONDS);
+
+ long flushPeriodNanos = flushPeriod(TimeUnit.NANOSECONDS);
+ Invariants.checkState(flushPeriodNanos > 0);
+ nanos = periodicFlushLagBlock.to(TimeUnit.NANOSECONDS) + flushPeriodNanos;
+ // it is possible for this to race and cache the wrong value after an update
+ flushCombinedBlockPeriod = nanos;
+ return nanos;
}
/**
diff --git a/src/java/org/apache/cassandra/config/Config.java b/src/java/org/apache/cassandra/config/Config.java
index a533ce9d82..2fa6338800 100644
--- a/src/java/org/apache/cassandra/config/Config.java
+++ b/src/java/org/apache/cassandra/config/Config.java
@@ -487,7 +487,6 @@ public class Config
public volatile int counter_cache_keys_to_save = Integer.MAX_VALUE;
public DataStorageSpec.LongMebibytesBound paxos_cache_size = null;
- public DataStorageSpec.LongMebibytesBound accord_cache_size = null;
public DataStorageSpec.LongMebibytesBound consensus_migration_cache_size = null;
diff --git a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
index 1d06cc8931..84a6a24dfd 100644
--- a/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
+++ b/src/java/org/apache/cassandra/config/DatabaseDescriptor.java
@@ -62,6 +62,7 @@ import com.google.common.collect.Sets;
import com.google.common.primitives.Ints;
import com.google.common.primitives.Longs;
import com.google.common.util.concurrent.RateLimiter;
+
import org.apache.cassandra.utils.Pair;
import org.apache.commons.lang3.ArrayUtils;
import org.apache.commons.lang3.StringUtils;
@@ -157,6 +158,7 @@ import static org.apache.cassandra.db.ConsistencyLevel.ONE;
import static org.apache.cassandra.db.ConsistencyLevel.QUORUM;
import static org.apache.cassandra.io.util.FileUtils.ONE_GIB;
import static org.apache.cassandra.io.util.FileUtils.ONE_MIB;
+import static org.apache.cassandra.journal.Params.FlushMode.PERIODIC;
import static org.apache.cassandra.utils.Clock.Global.logInitializationOutcome;
public class DatabaseDescriptor
@@ -216,6 +218,7 @@ public class DatabaseDescriptor
private static long keyCacheSizeInMiB;
private static long paxosCacheSizeInMiB;
private static long accordCacheSizeInMiB;
+ private static long accordWorkingSetSizeInMiB;
private static long consensusMigrationCacheSizeInMiB;
private static long counterCacheSizeInMiB;
private static long indexSummaryCapacityInMiB;
@@ -595,6 +598,16 @@ public class DatabaseDescriptor
logger.debug("Syncing log with a period of {}", conf.commitlog_sync_period.toString());
}
+ if (conf.accord.journal.flushPeriod == null)
+ {
+ conf.accord.journal.flushPeriod = conf.commitlog_sync_period;
+ if (conf.accord.journal.flushMode == PERIODIC && conf.commitlog_sync_period.toMilliseconds() == 0)
+ {
+ logger.warn("Accord journal is configured in periodic mode, while Cassandra commit log is configured in {} mode", conf.commitlog_sync);
+ conf.accord.journal.flushPeriod = conf.accord.journal.periodicFlushLagBlock;
+ }
+ }
+
/* evaluate the DiskAccessMode Config directive, which also affects indexAccessMode selection */
if (conf.disk_access_mode == DiskAccessMode.auto || conf.disk_access_mode == DiskAccessMode.mmap_index_only)
{
@@ -948,18 +961,32 @@ public class DatabaseDescriptor
try
{
- // if paxosCacheSizeInMiB option was set to "auto" then size of the cache should be "max(10% of Heap (in MB), 1MB)
- accordCacheSizeInMiB = (conf.accord_cache_size == null)
+ // if accordCacheSizeInMiB option was set to "auto" then size of the cache should be "max(10% of Heap (in MB), 1MB)
+ accordCacheSizeInMiB = (conf.accord.cache_size == null)
? Math.max(1, (int) ((Runtime.getRuntime().totalMemory() * 0.10) / 1024 / 1024))
- : conf.accord_cache_size.toMebibytes();
+ : conf.accord.cache_size.toMebibytes();
if (accordCacheSizeInMiB < 0)
throw new NumberFormatException(); // to escape duplicating error message
}
catch (NumberFormatException e)
{
- throw new ConfigurationException("paxos_cache_size option was set incorrectly to '"
- + conf.paxos_cache_size + "', supported values are <integer> >= 0.", false);
+ throw new ConfigurationException("accord.cache_size option was set incorrectly to '"
+ + conf.accord.cache_size + "', supported values are <integer> >= 0.", false);
+ }
+
+ try
+ {
+ // if accordWorkingSetSizeInMiB option was set to "auto" then size of the working set should be "max(5% of Heap (in MB), 1MB)
+ // if negative, there is no limit
+ accordWorkingSetSizeInMiB = (conf.accord.working_set_size == null)
+ ? Math.max(1, (int) ((Runtime.getRuntime().totalMemory() * 0.05) / 1024 / 1024))
+ : conf.accord.working_set_size.toMebibytes();
+ }
+ catch (NumberFormatException e)
+ {
+ throw new ConfigurationException("accord.working_set_size option was set incorrectly to '"
+ + conf.accord.working_set_size + "', supported values are <integer> >= 0.", false);
}
try
@@ -2586,7 +2613,7 @@ public class DatabaseDescriptor
conf.concurrent_materialized_view_writes = concurrent_materialized_view_writes;
}
- public static int getConcurrentAccordOps()
+ public static int getAccordConcurrentOps()
{
return conf.concurrent_accord_operations;
}
@@ -3474,41 +3501,6 @@ public class DatabaseDescriptor
return conf.paxos_topology_repair_strict_each_quorum;
}
- public static AccordSpec getAccord()
- {
- return conf == null ? null : conf.accord;
- }
-
- public static AccordSpec.TransactionalRangeMigration getTransactionalRangeMigration()
- {
- return conf.accord.range_migration;
- }
-
- public static void setTransactionalRangeMigration(AccordSpec.TransactionalRangeMigration val)
- {
- conf.accord.range_migration = Preconditions.checkNotNull(val);
- }
-
- public static int getAccordBarrierRetryAttempts()
- {
- return conf.accord.barrier_retry_attempts;
- }
-
- public static long getAccordBarrierRetryInitialBackoffMillis()
- {
- return conf.accord.barrier_retry_inital_backoff_millis.toMilliseconds();
- }
-
- public static long getAccordBarrierRetryMaxBackoffMillis()
- {
- return conf.accord.barrier_max_backoff.toMilliseconds();
- }
-
- public static long getAccordRangeBarrierTimeoutNanos()
- {
- return conf.accord.range_barrier_timeout.to(TimeUnit.NANOSECONDS);
- }
-
public static TransactionalMode defaultTransactionalMode()
{
return conf.accord.default_transactional_mode;
@@ -4062,6 +4054,11 @@ public class DatabaseDescriptor
return accordCacheSizeInMiB;
}
+ public static long getAccordWorkingSetSizeInMiB()
+ {
+ return accordWorkingSetSizeInMiB;
+ }
+
public static long getConsensusMigrationCacheSizeInMiB()
{
return consensusMigrationCacheSizeInMiB;
@@ -5085,6 +5082,42 @@ public class DatabaseDescriptor
}
}
+
+ public static AccordSpec getAccord()
+ {
+ return conf.accord;
+ }
+
+ public static AccordSpec.TransactionalRangeMigration getTransactionalRangeMigration()
+ {
+ return conf.accord.range_migration;
+ }
+
+ public static void setTransactionalRangeMigration(AccordSpec.TransactionalRangeMigration val)
+ {
+ conf.accord.range_migration = Preconditions.checkNotNull(val);
+ }
+
+ public static int getAccordBarrierRetryAttempts()
+ {
+ return conf.accord.barrier_retry_attempts;
+ }
+
+ public static long getAccordBarrierRetryInitialBackoffMillis()
+ {
+ return conf.accord.barrier_retry_inital_backoff_millis.toMilliseconds();
+ }
+
+ public static long getAccordBarrierRetryMaxBackoffMillis()
+ {
+ return conf.accord.barrier_max_backoff.toMilliseconds();
+ }
+
+ public static long getAccordRangeSyncPointTimeoutNanos()
+ {
+ return conf.accord.range_syncpoint_timeout.to(TimeUnit.NANOSECONDS);
+ }
+
public static boolean getAccordTransactionsEnabled()
{
return conf.accord.enabled;
@@ -5095,9 +5128,69 @@ public class DatabaseDescriptor
conf.accord.enabled = b;
}
- public static int getAccordShardCount()
+ public static AccordSpec.QueueShardModel getAccordQueueShardModel()
+ {
+ return conf.accord.queue_shard_model;
+ }
+
+ public static AccordSpec.QueueSubmissionModel getAccordQueueSubmissionModel()
+ {
+ return conf.accord.queue_submission_model;
+ }
+
+ public static int getAccordQueueShardCount()
+ {
+ switch (getAccordQueueShardModel())
+ {
+ default: throw new AssertionError("Unhandled queue_shard_model: " + conf.accord.queue_shard_model);
+ case THREAD_PER_SHARD:
+ case THREAD_PER_SHARD_SYNC_QUEUE:
+ return conf.accord.queue_shard_count.or(DatabaseDescriptor::getAvailableProcessors);
+ case THREAD_POOL_PER_SHARD:
+ case THREAD_POOL_PER_SHARD_EXCLUDES_IO:
+ int defaultMax = getAccordQueueSubmissionModel() == AccordSpec.QueueSubmissionModel.SYNC ? 8 : 4;
+ return conf.accord.queue_shard_count.or(Math.min(defaultMax, DatabaseDescriptor.getAvailableProcessors()));
+ }
+ }
+
+ public static int getAccordCommandStoreShardCount()
+ {
+ return conf.accord.command_store_shard_count.or(DatabaseDescriptor::getAvailableProcessors);
+ }
+
+ public static int getAccordMaxQueuedLoadCount()
+ {
+ return conf.accord.max_queued_loads.or(getAccordConcurrentOps());
+ }
+
+ public static int getAccordMaxQueuedRangeLoadCount()
+ {
+ return conf.accord.max_queued_range_loads.or(Math.max(4, getAccordConcurrentOps() / 4));
+ }
+
+ public static boolean getAccordCacheShrinkingOn()
+ {
+ return conf.accord.shrink_cache_entries_before_eviction;
+ }
+
+ public static long getAccordRecoverDelay(TimeUnit units)
+ {
+ return conf.accord.recover_delay.to(units);
+ }
+
+ public static void setAccordRecoverDelay(long time, TimeUnit units)
+ {
+ conf.accord.recover_delay = new IntMillisecondsBound(time, units);
+ }
+
+ public static long getAccordRangeSyncPointRecoverDelay(TimeUnit units)
+ {
+ return conf.accord.range_syncpoint_recover_delay.to(units);
+ }
+
+ public static void setAccordRangeSyncPointRecoverDelay(long time, TimeUnit units)
{
- return conf.accord.shard_count.or(DatabaseDescriptor::getAvailableProcessors);
+ conf.accord.range_syncpoint_recover_delay = new IntMillisecondsBound(time, units);
}
public static long getAccordFastPathUpdateDelayMillis()
diff --git a/src/java/org/apache/cassandra/db/compaction/CompactionIterator.java b/src/java/org/apache/cassandra/db/compaction/CompactionIterator.java
index d5c2c404cf..b6ddbca3e2 100644
--- a/src/java/org/apache/cassandra/db/compaction/CompactionIterator.java
+++ b/src/java/org/apache/cassandra/db/compaction/CompactionIterator.java
@@ -35,6 +35,7 @@ import com.google.common.collect.Ordering;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import accord.api.Agent;
import accord.local.Cleanup;
import accord.local.CommandStores;
import accord.local.CommandStores.RangesForEpoch;
@@ -103,6 +104,7 @@ import org.apache.cassandra.service.accord.AccordService;
import org.apache.cassandra.service.accord.IAccordService;
import org.apache.cassandra.service.accord.JournalKey;
import org.apache.cassandra.service.accord.SavedCommand;
+import org.apache.cassandra.service.accord.api.AccordAgent;
import org.apache.cassandra.service.accord.api.AccordRoutingKey.TokenKey;
import org.apache.cassandra.service.paxos.PaxosRepairHistory;
import org.apache.cassandra.service.paxos.uncommitted.PaxosRows;
@@ -805,6 +807,7 @@ public class CompactionIterator extends CompactionInfo.Holder implements Unfilte
class AccordCommandsPurger extends AbstractPurger
{
+ final Agent agent;
final Int2ObjectHashMap<RedundantBefore> redundantBefores;
final Int2ObjectHashMap<DurableBefore> durableBefores;
final Int2ObjectHashMap<RangesForEpoch> ranges;
@@ -814,7 +817,9 @@ public class CompactionIterator extends CompactionInfo.Holder implements Unfilte
AccordCommandsPurger(Supplier<IAccordService> accordService)
{
- IAccordService.CompactionInfo compactionInfo = accordService.get().getCompactionInfo();
+ IAccordService service = accordService.get();
+ IAccordService.CompactionInfo compactionInfo = service.getCompactionInfo();
+ this.agent = service.agent();
this.redundantBefores = compactionInfo.redundantBefores;
this.ranges = compactionInfo.ranges;
this.durableBefores = compactionInfo.durableBefores;
@@ -852,7 +857,7 @@ public class CompactionIterator extends CompactionInfo.Holder implements Unfilte
if (saveStatus.is(Status.Invalidated))
return saveStatusOnly(saveStatus, row, nowInSec);
- Cleanup cleanup = shouldCleanupPartial(txnId, saveStatus, durability, participants,
+ Cleanup cleanup = shouldCleanupPartial(agent, txnId, saveStatus, durability, participants,
redundantBefore, durableBefore);
switch (cleanup)
{
@@ -1019,6 +1024,7 @@ public class CompactionIterator extends CompactionInfo.Holder implements Unfilte
final ColumnMetadata recordColumn;
final ColumnMetadata versionColumn;
final AccordService service;
+ final AccordAgent agent;
JournalKey key = null;
Object builder = null;
@@ -1036,6 +1042,7 @@ public class CompactionIterator extends CompactionInfo.Holder implements Unfilte
userVersion = service.journalConfiguration().userVersion();
IAccordService.CompactionInfo compactionInfo = service.getCompactionInfo();
+ this.agent = service.agent();
this.redundantBefores = compactionInfo.redundantBefores;
this.ranges = compactionInfo.ranges;
this.durableBefores = compactionInfo.durableBefores;
@@ -1102,7 +1109,7 @@ public class CompactionIterator extends CompactionInfo.Holder implements Unfilte
RedundantBefore redundantBefore = redundantBefores.get(key.commandStoreId);
DurableBefore durableBefore = durableBefores.get(key.commandStoreId);
- Cleanup cleanup = commandBuilder.shouldCleanup(redundantBefore, durableBefore);
+ Cleanup cleanup = commandBuilder.shouldCleanup(agent, redundantBefore, durableBefore);
if (cleanup == ERASE)
return PartitionUpdate.fullPartitionDelete(metadata(), partition.partitionKey(), maxSeenTimestamp, nowInSec).unfilteredIterator();
diff --git a/src/java/org/apache/cassandra/db/marshal/AbstractType.java b/src/java/org/apache/cassandra/db/marshal/AbstractType.java
index c2743cc18e..a5d14ff58b 100644
--- a/src/java/org/apache/cassandra/db/marshal/AbstractType.java
+++ b/src/java/org/apache/cassandra/db/marshal/AbstractType.java
@@ -21,6 +21,7 @@ import java.io.IOException;
import java.lang.reflect.Method;
import java.nio.ByteBuffer;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.Comparator;
@@ -28,6 +29,7 @@ import java.util.List;
import java.util.Map;
import java.util.Objects;
+import io.netty.util.concurrent.FastThreadLocal;
import org.apache.cassandra.cql3.AssignmentTestable;
import org.apache.cassandra.cql3.CQL3Type;
import org.apache.cassandra.cql3.ColumnSpecification;
@@ -698,6 +700,39 @@ public abstract class AbstractType<T> implements Comparator<ByteBuffer>, Assignm
throw new UnsupportedOperationException(getClass().getSimpleName() + " does not implement asComparableBytes");
}
+ protected static final FastThreadLocal<byte[]> tmpFlattenBuffer = new FastThreadLocal<>();
+ public static byte[] flattenByteSource(ByteSource source)
+ {
+ byte[] tmpBytes = tmpFlattenBuffer.get();
+ byte[] bytes = tmpBytes;
+ if (bytes == null) bytes = new byte[16];
+ int c = 0;
+ while (true)
+ {
+ int b = source.next();
+ if (b == ByteSource.END_OF_STREAM)
+ break;
+
+ if (c == bytes.length)
+ bytes = Arrays.copyOf(bytes, c * 2);
+
+ bytes[c++] = (byte)b;
+ }
+
+ byte[] result = Arrays.copyOf(bytes, c);
+ if (bytes != tmpBytes) tmpFlattenBuffer.set(bytes);
+ return result;
+ }
+
+ public <V> byte[] asFlatComparableBytes(ValueAccessor<V> accessor, V value, ByteComparable.Version version)
+ {
+ ByteSource source = asComparableBytes(accessor, value, version);
+ if (source == null)
+ return null;
+
+ return flattenByteSource(source);
+ }
+
public final ByteSource asComparableBytes(ByteBuffer byteBuffer, ByteComparable.Version version)
{
return asComparableBytes(ByteBufferAccessor.instance, byteBuffer, version);
diff --git a/src/java/org/apache/cassandra/db/marshal/CompositeType.java b/src/java/org/apache/cassandra/db/marshal/CompositeType.java
index df7ee99070..e175045049 100644
--- a/src/java/org/apache/cassandra/db/marshal/CompositeType.java
+++ b/src/java/org/apache/cassandra/db/marshal/CompositeType.java
@@ -29,6 +29,7 @@ import com.google.common.annotations.VisibleForTesting;
import com.google.common.collect.ImmutableList;
import com.google.common.collect.Lists;
+import accord.utils.Invariants;
import org.apache.cassandra.exceptions.ConfigurationException;
import org.apache.cassandra.exceptions.SyntaxException;
import org.apache.cassandra.serializers.BytesSerializer;
@@ -39,8 +40,15 @@ import org.apache.cassandra.utils.bytecomparable.ByteComparable.Version;
import org.apache.cassandra.utils.bytecomparable.ByteSource;
import org.apache.cassandra.utils.bytecomparable.ByteSourceInverse;
+import static accord.utils.Invariants.Paranoia.CONSTANT;
+import static accord.utils.Invariants.Paranoia.LINEAR;
+import static accord.utils.Invariants.ParanoiaCostFactor.LOW;
import static com.google.common.collect.Iterables.any;
import static com.google.common.collect.Iterables.transform;
+import static org.apache.cassandra.utils.bytecomparable.ByteSource.END_OF_STREAM;
+import static org.apache.cassandra.utils.bytecomparable.ByteSource.NEXT_COMPONENT;
+import static org.apache.cassandra.utils.bytecomparable.ByteSource.NEXT_COMPONENT_NULL;
+import static org.apache.cassandra.utils.bytecomparable.ByteSource.TERMINATOR;
/*
* The encoding of a CompositeType column name should be:
@@ -250,7 +258,64 @@ public class CompositeType extends AbstractCompositeType
if (i * 2 + 1 < srcs.length)
srcs = Arrays.copyOfRange(srcs, 0, i * 2 + 1);
- return ByteSource.withTerminatorMaybeLegacy(version, ByteSource.END_OF_STREAM, srcs);
+ return ByteSource.withTerminatorMaybeLegacy(version, END_OF_STREAM, srcs);
+ }
+
+ @Override
+ public <V> byte[] asFlatComparableBytes(ValueAccessor<V> accessor, V data, Version version)
+ {
+ if (data == null || accessor.isEmpty(data))
+ return null;
+
+ byte[] tmpBytes = tmpFlattenBuffer.get();
+ byte[] bytes = tmpBytes;
+ if (bytes == null) bytes = new byte[16];
+
+ int c = 0;
+ int length = accessor.size(data);
+
+ // statics go first
+ boolean isStatic = readIsStaticInternal(data, accessor);
+ int offset = startingOffsetInternal(isStatic);
+ bytes[c++] = (byte) (isStatic ? NEXT_COMPONENT_NULL : NEXT_COMPONENT);
+ bytes[c++] = (byte) (NEXT_COMPONENT);
+
+ int i = 0;
+ byte lastEoc = 0;
+ while (offset < length)
+ {
+ // Only the end-of-component byte of the last component of this composite can be non-zero, so the
+ // component before can't have a non-zero end-of-component byte.
+ assert lastEoc == 0 : lastEoc;
+
+ int componentLength = accessor.getUnsignedShort(data, offset);
+ offset += 2;
+ ByteSource tmp = types.get(i).asComparableBytes(accessor, accessor.slice(data, offset, componentLength), version);
+ while (true)
+ {
+ int b = tmp.next();
+ if (b == END_OF_STREAM) break;
+
+ if (c == bytes.length) bytes = Arrays.copyOf(bytes, c * 2);
+ bytes[c++] = (byte)b;
+ }
+ offset += componentLength;
+ lastEoc = accessor.getByte(data, offset);
+ offset += 1;
+ if (c == bytes.length) bytes = Arrays.copyOf(bytes, c * 2);
+ bytes[c++] = (byte) NEXT_COMPONENT;
+ bytes[c++] = (byte) (lastEoc & 0xFF ^ 0x80); // end-of-component also takes part in comparison as signed byte
+ bytes[c++] = (byte) (offset < length ? NEXT_COMPONENT : version == Version.LEGACY ? END_OF_STREAM : TERMINATOR);
+ ++i;
+ }
+
+ byte[] result = Arrays.copyOf(bytes, c);
+ if (bytes != tmpBytes) tmpFlattenBuffer.set(bytes);
+ byte[] test = super.asFlatComparableBytes(accessor, data, version);
+ if (Invariants.isParanoid() && Invariants.testParanoia(LINEAR, CONSTANT, LOW)) Invariants.checkState(Arrays.equals(test, result));
+ V roundtrip = fromComparableBytes(accessor, ByteSource.peekable(ByteSource.of(result, version)), version);
+ Invariants.checkState(accessor.compare(data, roundtrip, accessor) == 0);
+ return result;
}
@Override
diff --git a/src/java/org/apache/cassandra/db/memtable/AbstractMemtable.java b/src/java/org/apache/cassandra/db/memtable/AbstractMemtable.java
index 2f2c2a2551..dd2254721b 100644
--- a/src/java/org/apache/cassandra/db/memtable/AbstractMemtable.java
+++ b/src/java/org/apache/cassandra/db/memtable/AbstractMemtable.java
@@ -40,6 +40,8 @@ import org.github.jamm.Unmetered;
public abstract class AbstractMemtable implements Memtable
{
+ private static final AtomicLong nextId = new AtomicLong();
+
private final AtomicReference<LifecycleTransaction> flushTransaction = new AtomicReference<>(null);
protected final AtomicLong currentOperations = new AtomicLong(0);
protected final ColumnsCollector columnsCollector;
@@ -48,6 +50,7 @@ public abstract class AbstractMemtable implements Memtable
protected AtomicLong minTimestamp = new AtomicLong(Long.MAX_VALUE);
// The smallest local deletion time for all partitions in this memtable
protected AtomicLong minLocalDeletionTime = new AtomicLong(Long.MAX_VALUE);
+ private final long id = nextId.incrementAndGet();
// Note: statsCollector has corresponding statistics to the two above, but starts with an epoch value which is not
// correct for their usage.
@@ -80,6 +83,12 @@ public abstract class AbstractMemtable implements Memtable
return currentOperations.get();
}
+ @Override
+ public long getMemtableId()
+ {
+ return id;
+ }
+
@Override
public long getMinTimestamp()
{
diff --git a/src/java/org/apache/cassandra/db/memtable/Memtable.java b/src/java/org/apache/cassandra/db/memtable/Memtable.java
index b34a617ef4..dc0b7b3a1d 100644
--- a/src/java/org/apache/cassandra/db/memtable/Memtable.java
+++ b/src/java/org/apache/cassandra/db/memtable/Memtable.java
@@ -179,6 +179,11 @@ public interface Memtable extends Comparable<Memtable>, UnfilteredSource
// Main write and read operations
+ default long put(PartitionUpdate update, UpdateTransaction indexer, OpOrder.Group opGroup)
+ {
+ return put(update, indexer, opGroup, false);
+ }
+
/**
* Put new data in the memtable. This operation may block until enough memory is available in the memory pool.
*
@@ -186,12 +191,14 @@ public interface Memtable extends Comparable<Memtable>, UnfilteredSource
* @param indexer receives information about the update's effect
* @param opGroup write operation group, used to permit the operation to complete if it is needed to complete a
* flush to free space.
+ * @param assumeMissing if true, the implementation MAY clone the key and attempt putIfAbsent without first
+ * looking for the keys' presence
*
* @return the smallest timestamp delta between corresponding rows from existing and update. A
* timestamp delta being computed as the difference between the cells and DeletionTimes from any existing partition
* and those in {@code update}. See CASSANDRA-7979.
*/
- long put(PartitionUpdate update, UpdateTransaction indexer, OpOrder.Group opGroup);
+ long put(PartitionUpdate update, UpdateTransaction indexer, OpOrder.Group opGroup, boolean assumeMissing);
// Read operations are provided by the UnfilteredSource interface.
@@ -363,6 +370,8 @@ public interface Memtable extends Comparable<Memtable>, UnfilteredSource
*/
boolean accepts(OpOrder.Group opGroup, CommitLogPosition commitLogPosition);
+ long getMemtableId();
+
/** Approximate commit log lower bound, <= getCommitLogLowerBound, used as a time stamp for ordering */
CommitLogPosition getApproximateCommitLogLowerBound();
diff --git a/src/java/org/apache/cassandra/db/memtable/ShardedSkipListMemtable.java b/src/java/org/apache/cassandra/db/memtable/ShardedSkipListMemtable.java
index eb4a44ebd2..9b9a531a2b 100644
--- a/src/java/org/apache/cassandra/db/memtable/ShardedSkipListMemtable.java
+++ b/src/java/org/apache/cassandra/db/memtable/ShardedSkipListMemtable.java
@@ -138,11 +138,11 @@ public class ShardedSkipListMemtable extends AbstractShardedMemtable
*
* commitLogSegmentPosition should only be null if this is a secondary index, in which case it is *expected* to be null
*/
- public long put(PartitionUpdate update, UpdateTransaction indexer, OpOrder.Group opGroup)
+ public long put(PartitionUpdate update, UpdateTransaction indexer, OpOrder.Group opGroup, boolean assumeMissing)
{
DecoratedKey key = update.partitionKey();
MemtableShard shard = shards[boundaries.getShardForKey(key)];
- return shard.put(key, update, indexer, opGroup);
+ return shard.put(key, update, indexer, opGroup, assumeMissing);
}
/**
@@ -366,10 +366,10 @@ public class ShardedSkipListMemtable extends AbstractShardedMemtable
this.metadata = metadata;
}
- public long put(DecoratedKey key, PartitionUpdate update, UpdateTransaction indexer, OpOrder.Group opGroup)
+ public long put(DecoratedKey key, PartitionUpdate update, UpdateTransaction indexer, OpOrder.Group opGroup, boolean assumeMissing)
{
Cloner cloner = allocator.cloner(opGroup);
- AtomicBTreePartition previous = partitions.get(key);
+ AtomicBTreePartition previous = assumeMissing ? null : partitions.get(key);
long initialSize = 0;
if (previous == null)
@@ -504,13 +504,13 @@ public class ShardedSkipListMemtable extends AbstractShardedMemtable
*
* commitLogSegmentPosition should only be null if this is a secondary index, in which case it is *expected* to be null
*/
- public long put(PartitionUpdate update, UpdateTransaction indexer, OpOrder.Group opGroup)
+ public long put(PartitionUpdate update, UpdateTransaction indexer, OpOrder.Group opGroup, boolean assumeMissing)
{
DecoratedKey key = update.partitionKey();
MemtableShard shard = shards[boundaries.getShardForKey(key)];
synchronized (shard)
{
- return shard.put(key, update, indexer, opGroup);
+ return shard.put(key, update, indexer, opGroup, assumeMissing);
}
}
diff --git a/src/java/org/apache/cassandra/db/memtable/SkipListMemtable.java b/src/java/org/apache/cassandra/db/memtable/SkipListMemtable.java
index 3f6fbcbd52..985dd310fd 100644
--- a/src/java/org/apache/cassandra/db/memtable/SkipListMemtable.java
+++ b/src/java/org/apache/cassandra/db/memtable/SkipListMemtable.java
@@ -114,15 +114,14 @@ public class SkipListMemtable extends AbstractAllocatorMemtable
* commitLogSegmentPosition should only be null if this is a secondary index, in which case it is *expected* to be null
*/
@Override
- public long put(PartitionUpdate update, UpdateTransaction indexer, OpOrder.Group opGroup)
+ public long put(PartitionUpdate update, UpdateTransaction indexer, OpOrder.Group opGroup, boolean assumeMissing)
{
- Cloner cloner = allocator.cloner(opGroup);
- AtomicBTreePartition previous = partitions.get(update.partitionKey());
-
long initialSize = 0;
+ Cloner cloner = allocator.cloner(opGroup);
+ AtomicBTreePartition previous = assumeMissing ? null : partitions.get(update.partitionKey());
if (previous == null)
{
- final DecoratedKey cloneKey = cloner.clone(update.partitionKey());
+ DecoratedKey cloneKey = cloner.clone(update.partitionKey());
AtomicBTreePartition empty = new AtomicBTreePartition(metadata, cloneKey, allocator);
// We'll add the columns later. This avoids wasting works if we get beaten in the putIfAbsent
previous = partitions.putIfAbsent(cloneKey, empty);
diff --git a/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java b/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java
index a8fc54b891..2a2813d0ad 100644
--- a/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java
+++ b/src/java/org/apache/cassandra/db/memtable/TrieMemtable.java
@@ -180,7 +180,7 @@ public class TrieMemtable extends AbstractShardedMemtable
* commitLogSegmentPosition should only be null if this is a secondary index, in which case it is *expected* to be null
*/
@Override
- public long put(PartitionUpdate update, UpdateTransaction indexer, OpOrder.Group opGroup)
+ public long put(PartitionUpdate update, UpdateTransaction indexer, OpOrder.Group opGroup, boolean assumeMissing)
{
try
{
diff --git a/src/java/org/apache/cassandra/db/virtual/AccordVirtualTables.java b/src/java/org/apache/cassandra/db/virtual/AccordVirtualTables.java
index ed1534674a..c2a3bc2de3 100644
--- a/src/java/org/apache/cassandra/db/virtual/AccordVirtualTables.java
+++ b/src/java/org/apache/cassandra/db/virtual/AccordVirtualTables.java
@@ -36,12 +36,9 @@ import com.google.common.collect.Sets;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import accord.local.CommandStores;
import accord.primitives.Status;
import accord.primitives.TxnId;
import accord.utils.Invariants;
-import accord.utils.async.AsyncChain;
-import accord.utils.async.AsyncChains;
import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.cql3.FieldIdentifier;
import org.apache.cassandra.cql3.statements.schema.CreateTableStatement;
@@ -57,10 +54,11 @@ import org.apache.cassandra.exceptions.InvalidRequestException;
import org.apache.cassandra.schema.Schema;
import org.apache.cassandra.schema.TableId;
import org.apache.cassandra.schema.TableMetadata;
-import org.apache.cassandra.service.accord.AccordCommandStore;
+import org.apache.cassandra.service.accord.AccordCache;
+import org.apache.cassandra.service.accord.AccordCommandStores;
+import org.apache.cassandra.service.accord.AccordExecutor;
import org.apache.cassandra.service.accord.AccordKeyspace;
import org.apache.cassandra.service.accord.AccordService;
-import org.apache.cassandra.service.accord.AccordStateCache;
import org.apache.cassandra.service.accord.CommandStoreTxnBlockedGraph;
import org.apache.cassandra.service.accord.api.AccordRoutingKey.TokenKey;
import org.apache.cassandra.service.consensus.migration.ConsensusMigrationState;
@@ -81,20 +79,20 @@ public class AccordVirtualTables
return Collections.emptyList();
return List.of(
- new CommandStoreCache(keyspace),
+ new ExecutorCache(keyspace),
new MigrationState(keyspace),
new CoordinationStatus(keyspace),
new TxnBlockedByTable(keyspace)
);
}
- public static final class CommandStoreCache extends AbstractVirtualTable
+ public static final class ExecutorCache extends AbstractVirtualTable
{
- private CommandStoreCache(String keyspace)
+ private ExecutorCache(String keyspace)
{
super(parse(keyspace,
- "Accord Command Store Cache Metrics",
- "CREATE TABLE accord_command_store_cache(\n" +
+ "Accord Executor Cache Metrics",
+ "CREATE TABLE accord_executor_cache(\n" +
" id int,\n" +
" scope text,\n" +
" queries bigint,\n" +
@@ -107,32 +105,22 @@ public class AccordVirtualTables
@Override
public DataSet data()
{
- CommandStores stores = ((AccordService) AccordService.instance()).node().commandStores();
-
- AsyncChain<List<Map<String, AccordStateCache.ImmutableStats>>> statsByStoreChain = stores.map(store -> {
- Map<String, AccordStateCache.ImmutableStats> snapshots = new HashMap<>(3);
- AccordCommandStore accordStore = (AccordCommandStore) store.commandStore();
- snapshots.put(AccordKeyspace.COMMANDS, accordStore.commandCache().statsSnapshot());
- snapshots.put(AccordKeyspace.COMMANDS_FOR_KEY, accordStore.commandsForKeyCache().statsSnapshot());
- snapshots.put(AccordKeyspace.TIMESTAMPS_FOR_KEY, accordStore.timestampsForKeyCache().statsSnapshot());
- return snapshots;
- });
-
- List<Map<String, AccordStateCache.ImmutableStats>> statsByStore = AsyncChains.getBlockingAndRethrow(statsByStoreChain);
+ AccordCommandStores stores = (AccordCommandStores) ((AccordService) AccordService.instance()).node().commandStores();
SimpleDataSet result = new SimpleDataSet(metadata());
-
- for (int storeID : stores.ids())
+ for (AccordExecutor executor : stores.executors())
{
- Map<String, AccordStateCache.ImmutableStats> storeStats = statsByStore.get(storeID);
- addRow(storeStats.get(AccordKeyspace.COMMANDS), result, storeID, AccordKeyspace.COMMANDS);
- addRow(storeStats.get(AccordKeyspace.COMMANDS_FOR_KEY), result, storeID, AccordKeyspace.COMMANDS_FOR_KEY);
- addRow(storeStats.get(AccordKeyspace.TIMESTAMPS_FOR_KEY), result, storeID, AccordKeyspace.TIMESTAMPS_FOR_KEY);
+ Map<String, AccordCache.ImmutableStats> snapshots = new HashMap<>(3);
+ try (AccordExecutor.ExclusiveGlobalCaches cache = executor.lockCaches())
+ {
+ addRow(cache.commands.statsSnapshot(), result, executor.executorId(), AccordKeyspace.COMMANDS);
+ addRow(cache.commandsForKey.statsSnapshot(), result, executor.executorId(), AccordKeyspace.COMMANDS_FOR_KEY);
+ addRow(cache.timestampsForKey.statsSnapshot(), result, executor.executorId(), AccordKeyspace.TIMESTAMPS_FOR_KEY);
+ }
}
-
return result;
}
- private static void addRow(AccordStateCache.ImmutableStats stats, SimpleDataSet result, int storeID, String scope)
+ private static void addRow(AccordCache.ImmutableStats stats, SimpleDataSet result, int storeID, String scope)
{
result.row(storeID, scope);
result.column("queries", stats.queries);
diff --git a/src/java/org/apache/cassandra/dht/LocalPartitioner.java b/src/java/org/apache/cassandra/dht/LocalPartitioner.java
index 2f060c6b67..4c45887dc4 100644
--- a/src/java/org/apache/cassandra/dht/LocalPartitioner.java
+++ b/src/java/org/apache/cassandra/dht/LocalPartitioner.java
@@ -186,7 +186,6 @@ public class LocalPartitioner implements IPartitioner
{
// todo (tcm); seems partitioner got mutated on alter type (for example) before tcm, now we create a new one - not sure its enough just making sure that its the same type of partitioner
assert o.getPartitioner().getClass().equals(getPartitioner().getClass()) : String.format("partitioners do not match; %s != %s", getPartitioner(), o.getPartitioner());
-// assert getPartitioner() == o.getPartitioner() : String.format("partitioners do not match; %s != %s", getPartitioner(), o.getPartitioner());
return comparator.compare(token, ((LocalToken) o).token);
}
diff --git a/src/java/org/apache/cassandra/hints/HintsBuffer.java b/src/java/org/apache/cassandra/hints/HintsBuffer.java
index 646dd72feb..41c9cdf234 100644
--- a/src/java/org/apache/cassandra/hints/HintsBuffer.java
+++ b/src/java/org/apache/cassandra/hints/HintsBuffer.java
@@ -179,7 +179,19 @@ final class HintsBuffer
return new Allocation(offset, totalSize, opGroup);
}
- // allocate bytes in the slab, or return negative if not enough space
+ /**
+ * Allocate bytes in the segment, or return -1 if not enough space. Method ensures that marker bytes
+ * for each allocation (i.e. offset of its end) is written as a 32 bit integer at its beginning, and
+ * that these marker bytes are always written sequentially. In other words, if allocation A has a lower
+ * starting offset than allocation B, A's marker will always be written before the offset for B is returned.
+ *
+ * `allocateOffset` consists of two integers:
+ * 64 32 0
+ * | (i32) inProgress | (i32) writtenTo |
+ *
+ * If inProgress bytes are not zeroes, they contain an unwritten offset. Before allocating any bytes,
+ * inProgresss bytes need to be written at the writtenTo location in the target buffer.
+ */
private int allocateBytes(int totalSize)
{
long prev = position.getAndAdd(totalSize);
diff --git a/src/java/org/apache/cassandra/index/accord/RoutesSearcher.java b/src/java/org/apache/cassandra/index/accord/RoutesSearcher.java
index 268739290a..3c059f678f 100644
--- a/src/java/org/apache/cassandra/index/accord/RoutesSearcher.java
+++ b/src/java/org/apache/cassandra/index/accord/RoutesSearcher.java
@@ -18,12 +18,10 @@
package org.apache.cassandra.index.accord;
-import java.util.Collections;
-import java.util.Set;
+import java.util.function.Consumer;
import accord.primitives.Timestamp;
import accord.primitives.TxnId;
-import org.agrona.collections.ObjectHashSet;
import org.apache.cassandra.cql3.Operator;
import org.apache.cassandra.db.ColumnFamilyStore;
import org.apache.cassandra.db.DataRange;
@@ -99,14 +97,13 @@ public class RoutesSearcher
}
}
- public Set<TxnId> intersects(int store, TokenRange range, TxnId minTxnId, Timestamp maxTxnId)
+ public void intersects(int store, TokenRange range, TxnId minTxnId, Timestamp maxTxnId, Consumer<TxnId> forEach)
{
- return intersects(store, range.start(), range.end(), minTxnId, maxTxnId);
+ intersects(store, range.start(), range.end(), minTxnId, maxTxnId, forEach);
}
- public Set<TxnId> intersects(int store, AccordRoutingKey start, AccordRoutingKey end, TxnId minTxnId, Timestamp maxTxnId)
+ void intersects(int store, AccordRoutingKey start, AccordRoutingKey end, TxnId minTxnId, Timestamp maxTxnId, Consumer<TxnId> forEach)
{
- var set = new ObjectHashSet<TxnId>();
try (var it = searchKeysAccord(store, start, end))
{
while (it.hasNext())
@@ -114,10 +111,9 @@ public class RoutesSearcher
Entry next = it.next();
if (next.store_id != store) continue; // the index should filter out, but just in case...
if (next.txnId.compareTo(minTxnId) >= 0 && next.txnId.compareTo(maxTxnId) < 0)
- set.add(next.txnId);
+ forEach.accept(next.txnId);
}
}
- return set.isEmpty() ? Collections.emptySet() : set;
}
private static final class Entry
diff --git a/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java b/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
index 3e157b6b9a..228aa1755a 100644
--- a/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
+++ b/src/java/org/apache/cassandra/io/util/DataOutputBuffer.java
@@ -60,7 +60,7 @@ public class DataOutputBuffer extends BufferedDataOutputStreamPlus
* Scratch buffers used mostly for serializing in memory. It's important to call #close() when finished
* to keep the memory overhead from being too large in the system.
*/
- public static final FastThreadLocal<DataOutputBuffer> scratchBuffer = new FastThreadLocal<DataOutputBuffer>()
+ public static final FastThreadLocal<DataOutputBuffer> scratchBuffer = new FastThreadLocal<>()
{
@Override
protected DataOutputBuffer initialValue()
diff --git a/src/java/org/apache/cassandra/journal/ActiveSegment.java b/src/java/org/apache/cassandra/journal/ActiveSegment.java
index 69f2f3323c..f9ff3fd937 100644
--- a/src/java/org/apache/cassandra/journal/ActiveSegment.java
+++ b/src/java/org/apache/cassandra/journal/ActiveSegment.java
@@ -24,6 +24,7 @@ import java.nio.channels.FileChannel;
import java.nio.file.StandardOpenOption;
import java.util.*;
import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
+import java.util.concurrent.atomic.AtomicLongFieldUpdater;
import java.util.concurrent.locks.LockSupport;
import com.codahale.metrics.Timer;
@@ -44,16 +45,18 @@ final class ActiveSegment<K, V> extends Segment<K, V>
private final OpOrder appendOrder = new OpOrder();
// position in the buffer we are allocating from
- private volatile int allocateOffset = 0;
- private static final AtomicIntegerFieldUpdater<ActiveSegment> allocateOffsetUpdater = AtomicIntegerFieldUpdater.newUpdater(ActiveSegment.class, "allocateOffset");
+ private volatile long allocateOffset = 0;
+ private static final AtomicLongFieldUpdater<ActiveSegment> allocateOffsetUpdater = AtomicLongFieldUpdater.newUpdater(ActiveSegment.class, "allocateOffset");
/*
* Everything before this offset has been written and flushed.
*/
- private volatile int lastFlushedOffset = 0;
- private volatile int lastFsyncOffset = 0;
+ private volatile int writtenTo = 0;
+ private volatile int fsyncedTo = 0;
@SuppressWarnings("rawtypes")
- private static final AtomicIntegerFieldUpdater<ActiveSegment> lastFsyncOffsetUpdater = AtomicIntegerFieldUpdater.newUpdater(ActiveSegment.class, "lastFsyncOffset");
+ private static final AtomicIntegerFieldUpdater<ActiveSegment> writtenToUpdater = AtomicIntegerFieldUpdater.newUpdater(ActiveSegment.class, "writtenTo");
+ @SuppressWarnings("rawtypes")
+ private static final AtomicIntegerFieldUpdater<ActiveSegment> fsyncedToUpdater = AtomicIntegerFieldUpdater.newUpdater(ActiveSegment.class, "fsyncedTo");
/*
* End position of the buffer; initially set to its capacity and
@@ -70,16 +73,16 @@ final class ActiveSegment<K, V> extends Segment<K, V>
final InMemoryIndex<K> index;
private ActiveSegment(
- Descriptor descriptor, Params params, SyncedOffsets syncedOffsets, InMemoryIndex<K> index, Metadata metadata, KeySupport<K> keySupport)
+ Descriptor descriptor, Params params, InMemoryIndex<K> index, Metadata metadata, KeySupport<K> keySupport)
{
- super(descriptor, syncedOffsets, metadata, keySupport);
+ super(descriptor, metadata, keySupport);
this.index = index;
try
{
channel = FileChannel.open(file.toPath(), StandardOpenOption.WRITE, StandardOpenOption.READ, StandardOpenOption.CREATE);
buffer = channel.map(FileChannel.MapMode.READ_WRITE, 0, params.segmentSize());
endOfBuffer = buffer.capacity();
- selfRef = new Ref<>(this, new Tidier(descriptor, channel, buffer, syncedOffsets));
+ selfRef = new Ref<>(this, new Tidier(descriptor, channel, buffer));
}
catch (IOException e)
{
@@ -87,13 +90,11 @@ final class ActiveSegment<K, V> extends Segment<K, V>
}
}
- @SuppressWarnings("resource")
static <K, V> ActiveSegment<K, V> create(Descriptor descriptor, Params params, KeySupport<K> keySupport)
{
- SyncedOffsets syncedOffsets = SyncedOffsets.active(descriptor);
InMemoryIndex<K> index = InMemoryIndex.create(keySupport);
Metadata metadata = Metadata.create();
- return new ActiveSegment<>(descriptor, params, syncedOffsets, index, metadata, keySupport);
+ return new ActiveSegment<>(descriptor, params, index, metadata, keySupport);
}
@Override
@@ -147,40 +148,40 @@ final class ActiveSegment<K, V> extends Segment<K, V>
/**
* Stop writing to this file, flush and close it. Does nothing if the file is already closed.
*/
- @Override
- public synchronized void close()
+ public synchronized void close(Journal<K, V> journal)
{
- close(true);
+ close(journal, true);
}
/**
* @return true if the closed segment was definitely empty, false otherwise
*/
- private synchronized boolean close(boolean persistComponents)
+ private synchronized boolean close(Journal<K, V> journal, boolean persistComponents)
{
boolean isEmpty = discardUnusedTail();
if (!isEmpty)
{
- flush(true);
+ updateWrittenTo();
+ fsync();
if (persistComponents) persistComponents();
}
- release();
+ release(journal);
return isEmpty;
}
/**
* Close and discard a pre-allocated, available segment, that's never been exposed
*/
- void closeAndDiscard()
+ void closeAndDiscard(Journal<K, V> journal)
{
- boolean isEmpty = close(false);
+ boolean isEmpty = close(journal, false);
if (!isEmpty) throw new IllegalStateException();
discard();
}
- void closeAndIfEmptyDiscard()
+ void closeAndIfEmptyDiscard(Journal<K, V> journal)
{
- boolean isEmpty = close(true);
+ boolean isEmpty = close(journal, true);
if (isEmpty) discard();
}
@@ -188,7 +189,6 @@ final class ActiveSegment<K, V> extends Segment<K, V>
{
index.persist(descriptor);
metadata.persist(descriptor);
- syncedOffsets.fsync();
SyncUtil.trySyncDir(descriptor.directory);
}
@@ -199,13 +199,6 @@ final class ActiveSegment<K, V> extends Segment<K, V>
descriptor.fileFor(Component.DATA).deleteIfExists();
descriptor.fileFor(Component.INDEX).deleteIfExists();
descriptor.fileFor(Component.METADATA).deleteIfExists();
- descriptor.fileFor(Component.SYNCED_OFFSETS).deleteIfExists();
- }
-
- @Override
- void release()
- {
- selfRef.release();
}
@Override
@@ -220,23 +213,27 @@ final class ActiveSegment<K, V> extends Segment<K, V>
return selfRef.ref();
}
- private static final class Tidier implements Tidy
+ @Override
+ public Ref<Segment<K, V>> selfRef()
+ {
+ return selfRef;
+ }
+
+ private static final class Tidier extends Segment.Tidier implements Tidy
{
private final Descriptor descriptor;
private final FileChannel channel;
private final ByteBuffer buffer;
- private final SyncedOffsets syncedOffsets;
- Tidier(Descriptor descriptor, FileChannel channel, ByteBuffer buffer, SyncedOffsets syncedOffsets)
+ Tidier(Descriptor descriptor, FileChannel channel, ByteBuffer buffer)
{
this.descriptor = descriptor;
this.channel = channel;
this.buffer = buffer;
- this.syncedOffsets = syncedOffsets;
}
@Override
- public void tidy()
+ void onUnreferenced()
{
FileUtils.clean(buffer);
try
@@ -247,7 +244,6 @@ final class ActiveSegment<K, V> extends Segment<K, V>
{
throw new JournalWriteError(descriptor, Component.DATA, e);
}
- syncedOffsets.close();
}
@Override
@@ -257,68 +253,49 @@ final class ActiveSegment<K, V> extends Segment<K, V>
}
}
- /*
- * Flush logic; closing and component flushing
- */
-
- boolean shouldFlush()
+ public boolean isFlushed(long position)
{
- int allocateOffset = this.allocateOffset;
- return lastFlushedOffset < allocateOffset;
+ return writtenTo >= position;
}
- public boolean isFlushed(long position)
+ public int writtenToAtLeast()
{
- return lastFlushedOffset >= position;
+ return writtenTo;
}
- public long lastFlushedOffset()
+ public int fsyncedTo()
{
- return lastFlushedOffset;
+ return fsyncedTo;
}
- /**
- * Possibly force a disk flush for this segment file.
- * TODO FIXME: calls from outside Flusher + callbacks
- * @return last synced offset
- */
- synchronized int flush(boolean fsync)
+ public int updateWrittenTo()
{
- int allocateOffset = this.allocateOffset;
- if (lastFlushedOffset >= allocateOffset)
- return lastFlushedOffset;
+ int allocatedTo = (int)allocateOffset;
+ if (writtenTo >= allocatedTo)
+ return writtenTo;
waitForModifications();
- if (fsync)
- {
- fsyncInternal();
- lastFsyncOffsetUpdater.accumulateAndGet(this, allocateOffset, Math::max);
- }
- lastFlushedOffset = allocateOffset;
- int syncedOffset = Math.min(allocateOffset, endOfBuffer);
- syncedOffsets.mark(syncedOffset, fsync);
- flushComplete.signalAll();
- return syncedOffset;
+ return writtenToUpdater.accumulateAndGet(this, allocatedTo, Math::max);
}
// provides no ordering guarantees
void fsync()
{
- int lastFlushed = lastFlushedOffset;
- if (lastFsyncOffset >= lastFlushed)
+ int writtenTo = this.writtenTo;
+ if (fsyncedTo >= writtenTo)
return;
fsyncInternal();
- syncedOffsets.fsync();
- lastFsyncOffsetUpdater.accumulateAndGet(this, lastFlushed, Math::max);
+ fsyncedToUpdater.accumulateAndGet(this, writtenTo, Math::max);
+ flushComplete.signalAll();
}
private void waitForFlush(int position)
{
- while (lastFlushedOffset < position)
+ while (fsyncedTo < position)
{
WaitQueue.Signal signal = flushComplete.register();
- if (lastFlushedOffset < position)
+ if (fsyncedTo < position)
signal.awaitThrowUncheckedOnInterrupt();
else
signal.cancel();
@@ -346,12 +323,6 @@ final class ActiveSegment<K, V> extends Segment<K, V>
}
}
- boolean isFullyFlushed()
- {
- int allocateOffset = this.allocateOffset;
- return lastFsyncOffset >= allocateOffset;
- }
-
/**
* Ensures no more of this segment is writeable, by allocating any unused section at the end
* and marking it discarded void discartUnusedTail()
@@ -364,10 +335,10 @@ final class ActiveSegment<K, V> extends Segment<K, V>
{
while (true)
{
- int prev = allocateOffset;
+ long prev = completeInProgress();
int next = endOfBuffer + 1;
- if (prev >= next)
+ if ((int)prev >= next)
{
// already stopped allocating, might also be closed
assert buffer == null || prev == buffer.capacity() + 1;
@@ -377,10 +348,11 @@ final class ActiveSegment<K, V> extends Segment<K, V>
if (allocateOffsetUpdater.compareAndSet(this, prev, next))
{
// stopped allocating now; can only succeed once, no further allocation or discardUnusedTail can succeed
- endOfBuffer = prev;
+ endOfBuffer = (int)prev;
assert buffer != null && next == buffer.capacity() + 1;
return prev == 0;
}
+ LockSupport.parkNanos(1);
}
}
}
@@ -414,7 +386,8 @@ final class ActiveSegment<K, V> extends Segment<K, V>
private int totalEntrySize(Set<Integer> hosts, int recordSize)
{
return EntrySerializer.fixedEntrySize(keySupport, descriptor.userVersion)
- + EntrySerializer.variableEntrySize(hosts.size(), recordSize);
+ + EntrySerializer.variableEntrySize(hosts.size())
+ + recordSize;
}
// allocate bytes in the segment, or return -1 if not enough space
@@ -422,16 +395,30 @@ final class ActiveSegment<K, V> extends Segment<K, V>
{
while (true)
{
- int prev = allocateOffset;
- int next = prev + size;
+ long prev = maybeCompleteInProgress();
+ if (prev < 0)
+ {
+ LockSupport.parkNanos(1); // ConstantBackoffCAS Algorithm from https://arxiv.org/pdf/1305.5800.pdf
+ continue;
+ }
+
+ long next = prev + size;
if (next >= endOfBuffer)
return -1;
- if (allocateOffsetUpdater.compareAndSet(this, prev, next))
+
+ // TODO (expected): if we write a "safe shutdown" marker we don't need this,
+ // but this provides safe restart in the event the process terminates abruptly but the host remains stable
+ long inProgress = prev | (next << 32);
+ if (!allocateOffsetUpdater.compareAndSet(this, prev, inProgress))
{
- assert buffer != null;
- return prev;
+ LockSupport.parkNanos(1); // ConstantBackoffCAS Algorithm from https://arxiv.org/pdf/1305.5800.pdf
+ continue;
}
- LockSupport.parkNanos(1); // ConstantBackoffCAS Algorithm from https://arxiv.org/pdf/1305.5800.pdf
+
+ assert buffer != null;
+ buffer.putInt((int)prev, (int)next);
+ allocateOffsetUpdater.compareAndSet(this, inProgress, next);
+ return (int) prev;
}
}
@@ -450,14 +437,13 @@ final class ActiveSegment<K, V> extends Segment<K, V>
this.length = length;
}
- RecordPointer write(K id, ByteBuffer record, Set<Integer> hosts)
+ void write(K id, ByteBuffer record, Set<Integer> hosts)
{
- try (BufferedDataOutputStreamPlus out = new DataOutputBufferFixed(buffer))
+ try
{
- EntrySerializer.write(id, record, hosts, keySupport, out, descriptor.userVersion);
- index.update(id, start, length);
+ EntrySerializer.write(id, record, hosts, keySupport, buffer, descriptor.userVersion);
metadata.update(hosts);
- return new RecordPointer(descriptor.timestamp, start);
+ index.update(id, start, length);
}
catch (IOException e)
{
@@ -472,9 +458,9 @@ final class ActiveSegment<K, V> extends Segment<K, V>
// Variant of write that does not allocate/return a record pointer
void writeInternal(K id, ByteBuffer record, Set<Integer> hosts)
{
- try (BufferedDataOutputStreamPlus out = new DataOutputBufferFixed(buffer))
+ try
{
- EntrySerializer.write(id, record, hosts, keySupport, out, descriptor.userVersion);
+ EntrySerializer.write(id, record, hosts, keySupport, buffer, descriptor.userVersion);
index.update(id, start, length);
metadata.update(hosts);
}
@@ -488,12 +474,48 @@ final class ActiveSegment<K, V> extends Segment<K, V>
}
}
- void awaitFlush(Timer waitingOnFlush)
+ void awaitDurable(Timer waitingOnFlush)
{
try (Timer.Context ignored = waitingOnFlush.time())
{
waitForFlush(start);
}
}
+
+ boolean isFsynced()
+ {
+ return fsyncedTo >= start + length;
+ }
+
+ Descriptor descriptor()
+ {
+ return descriptor;
+ }
+
+ int start()
+ {
+ return start;
+ }
+ }
+
+ private int maybeCompleteInProgress()
+ {
+ long cur = allocateOffset;
+ int inProgress = (int) (cur >>> 32);
+ if (inProgress == 0) return (int) cur;
+ // finish up the in-progress allocation
+ buffer.putInt((int)cur, inProgress);
+ if (!allocateOffsetUpdater.compareAndSet(this, cur, inProgress))
+ return -1;
+
+ return inProgress;
+ }
+
+ private int completeInProgress()
+ {
+ int result = maybeCompleteInProgress();
+ while (result < 0)
+ result = maybeCompleteInProgress();
+ return result;
}
}
diff --git a/src/java/org/apache/cassandra/journal/Compactor.java b/src/java/org/apache/cassandra/journal/Compactor.java
index 4ecfb74091..0062b7471d 100644
--- a/src/java/org/apache/cassandra/journal/Compactor.java
+++ b/src/java/org/apache/cassandra/journal/Compactor.java
@@ -46,7 +46,7 @@ public final class Compactor<K, V> implements Runnable, Shutdownable
synchronized void start()
{
if (journal.params.enableCompaction())
- schedule(journal.params.compactionPeriodMillis(), TimeUnit.MILLISECONDS);
+ schedule(journal.params.compactionPeriod(TimeUnit.MILLISECONDS), TimeUnit.MILLISECONDS);
}
private synchronized void schedule(long period, TimeUnit units)
@@ -85,7 +85,7 @@ public final class Compactor<K, V> implements Runnable, Shutdownable
journal.replaceCompactedSegments(toCompact, newSegments);
for (StaticSegment<K, V> segment : toCompact)
- segment.discard();
+ segment.discard(journal);
}
catch (IOException e)
{
diff --git a/src/java/org/apache/cassandra/journal/Component.java b/src/java/org/apache/cassandra/journal/Component.java
index c9de451adb..f7cf944f86 100644
--- a/src/java/org/apache/cassandra/journal/Component.java
+++ b/src/java/org/apache/cassandra/journal/Component.java
@@ -17,15 +17,19 @@
*/
package org.apache.cassandra.journal;
+import java.util.List;
+
+import static accord.utils.SortedArrays.SortedArrayList.ofSorted;
+
enum Component
{
DATA ("data"),
INDEX ("indx"),
- METADATA ("meta"),
- SYNCED_OFFSETS ("sync");
+ METADATA ("meta");
//OFFSET_MAP (".offs"),
//INVLALIDATIONS (".invl");
+ public static final List<Component> VALUES = ofSorted(values());
final String extension;
Component(String extension)
diff --git a/src/java/org/apache/cassandra/journal/EntrySerializer.java b/src/java/org/apache/cassandra/journal/EntrySerializer.java
index 2a707e7d73..a90cfa38f0 100644
--- a/src/java/org/apache/cassandra/journal/EntrySerializer.java
+++ b/src/java/org/apache/cassandra/journal/EntrySerializer.java
@@ -23,17 +23,13 @@ import java.nio.ByteBuffer;
import java.util.Set;
import java.util.zip.CRC32;
+import accord.utils.Invariants;
import org.agrona.collections.IntHashSet;
import org.apache.cassandra.db.TypeSizes;
-import org.apache.cassandra.io.util.DataInputBuffer;
-import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.utils.Crc;
import static org.apache.cassandra.journal.Journal.validateCRC;
-import static org.apache.cassandra.utils.FBUtilities.updateChecksum;
-import static org.apache.cassandra.utils.FBUtilities.updateChecksumInt;
-import static org.apache.cassandra.utils.FBUtilities.updateChecksumShort;
public final class EntrySerializer
{
@@ -41,175 +37,172 @@ public final class EntrySerializer
ByteBuffer record,
Set<Integer> hosts,
KeySupport<K> keySupport,
- DataOutputPlus out,
+ ByteBuffer out,
int userVersion)
throws IOException
{
- CRC32 crc = Crc.crc32();
+ int start = out.position();
+ int totalSize = out.getInt() - start;
+ Invariants.checkState(totalSize == out.remaining() + TypeSizes.INT_SIZE);
+ Invariants.checkState(totalSize == record.remaining() + fixedEntrySize(keySupport, userVersion) + variableEntrySize(hosts.size()));
keySupport.serialize(key, out, userVersion);
- keySupport.updateChecksum(crc, key, userVersion);
-
- out.writeShort(hosts.size());
- updateChecksumShort(crc, (short) hosts.size());
-
- int recordSize = record.remaining();
- out.writeInt(recordSize);
- updateChecksumInt(crc, recordSize);
+ out.putShort((short)hosts.size());
- out.writeInt((int) crc.getValue());
+ int fixedCrcPosition = out.position();
+ out.position(fixedCrcPosition + TypeSizes.INT_SIZE);
for (int host : hosts)
- {
- out.writeInt(host);
- updateChecksumInt(crc, host);
- }
+ out.putInt(host);
- out.write(record);
- Crc.updateCrc32(crc, record, record.position(), record.limit());
+ int recordSize = record.remaining();
+ int recordEnd = out.position() + recordSize;
+ Invariants.checkState(out.limit() == recordEnd + TypeSizes.INT_SIZE);
+ ByteBufferUtil.copyBytes(record, record.position(), out, out.position(), recordSize);
- out.writeInt((int) crc.getValue());
+ // update and write crcs
+ CRC32 crc = Crc.crc32();
+ out.position(start);
+ out.limit(fixedCrcPosition);
+ crc.update(out);
+ out.limit(recordEnd);
+ out.putInt((int) crc.getValue());
+ crc.update(out);
+ out.limit(recordEnd + 4);
+ out.putInt((int) crc.getValue());
}
+ // we reuse record as the value we return
static <K> void read(EntryHolder<K> into,
KeySupport<K> keySupport,
ByteBuffer from,
int userVersion)
throws IOException
{
- CRC32 crc = Crc.crc32();
into.clear();
- try (DataInputBuffer in = new DataInputBuffer(from, false))
+ int start = from.position();
{
- K key = keySupport.deserialize(in, userVersion);
- keySupport.updateChecksum(crc, key, userVersion);
- into.key = key;
-
- int hostCount = in.readShort();
- updateChecksumShort(crc, (short) hostCount);
-
- int entrySize = in.readInt();
- updateChecksumInt(crc, entrySize);
+ int totalSize = from.getInt(start) - start;
+ Invariants.checkState(totalSize == from.remaining());
- validateCRC(crc, in.readInt());
-
- for (int i = 0; i < hostCount; i++)
- {
- int hostId = in.readInt();
- updateChecksumInt(crc, hostId);
- into.hosts.add(hostId);
- }
-
- // TODO: try to avoid allocating another buffer here
- ByteBuffer entry = ByteBufferUtil.read(in, entrySize);
- updateChecksum(crc, entry);
- into.value = entry;
- into.userVersion = userVersion;
+ CRC32 crc = Crc.crc32();
+ int fixedSize = EntrySerializer.fixedEntrySize(keySupport, userVersion);
+ int fixedCrc = readAndUpdateFixedCrc(crc, from, fixedSize);
+ validateCRC(crc, fixedCrc);
- validateCRC(crc, in.readInt());
+ int recordCrc = readAndUpdateRecordCrc(crc, from, start + totalSize);
+ validateCRC(crc, recordCrc);
}
+
+ readValidated(into, from, start, keySupport, userVersion);
}
- static <K> boolean tryRead(EntryHolder<K> into,
- KeySupport<K> keySupport,
- ByteBuffer from,
- DataInputBuffer in,
- int syncedOffset,
- int userVersion)
+ // slices the provided buffer to assign to into.value
+ static <K> int tryRead(EntryHolder<K> into,
+ KeySupport<K> keySupport,
+ ByteBuffer from,
+ int syncedOffset,
+ int userVersion)
throws IOException
{
CRC32 crc = Crc.crc32();
into.clear();
- int fixedSize = EntrySerializer.fixedEntrySize(keySupport, userVersion);
- if (from.remaining() < fixedSize)
- return handleReadException(new EOFException(), from.limit(), syncedOffset);
+ int start = from.position();
+ if (from.remaining() < TypeSizes.INT_SIZE)
+ return -1;
- updateChecksum(crc, from, from.position(), fixedSize - TypeSizes.INT_SIZE);
- int fixedCrc = from.getInt(from.position() + fixedSize - TypeSizes.INT_SIZE);
+ int totalSize = from.getInt(start) - start;
+ if (totalSize == 0)
+ return -1;
- try
- {
- validateCRC(crc, fixedCrc);
- }
- catch (IOException e)
- {
- return handleReadException(e, from.position() + fixedSize, syncedOffset);
- }
+ if (from.remaining() < totalSize)
+ return handleReadException(new EOFException(), from.limit(), syncedOffset);
- int hostCount, recordSize;
- try
{
- into.key = keySupport.deserialize(in, userVersion);
- hostCount = in.readShort();
- recordSize = in.readInt();
- in.skipBytesFully(TypeSizes.INT_SIZE);
- }
- catch (IOException e)
- {
- throw new RuntimeException(); // can't happen unless deserializer is buggy
- }
+ int fixedSize = EntrySerializer.fixedEntrySize(keySupport, userVersion);
+ int fixedCrc = readAndUpdateFixedCrc(crc, from, fixedSize);
+ try
+ {
+ validateCRC(crc, fixedCrc);
+ }
+ catch (IOException e)
+ {
+ return handleReadException(e, from.position() + fixedSize, syncedOffset);
+ }
- int variableSize = EntrySerializer.variableEntrySize(hostCount, recordSize);
- if (from.remaining() < variableSize)
- return handleReadException(new EOFException(), from.limit(), syncedOffset);
+ int recordCrc = readAndUpdateRecordCrc(crc, from, start + totalSize);
+ try
+ {
+ validateCRC(crc, recordCrc);
+ }
+ catch (IOException e)
+ {
+ return handleReadException(e, from.position(), syncedOffset);
+ }
+ }
- updateChecksum(crc, from, from.position(), variableSize - TypeSizes.INT_SIZE);
- int variableCrc = from.getInt(from.position() + variableSize - TypeSizes.INT_SIZE);
+ readValidated(into, from, start, keySupport, userVersion);
+ return totalSize;
+ }
- try
- {
- validateCRC(crc, variableCrc);
- }
- catch (IOException e)
- {
- return handleReadException(e, from.position() + variableSize, syncedOffset);
- }
+ private static <K> void readValidated(EntryHolder<K> into, ByteBuffer from, int start, KeySupport<K> keySupport, int userVersion)
+ {
+ from.position(start + TypeSizes.INT_SIZE);
+ into.key = keySupport.deserialize(from, userVersion);
+ int hostCount = from.getShort();
+ from.position(from.position() + 4);
for (int i = 0; i < hostCount; i++)
{
- into.hosts.add(in.readInt());
+ int hostId = from.getInt();
+ into.hosts.add(hostId);
}
- try
- {
- in.skipBytesFully(recordSize);
- }
- catch (IOException e)
- {
- throw new AssertionError(); // can't happen
- }
-
- into.value = from.duplicate()
- .position(from.position() - recordSize)
- .limit(from.position());
+ into.value = from;
into.userVersion = userVersion;
+ }
+
+ private static int readAndUpdateFixedCrc(CRC32 crc, ByteBuffer from, int fixedSize)
+ {
+ int fixedEnd = from.position() + fixedSize - TypeSizes.INT_SIZE;
+ int fixedCrc = from.getInt(fixedEnd);
+ from.limit(fixedEnd);
+ crc.update(from);
+ return fixedCrc;
+ }
- in.skipBytesFully(TypeSizes.INT_SIZE);
- return true;
+ private static int readAndUpdateRecordCrc(CRC32 crc, ByteBuffer from, int limit)
+ {
+ int recordEnd = limit - TypeSizes.INT_SIZE;
+ from.limit(limit);
+ int recordCrc = from.getInt(recordEnd);
+ from.position(from.position() + 4);
+ from.limit(recordEnd);
+ crc.update(from);
+ return recordCrc;
}
- private static boolean handleReadException(IOException e, int bufferPosition, int fsyncedLimit) throws IOException
+ private static int handleReadException(IOException e, int bufferPosition, int fsyncedLimit) throws IOException
{
if (bufferPosition <= fsyncedLimit)
throw e;
else
- return false;
+ return -1;
}
static <K> int fixedEntrySize(KeySupport<K> keySupport, int userVersion)
{
return keySupport.serializedSize(userVersion) // key/id
+ TypeSizes.SHORT_SIZE // host count
- + TypeSizes.INT_SIZE // record size
+ + TypeSizes.INT_SIZE // total size
+ TypeSizes.INT_SIZE; // CRC
}
- static int variableEntrySize(int hostCount, int recordSize)
+ static int variableEntrySize(int hostCount)
{
return TypeSizes.INT_SIZE * hostCount // hosts
- + recordSize // record
+ TypeSizes.INT_SIZE; // CRC
}
diff --git a/src/java/org/apache/cassandra/journal/Flusher.java b/src/java/org/apache/cassandra/journal/Flusher.java
index 5dfa76e894..b39caafe2e 100644
--- a/src/java/org/apache/cassandra/journal/Flusher.java
+++ b/src/java/org/apache/cassandra/journal/Flusher.java
@@ -45,7 +45,6 @@ import static org.apache.cassandra.concurrent.InfiniteLoopExecutor.SimulatorSafe
import static org.apache.cassandra.concurrent.Interruptible.State.NORMAL;
import static org.apache.cassandra.concurrent.Interruptible.State.SHUTTING_DOWN;
import static org.apache.cassandra.journal.Params.FlushMode.PERIODIC;
-import static org.apache.cassandra.utils.Clock.Global.nanoTime;
import static org.apache.cassandra.utils.MonotonicClock.Global.preciseTime;
import static org.apache.cassandra.utils.Simulate.With.GLOBAL_CLOCK;
import static org.apache.cassandra.utils.Simulate.With.LOCK_SUPPORT;
@@ -68,35 +67,36 @@ final class Flusher<K, V>
private final AtomicLong written = new AtomicLong(0);
// the time of the last initiated flush
- volatile long flushStartedAt = nanoTime();
+ volatile long flushStartedAt;
// the time of the earliest flush that has completed an fsync; all Allocations written before this time are durable
volatile long fsyncFinishedFor = flushStartedAt;
+ volatile RecordPointer fsyncFinishedForPosition = new RecordPointer(0, 0);
// a signal that writers can wait on to be notified of a completed flush in PERIODIC FlushMode
private final WaitQueue fsyncComplete = newWaitQueue(); // TODO (expected): this is only used for testing, can we remove this?
+ private final MonotonicClock clock = preciseTime;
// a signal and flag that callers outside the flusher thread can use
// to signal they want the journal segments to be flushed to disk
private final Semaphore haveWork = newSemaphore(1);
private volatile boolean flushRequested;
- private final FlushMethod<K, V> syncFlushMethod;
- private final FlushMethod<K, V> asyncFlushMethod;
+ private final Mode<K, V> mode;
private final Callbacks callbacks;
Flusher(Journal<K, V> journal, Callbacks callbacks)
{
this.journal = journal;
this.params = journal.params;
- this.syncFlushMethod = syncFlushMethod(params);
- this.asyncFlushMethod = asyncFlushMethod(params);
+ this.mode = mode(params);
this.callbacks = callbacks;
}
void start()
{
String flushExecutorName = journal.name + "-disk-flusher-" + params.flushMode().toString().toLowerCase();
- flushExecutor = executorFactory().infiniteLoop(flushExecutorName, new FlushRunnable(preciseTime), SAFE, NON_DAEMON, SYNCHRONIZED);
+ flushStartedAt = clock.now();
+ flushExecutor = executorFactory().infiniteLoop(flushExecutorName, new FlushRunnable(), SAFE, NON_DAEMON, SYNCHRONIZED);
}
void shutdown() throws InterruptedException
@@ -111,6 +111,7 @@ final class Flusher<K, V>
}
@Simulate(with={MONITORS,GLOBAL_CLOCK,LOCK_SUPPORT})
+ // waits for writes to complete before triggering an fsync
private class FlushRunnable implements Interruptible.Task
{
@Simulate(with={MONITORS,GLOBAL_CLOCK,LOCK_SUPPORT})
@@ -181,6 +182,7 @@ final class Flusher<K, V>
fsyncStartedFor = startedAt;
// synchronized to prevent thread interrupts while performing IO operations and also
// clear interrupted status to prevent ClosedByInterruptException in ActiveSegment::flush
+ int fsyncedTo;
synchronized (this)
{
boolean ignore = Thread.interrupted();
@@ -190,17 +192,19 @@ final class Flusher<K, V>
journal.closeActiveSegmentAndOpenAsStatic(fsyncing);
fsyncing = journal.getActiveSegment(fsyncing.descriptor.timestamp + 1);
}
- fsyncing.fsync();
+ fsyncedTo = fsyncTo.writtenToAtLeast();
+ fsyncTo.fsync();
}
+ fsyncFinishedForPosition = new RecordPointer(fsyncTo.descriptor.timestamp, fsyncedTo, startedAt);
fsyncFinishedFor = startedAt;
fsyncComplete.signalAll();
long finishedAt = clock.now();
processDuration(startedAt, finishedAt);
}
- void afterFlush(long startedAt, ActiveSegment<K, V> segment, int syncedOffset)
+ void afterFlush(long startedAt, ActiveSegment<K, V> segment)
{
- long requireFsyncTo = startedAt - periodicFlushLagBlockNanos();
+ long requireFsyncTo = startedAt - periodicBlockNanos();
fsyncUpTo = segment;
fsyncWaitingSince = startedAt;
@@ -209,26 +213,11 @@ final class Flusher<K, V>
if (requireFsyncTo > fsyncFinishedFor)
awaitFsyncAt(requireFsyncTo, journal.metrics.waitingOnFlush.time());
- callbacks.onFlush(segment.descriptor.timestamp, syncedOffset);
- }
-
- private void doNoOpFlush(long startedAt)
- {
- if (fsyncFinishedFor >= fsyncWaitingSince)
- {
- fsyncFinishedFor = startedAt;
- }
- else
- {
- // if the flusher is still running, update the waitingSince register
- fsyncWaitingSince = startedAt;
- notify(awaitingWork);
- }
+ callbacks.onFsync();
}
}
private final NoSpamLogger noSpamLogger;
- private final MonotonicClock clock;
private final @Nullable FSyncRunnable fSyncRunnable;
private ActiveSegment<K, V> current = null;
@@ -239,10 +228,9 @@ final class Flusher<K, V>
private long duration = 0; // time spent flushing since firstLaggedAt
private long lagDuration = 0; // cumulative lag since firstLaggedAt
- FlushRunnable(MonotonicClock clock)
+ FlushRunnable()
{
this.noSpamLogger = NoSpamLogger.wrap(logger, 5, MINUTES);
- this.clock = clock;
this.fSyncRunnable = params.flushMode() == PERIODIC ? newFsyncRunnable() : null;
}
@@ -286,6 +274,7 @@ final class Flusher<K, V>
if (flushPeriodNanos <= 0)
{
+ Invariants.checkState(params.flushMode() != PERIODIC);
haveWork.acquire(1);
}
else
@@ -295,7 +284,7 @@ final class Flusher<K, V>
}
}
- private void doFlush(long startedAt) throws InterruptedException
+ private void doFlush(long startedAt)
{
boolean synchronousFsync = fSyncRunnable == null;
@@ -303,32 +292,33 @@ final class Flusher<K, V>
current = journal.oldestActiveSegment();
ActiveSegment<K, V> newCurrent = journal.currentActiveSegment();
- if (newCurrent == current && (newCurrent == null || !newCurrent.shouldFlush()))
- {
- if (synchronousFsync) fsyncFinishedFor = startedAt;
- else fSyncRunnable.doNoOpFlush(startedAt);
-
- if (current != null)
- callbacks.onFlush(current.descriptor.timestamp, (int) current.lastFlushedOffset());
+ if (newCurrent == null)
return;
- }
-
- Invariants.checkState(newCurrent != null);
try
{
while (current != newCurrent)
{
current.discardUnusedTail();
- current.flush(synchronousFsync);
+ current.updateWrittenTo();
if (synchronousFsync)
+ {
+ current.fsync();
journal.closeActiveSegmentAndOpenAsStatic(current);
+ }
current = journal.getActiveSegment(current.descriptor.timestamp + 1);
}
- int syncedOffset = current.flush(synchronousFsync);
- if (synchronousFsync) afterFSync(startedAt, current.descriptor.timestamp, syncedOffset);
- else fSyncRunnable.afterFlush(startedAt, current, syncedOffset);
+ int writtenTo = current.updateWrittenTo();
+ if (synchronousFsync)
+ {
+ current.fsync();
+ afterFSync(startedAt, current.descriptor.timestamp, writtenTo);
+ }
+ else
+ {
+ fSyncRunnable.afterFlush(startedAt, current);
+ }
}
catch (Throwable t)
{
@@ -372,10 +362,11 @@ final class Flusher<K, V>
}
}
- private void afterFSync(long startedAt, long syncedSegment, int syncedOffset)
+ private void afterFSync(long startedAt, long segment, int position)
{
+ fsyncFinishedForPosition = new RecordPointer(segment, position, startedAt);
fsyncFinishedFor = startedAt;
- callbacks.onFlush(syncedSegment, syncedOffset);
+ callbacks.onFsync();
fsyncComplete.signalAll();
long finishedAt = clock.now();
processDuration(startedAt, finishedAt);
@@ -389,88 +380,120 @@ final class Flusher<K, V>
}
}
- @FunctionalInterface
- private interface FlushMethod<K, V>
+ private interface Mode<K, V>
{
- void flush(ActiveSegment<K, V>.Allocation allocation);
+ void flushAndAwaitDurable(ActiveSegment<K, V>.Allocation alloc);
+ RecordPointer flushAsync(ActiveSegment<K, V>.Allocation alloc);
+ boolean isDurable(RecordPointer recordPointer);
}
- private FlushMethod<K, V> syncFlushMethod(Params params)
+ private class BatchMode implements Mode<K, V>
{
- switch (params.flushMode())
+ @Override
+ public void flushAndAwaitDurable(ActiveSegment<K, V>.Allocation alloc)
{
- default: throw new IllegalArgumentException();
- case BATCH: return this::waitForFlushBatch;
- case GROUP: return this::waitForFlushGroup;
- case PERIODIC: return this::waitForFlushPeriodic;
+ pending.incrementAndGet();
+ requestExtraFlush();
+ alloc.awaitDurable(journal.metrics.waitingOnFlush);
+ pending.decrementAndGet();
+ written.incrementAndGet();
}
- }
- private FlushMethod<K, V> asyncFlushMethod(Params params)
- {
- switch (params.flushMode())
+ @Override
+ public RecordPointer flushAsync(ActiveSegment<K, V>.Allocation alloc)
{
- default: throw new IllegalArgumentException();
- case BATCH: return this::asyncFlushBatch;
- case GROUP: return this::asyncFlushGroup;
- case PERIODIC: return this::asyncFlushPeriodic;
+ requestExtraFlush();
+ written.incrementAndGet();
+ return new RecordPointer(alloc.descriptor().timestamp, alloc.start());
}
- }
- void waitForFlush(ActiveSegment<K, V>.Allocation alloc)
- {
- syncFlushMethod.flush(alloc);
+ @Override
+ public boolean isDurable(RecordPointer pointer)
+ {
+ return pointer.compareTo(fsyncFinishedForPosition) <= 0;
+ }
}
- void asyncFlush(ActiveSegment<K, V>.Allocation alloc)
+ private class GroupMode implements Mode<K, V>
{
- asyncFlushMethod.flush(alloc);
- }
+ @Override
+ public void flushAndAwaitDurable(ActiveSegment<K, V>.Allocation alloc)
+ {
+ pending.incrementAndGet();
+ alloc.awaitDurable(journal.metrics.waitingOnFlush);
+ pending.decrementAndGet();
+ written.incrementAndGet();
+ }
- private void waitForFlushBatch(ActiveSegment<K, V>.Allocation alloc)
- {
- pending.incrementAndGet();
- requestExtraFlush();
- alloc.awaitFlush(journal.metrics.waitingOnFlush);
- pending.decrementAndGet();
- written.incrementAndGet();
+ @Override
+ public RecordPointer flushAsync(ActiveSegment<K, V>.Allocation alloc)
+ {
+ written.incrementAndGet();
+ return new RecordPointer(alloc.descriptor().timestamp, alloc.start());
+ }
+
+ @Override
+ public boolean isDurable(RecordPointer pointer)
+ {
+ return pointer.compareTo(fsyncFinishedForPosition) <= 0;
+ }
}
- private void waitForFlushGroup(ActiveSegment<K, V>.Allocation alloc)
+ private class PeriodicMode implements Mode<K, V>
{
- pending.incrementAndGet();
- alloc.awaitFlush(journal.metrics.waitingOnFlush);
- pending.decrementAndGet();
- written.incrementAndGet();
+ @Override
+ public void flushAndAwaitDurable(ActiveSegment<K, V>.Allocation alloc)
+ {
+ RecordPointer pointer = flushAsync(alloc);
+
+ long expectedFsyncTime = pointer.writtenAt - periodicBlockNanos();
+ if (expectedFsyncTime > fsyncFinishedFor)
+ {
+ pending.incrementAndGet();
+ awaitFsyncAt(expectedFsyncTime, journal.metrics.waitingOnFlush.time());
+ pending.decrementAndGet();
+ }
+ }
+
+ @Override
+ public RecordPointer flushAsync(ActiveSegment<K, V>.Allocation alloc)
+ {
+ written.incrementAndGet();
+ return new RecordPointer(alloc.descriptor().timestamp, alloc.start(), clock.now());
+ }
+
+ @Override
+ public boolean isDurable(RecordPointer alloc)
+ {
+ long expectedFsyncTime = alloc.writtenAt - periodicBlockNanos();
+ return expectedFsyncTime <= fsyncFinishedFor;
+ }
}
- private void waitForFlushPeriodic(ActiveSegment<K, V>.Allocation ignore)
+ Mode<K, V> mode(Params params)
{
- long expectedFlushTime = nanoTime() - periodicFlushLagBlockNanos();
- if (fsyncFinishedFor < expectedFlushTime)
+ switch (params.flushMode())
{
- pending.incrementAndGet();
- awaitFsyncAt(expectedFlushTime, journal.metrics.waitingOnFlush.time());
- pending.decrementAndGet();
+ default: throw new AssertionError("Unexpected FlushMode: " + params.flushMode());
+ case BATCH: return new BatchMode();
+ case GROUP: return new GroupMode();
+ case PERIODIC: return new PeriodicMode();
}
- written.incrementAndGet();
}
- private void asyncFlushBatch(ActiveSegment<K, V>.Allocation alloc)
+ RecordPointer flush(ActiveSegment<K, V>.Allocation alloc)
{
- requestExtraFlush();
- written.incrementAndGet();
+ return mode.flushAsync(alloc);
}
- private void asyncFlushGroup(ActiveSegment<K, V>.Allocation alloc)
+ void flushAndAwaitDurable(ActiveSegment<K, V>.Allocation alloc)
{
- written.incrementAndGet();
+ mode.flushAndAwaitDurable(alloc);
}
- private void asyncFlushPeriodic(ActiveSegment<K, V>.Allocation ignore)
+ boolean isDurable(RecordPointer pointer)
{
- requestExtraFlush();
- written.incrementAndGet();
+ return mode.isDurable(pointer);
}
/**
@@ -504,12 +527,12 @@ final class Flusher<K, V>
private long flushPeriodNanos()
{
- return 1_000_000L * params.flushPeriodMillis();
+ return params.flushPeriod(NANOSECONDS);
}
- private long periodicFlushLagBlockNanos()
+ private long periodicBlockNanos()
{
- return 1_000_000L * params.periodicFlushLagBlock();
+ return params.periodicBlockPeriod(NANOSECONDS);
}
long pendingEntries()
@@ -530,8 +553,9 @@ final class Flusher<K, V>
* completed and also flushed.
* callbacks for all entries earlier than (segment, position) have finished execution.
*/
- void onFlush(long segment, int position);
+ void onFsync();
+ // TODO (required): tie this to specific allocations..
void onFlushFailed(Throwable cause);
}
}
diff --git a/src/java/org/apache/cassandra/journal/InMemoryIndex.java b/src/java/org/apache/cassandra/journal/InMemoryIndex.java
index 77fd7352ee..baa56853cc 100644
--- a/src/java/org/apache/cassandra/journal/InMemoryIndex.java
+++ b/src/java/org/apache/cassandra/journal/InMemoryIndex.java
@@ -102,7 +102,10 @@ final class InMemoryIndex<K> extends Index<K>
@Override
public long[] lookUp(K id)
{
- return mayContainId(id) ? index.getOrDefault(id, EMPTY) : EMPTY;
+ K lastId = lastId();
+ if (lastId == null || keySupport.compare(id, lastId) <= 0)
+ return index.getOrDefault(id, EMPTY);
+ return EMPTY;
}
@Override
diff --git a/src/java/org/apache/cassandra/journal/Index.java b/src/java/org/apache/cassandra/journal/Index.java
index ac1e7c1d91..fba9b99f86 100644
--- a/src/java/org/apache/cassandra/journal/Index.java
+++ b/src/java/org/apache/cassandra/journal/Index.java
@@ -21,8 +21,6 @@ import javax.annotation.Nullable;
import org.apache.cassandra.utils.Closeable;
-import static com.google.common.collect.Iterables.any;
-
/**
* Mapping of client supplied ids to in-segment offsets
*/
@@ -75,15 +73,7 @@ abstract class Index<K> implements Closeable
K firstId = firstId();
K lastId = lastId();
- return null != firstId && null != lastId && keySupport.compare(id, firstId) >= 0 && keySupport.compare(id, lastId) <= 0;
- }
-
- /**
- * @return whether any of the ids falls within lower/upper bounds of the index
- */
- boolean mayContainIds(Iterable<K> ids)
- {
- return any(ids, this::mayContainId);
+ return null != firstId && keySupport.compare(id, firstId) >= 0 && (null == lastId || keySupport.compare(id, lastId) <= 0);
}
/**
diff --git a/src/java/org/apache/cassandra/journal/Journal.java b/src/java/org/apache/cassandra/journal/Journal.java
index 34aff492ea..f2dce51540 100644
--- a/src/java/org/apache/cassandra/journal/Journal.java
+++ b/src/java/org/apache/cassandra/journal/Journal.java
@@ -41,7 +41,6 @@ import org.slf4j.LoggerFactory;
import accord.utils.Invariants;
import com.codahale.metrics.Timer.Context;
-import org.agrona.collections.ObjectHashSet;
import org.apache.cassandra.concurrent.Interruptible;
import org.apache.cassandra.concurrent.Interruptible.TerminateException;
import org.apache.cassandra.concurrent.SequentialExecutorPlus;
@@ -51,12 +50,12 @@ import org.apache.cassandra.io.util.DataOutputBuffer;
import org.apache.cassandra.io.util.DataOutputPlus;
import org.apache.cassandra.io.util.File;
import org.apache.cassandra.io.util.PathUtils;
-import org.apache.cassandra.journal.Segments.ReferencedSegment;
import org.apache.cassandra.journal.Segments.ReferencedSegments;
import org.apache.cassandra.service.StorageService;
import org.apache.cassandra.utils.Crc;
import org.apache.cassandra.utils.JVMStabilityInspector;
import org.apache.cassandra.utils.Simulate;
+import org.apache.cassandra.utils.concurrent.OpOrder;
import org.apache.cassandra.utils.concurrent.WaitQueue;
import org.jctools.queues.MpscUnboundedArrayQueue;
@@ -119,8 +118,9 @@ public class Journal<K, V> implements Shutdownable
private final WaitQueue allocatorThreadWaitQueue = newWaitQueue();
private final BooleanSupplier allocatorThreadWaitCondition = () -> (availableSegment == null);
private final FlusherCallbacks flusherCallbacks;
+ final OpOrder readOrder = new OpOrder();
- SequentialExecutorPlus closer;
+ SequentialExecutorPlus closer, releaser;
private class FlusherCallbacks implements Flusher.Callbacks
{
@@ -128,17 +128,14 @@ public class Journal<K, V> implements Shutdownable
private List<WaitingFor> drained = new ArrayList<>();
@Override
- public void onFlush(long segment, int position)
+ public void onFsync()
{
- // TODO (required): this seems to be a big source of allocations
waitingFor.drain(drained::add);
List<WaitingFor> remaining = new ArrayList<>();
for (WaitingFor wait : drained)
{
- if (wait.segment == segment && wait.position <= position)
- wait.run();
- else
- remaining.add(wait);
+ if (flusher.isDurable(wait)) wait.run();
+ else remaining.add(wait);
}
drained = remaining;
}
@@ -151,13 +148,10 @@ public class Journal<K, V> implements Shutdownable
private void submit(RecordPointer pointer, Runnable runnable)
{
- if (isFlushed(pointer))
+ if (flusher.isDurable(pointer))
runnable.run();
else
- {
- waitingFor.add(new WaitingFor(pointer.segment, pointer.position, runnable));
- flusher.requestExtraFlush();
- }
+ waitingFor.add(new WaitingFor(pointer, runnable));
}
}
@@ -165,9 +159,9 @@ public class Journal<K, V> implements Shutdownable
{
private final Runnable onFlush;
- public WaitingFor(long segment, int position, Runnable onFlush)
+ public WaitingFor(RecordPointer pointer, Runnable onFlush)
{
- super(segment, position);
+ super(pointer);
this.onFlush = onFlush;
}
@@ -197,19 +191,9 @@ public class Journal<K, V> implements Shutdownable
this.compactor = new Compactor<>(this, segmentCompactor);
}
- public boolean isFlushed(RecordPointer recordPointer)
+ public void onDurable(RecordPointer recordPointer, Runnable runnable)
{
- Segment<K, V> current = currentSegment;
- if (current.descriptor.timestamp == recordPointer.segment)
- return current.isFlushed(recordPointer.position);
-
- return segments.get().isFlushed(recordPointer);
- }
-
- public void onFlush(RecordPointer recordPointer, Runnable runnable)
- {
- if (isFlushed(recordPointer)) runnable.run();
- else flusherCallbacks.submit(recordPointer, runnable);
+ flusherCallbacks.submit(recordPointer, runnable);
}
public void start()
@@ -230,6 +214,7 @@ public class Journal<K, V> implements Shutdownable
segments.set(Segments.of(StaticSegment.open(descriptors, keySupport)));
closer = executorFactory().sequential(name + "-closer");
+ releaser = executorFactory().sequential(name + "-releaser");
allocator = executorFactory().infiniteLoop(name + "-allocator", new AllocateRunnable(), SAFE, NON_DAEMON, SYNCHRONIZED);
advanceSegment(null);
flusher.start();
@@ -272,13 +257,16 @@ public class Journal<K, V> implements Shutdownable
"Unexpected journal state while trying to shut down", state);
allocator.shutdown();
wakeAllocator(); // Wake allocator to force it into shutdown
+ // TODO (expected): why are we awaitingTermination here when we have a separate method for it?
allocator.awaitTermination(1, TimeUnit.MINUTES);
segmentPrepared.signalAll(); // Wake up all threads waiting on the new segment
compactor.shutdown();
compactor.awaitTermination(1, TimeUnit.MINUTES);
flusher.shutdown();
closer.shutdown();
+ releaser.shutdown();
closer.awaitTermination(1, TimeUnit.MINUTES);
+ releaser.awaitTermination(1, TimeUnit.MINUTES);
closeAllSegments();
metrics.deregister();
Invariants.checkState(state.compareAndSet(State.SHUTDOWN, State.TERMINATED),
@@ -303,6 +291,7 @@ public class Journal<K, V> implements Shutdownable
boolean r = true;
r &= allocator.awaitTermination(timeout, units);
r &= closer.awaitTermination(timeout, units);
+ r &= releaser.awaitTermination(timeout, units);
return r;
}
@@ -323,9 +312,9 @@ public class Journal<K, V> implements Shutdownable
{
EntrySerializer.EntryHolder<K> holder = new EntrySerializer.EntryHolder<>();
- try (ReferencedSegments<K, V> segments = selectAndReference(id))
+ try (OpOrder.Group group = readOrder.start())
{
- for (Segment<K, V> segment : segments.allSorted(true))
+ for (Segment<K, V> segment : segments.get().allSorted(true))
{
if (segment.readLast(id, holder))
{
@@ -347,9 +336,9 @@ public class Journal<K, V> implements Shutdownable
public void readAll(K id, RecordConsumer<K> consumer)
{
EntrySerializer.EntryHolder<K> holder = new EntrySerializer.EntryHolder<>();
- try (ReferencedSegments<K, V> segments = selectAndReference(id))
+ try (OpOrder.Group group = readOrder.start())
{
- for (Segment<K, V> segment : segments.allSorted(false))
+ for (Segment<K, V> segment : segments.get().allSorted(false))
segment.readAll(id, holder, consumer);
}
}
@@ -390,9 +379,9 @@ public class Journal<K, V> implements Shutdownable
{
EntrySerializer.EntryHolder<K> holder = new EntrySerializer.EntryHolder<>();
- try (ReferencedSegments<K, V> segments = selectAndReference(id))
+ try (OpOrder.Group group = readOrder.start())
{
- for (Segment<K, V> segment : segments.all())
+ for (Segment<K, V> segment : segments.get().all())
{
long[] offsets = segment.index().lookUp(id);
for (long offsetAndSize : offsets)
@@ -436,40 +425,18 @@ public class Journal<K, V> implements Shutdownable
@SuppressWarnings("unused")
public boolean readLast(K id, RecordConsumer<K> consumer)
{
- try (ReferencedSegments<K, V> segments = selectAndReference(id))
+ try (OpOrder.Group group = readOrder.start())
{
- for (Segment<K, V> segment : segments.all())
+ for (Segment<K, V> segment : segments.get().allSorted(false))
+ {
+ if (!segment.index().mayContainId(id))
+ continue;
+
if (segment.readLast(id, consumer))
return true;
- }
- return false;
- }
-
- /**
- * Test for existence of entries with specified ids.
- *
- * @return subset of ids to test that have been found in the journal
- */
- @SuppressWarnings("unused")
- public Set<K> test(Set<K> test)
- {
- Set<K> present = new ObjectHashSet<>(test.size() + 1, 0.9f);
- try (ReferencedSegments<K, V> segments = selectAndReference(test))
- {
- for (Segment<K, V> segment : segments.all())
- {
- for (K id : test)
- {
- if (segment.index().lookUpLast(id) != -1)
- {
- present.add(id);
- if (test.size() == present.size())
- return present;
- }
- }
}
}
- return present;
+ return false;
}
/**
@@ -488,7 +455,7 @@ public class Journal<K, V> implements Shutdownable
valueSerializer.serialize(id, record, dob, params.userVersion());
ActiveSegment<K, V>.Allocation alloc = allocate(dob.getLength(), hosts);
alloc.writeInternal(id, dob.unsafeGetBufferAndFlip(), hosts);
- flusher.waitForFlush(alloc);
+ flusher.flushAndAwaitDurable(alloc);
}
catch (IOException e)
{
@@ -514,20 +481,18 @@ public class Journal<K, V> implements Shutdownable
public RecordPointer asyncWrite(K id, Writer writer, Set<Integer> hosts)
{
- RecordPointer recordPointer;
try (DataOutputBuffer dob = DataOutputBuffer.scratchBuffer.get())
{
writer.write(dob, params.userVersion());
ActiveSegment<K, V>.Allocation alloc = allocate(dob.getLength(), hosts);
- recordPointer = alloc.write(id, dob.unsafeGetBufferAndFlip(), hosts);
- flusher.asyncFlush(alloc);
+ alloc.write(id, dob.unsafeGetBufferAndFlip(), hosts);
+ return flusher.flush(alloc);
}
catch (IOException e)
{
// exception during record serialization into the scratch buffer
throw new RuntimeException(e);
}
- return recordPointer;
}
private ActiveSegment<K, V>.Allocation allocate(int entrySize, Set<Integer> hosts)
@@ -615,7 +580,7 @@ public class Journal<K, V> implements Shutdownable
availableSegment = null;
}
if (next != null)
- next.closeAndDiscard();
+ next.closeAndDiscard(this);
}
private class AllocateRunnable implements Interruptible.Task
@@ -715,52 +680,9 @@ public class Journal<K, V> implements Shutdownable
for (Segment<K, V> segment : segments.all())
{
if (segment.isActive())
- ((ActiveSegment<K, V>) segment).closeAndIfEmptyDiscard();
+ ((ActiveSegment<K, V>) segment).closeAndIfEmptyDiscard(this);
else
- segment.close();
- }
- }
-
- /**
- * Select segments that could potentially have any entry with the specified id and
- * attempt to grab references to them all.
- *
- * @return a subset of segments with references to them
- */
- ReferencedSegments<K, V> selectAndReference(K id)
- {
- while (true)
- {
- ReferencedSegments<K, V> referenced = segments().selectAndReference(s -> s.index().mayContainId(id));
- if (null != referenced)
- return referenced;
- }
- }
-
- /**
- * Select segments that could potentially have any entry with the specified ids and
- * attempt to grab references to them all.
- *
- * @return a subset of segments with references to them
- */
- ReferencedSegments<K, V> selectAndReference(Iterable<K> ids)
- {
- while (true)
- {
- ReferencedSegments<K, V> referenced = segments().selectAndReference(s -> s.index().mayContainIds(ids));
- if (null != referenced)
- return referenced;
- }
- }
-
- @SuppressWarnings("unused")
- ReferencedSegment<K, V> selectAndReference(long segmentTimestamp)
- {
- while (true)
- {
- ReferencedSegment<K, V> referenced = segments().selectAndReference(segmentTimestamp);
- if (null != referenced)
- return referenced;
+ segment.close(this);
}
}
@@ -886,10 +808,11 @@ public class Journal<K, V> implements Shutdownable
public void run()
{
activeSegment.discardUnusedTail();
- activeSegment.flush(true);
+ activeSegment.updateWrittenTo();
+ activeSegment.fsync();
activeSegment.persistComponents();
replaceCompletedSegment(activeSegment, StaticSegment.open(activeSegment.descriptor, keySupport));
- activeSegment.release();
+ activeSegment.release(Journal.this);
}
}
@@ -898,7 +821,7 @@ public class Journal<K, V> implements Shutdownable
if (activeSegment.isEmpty())
{
removeEmptySegment(activeSegment);
- activeSegment.closeAndDiscard();
+ activeSegment.closeAndDiscard(this);
return;
}
@@ -1000,6 +923,7 @@ public class Journal<K, V> implements Shutdownable
*/
public class StaticSegmentIterator implements Closeable
{
+ // TODO (expected): use MergeIterator
private final PriorityQueue<StaticSegment.KeyOrderReader<K>> readers;
private final ReferencedSegments<K, V> segments;
diff --git a/src/java/org/apache/cassandra/journal/KeySupport.java b/src/java/org/apache/cassandra/journal/KeySupport.java
index 13cb902ddf..efc41aa6c8 100644
--- a/src/java/org/apache/cassandra/journal/KeySupport.java
+++ b/src/java/org/apache/cassandra/journal/KeySupport.java
@@ -36,10 +36,12 @@ public interface KeySupport<K> extends Comparator<K>
int serializedSize(int userVersion);
void serialize(K key, DataOutputPlus out, int userVersion) throws IOException;
+ void serialize(K key, ByteBuffer out, int userVersion) throws IOException;
K deserialize(DataInputPlus in, int userVersion) throws IOException;
K deserialize(ByteBuffer buffer, int position, int userVersion);
+ K deserialize(ByteBuffer buffer, int userVersion);
void updateChecksum(Checksum crc, K key, int userVersion);
diff --git a/src/java/org/apache/cassandra/journal/Metadata.java b/src/java/org/apache/cassandra/journal/Metadata.java
index e8224ca64e..fcdbe1fba2 100644
--- a/src/java/org/apache/cassandra/journal/Metadata.java
+++ b/src/java/org/apache/cassandra/journal/Metadata.java
@@ -17,6 +17,7 @@
*/
package org.apache.cassandra.journal;
+import java.io.EOFException;
import java.io.IOException;
import java.util.Collections;
import java.util.Map;
@@ -45,6 +46,7 @@ final class Metadata
private final Set<Integer> unmodifiableHosts;
private final Map<Integer, Integer> recordsPerHost;
+ private int fsyncLimit;
private volatile int recordsCount;
private static final AtomicIntegerFieldUpdater<Metadata> recordsCountUpdater =
AtomicIntegerFieldUpdater.newUpdater(Metadata.class, "recordsCount");
@@ -73,6 +75,11 @@ final class Metadata
recordsPerHost.compute(host, (k, v) -> null == v ? 1 : v + 1);
}
+ int fsyncLimit()
+ {
+ return fsyncLimit;
+ }
+
private void incrementRecordsCount()
{
recordsCountUpdater.incrementAndGet(this);
@@ -186,12 +193,12 @@ final class Metadata
}
}
- static <K> Metadata rebuild(Descriptor descriptor, KeySupport<K> keySupport, int fsyncedLimit)
+ static <K> Metadata rebuild(Descriptor descriptor, KeySupport<K> keySupport)
{
Int2IntHashMap recordsPerHost = new Int2IntHashMap(Integer.MIN_VALUE);
int recordsCount = 0;
- try (StaticSegment.SequentialReader<K> reader = StaticSegment.sequentialReader(descriptor, keySupport, fsyncedLimit))
+ try (StaticSegment.SequentialReader<K> reader = StaticSegment.sequentialReader(descriptor, keySupport, Integer.MAX_VALUE))
{
while (reader.advance())
{
@@ -203,13 +210,19 @@ final class Metadata
++recordsCount;
}
}
+ catch (JournalReadError e)
+ {
+ // we expect EOF when rebuilding
+ if (!(e.getCause() instanceof EOFException))
+ throw e;
+ }
return new Metadata(recordsPerHost, recordsCount);
}
- static <K> Metadata rebuildAndPersist(Descriptor descriptor, KeySupport<K> keySupport, int fsyncedLimit)
+ static <K> Metadata rebuildAndPersist(Descriptor descriptor, KeySupport<K> keySupport)
{
- Metadata metadata = rebuild(descriptor, keySupport, fsyncedLimit);
+ Metadata metadata = rebuild(descriptor, keySupport);
metadata.persist(descriptor);
return metadata;
}
diff --git a/src/java/org/apache/cassandra/journal/Params.java b/src/java/org/apache/cassandra/journal/Params.java
index 56bacce1d9..452f024fbc 100644
--- a/src/java/org/apache/cassandra/journal/Params.java
+++ b/src/java/org/apache/cassandra/journal/Params.java
@@ -17,6 +17,8 @@
*/
package org.apache.cassandra.journal;
+import java.util.concurrent.TimeUnit;
+
public interface Params
{
enum FlushMode { BATCH, GROUP, PERIODIC }
@@ -40,23 +42,18 @@ public interface Params
boolean enableCompaction();
- int compactionPeriodMillis();
+ long compactionPeriod(TimeUnit units);
/**
* @return milliseconds between journal flushes
*/
- int flushPeriodMillis();
-
- default int flushPeriodNanos()
- {
- return flushPeriodMillis() * 1_000_000;
- }
+ long flushPeriod(TimeUnit units);
/**
- * @return milliseconds to block writes for while waiting for a slow disk flush to complete
- * when in {@link FlushMode#PERIODIC} mode
+ * @return to block writes for while waiting for a slow fsync to complete
+ * when in {@link FlushMode#PERIODIC} mode
*/
- int periodicFlushLagBlock();
+ long periodicBlockPeriod(TimeUnit units);
/**
* @return user provided version to use for key and value serialization
diff --git a/src/java/org/apache/cassandra/journal/RecordPointer.java b/src/java/org/apache/cassandra/journal/RecordPointer.java
index 2b3e8ea6b8..1439e63c90 100644
--- a/src/java/org/apache/cassandra/journal/RecordPointer.java
+++ b/src/java/org/apache/cassandra/journal/RecordPointer.java
@@ -26,11 +26,23 @@ public class RecordPointer implements Comparable<RecordPointer>
{
public final long segment; // unique segment id
public final int position; // record start position within the segment
+ public final long writtenAt; // only set for periodic mode
public RecordPointer(long segment, int position)
+ {
+ this(segment, position, 0);
+ }
+
+ public RecordPointer(long segment, int position, long writtenAt)
{
this.segment = segment;
this.position = position;
+ this.writtenAt = writtenAt;
+ }
+
+ public RecordPointer(RecordPointer pointer)
+ {
+ this(pointer.segment, pointer.position, pointer.writtenAt);
}
@Override
diff --git a/src/java/org/apache/cassandra/journal/Segment.java b/src/java/org/apache/cassandra/journal/Segment.java
index 77f7c68fea..6f8cde0358 100644
--- a/src/java/org/apache/cassandra/journal/Segment.java
+++ b/src/java/org/apache/cassandra/journal/Segment.java
@@ -20,25 +20,44 @@ package org.apache.cassandra.journal;
import java.nio.ByteBuffer;
import accord.utils.Invariants;
+import org.apache.cassandra.concurrent.ExecutorPlus;
import org.apache.cassandra.io.util.File;
-import org.apache.cassandra.utils.*;
-import org.apache.cassandra.utils.concurrent.RefCounted;
+import org.apache.cassandra.utils.concurrent.OpOrder;
+import org.apache.cassandra.utils.concurrent.Ref;
+import org.apache.cassandra.utils.concurrent.SelfRefCounted;
-public abstract class Segment<K, V> implements Closeable, RefCounted<Segment<K, V>>
+public abstract class Segment<K, V> implements SelfRefCounted<Segment<K, V>>, Comparable<Segment<K, V>>
{
+ protected abstract static class Tidier implements Tidy, Runnable
+ {
+ OpOrder.Barrier await;
+ ExecutorPlus executor;
+
+ abstract void onUnreferenced();
+
+ public final void run()
+ {
+ await.await();
+ onUnreferenced();
+ }
+
+ public final void tidy()
+ {
+ executor.execute(this);
+ }
+ }
+
final File file;
final Descriptor descriptor;
- final SyncedOffsets syncedOffsets;
final Metadata metadata;
final KeySupport<K> keySupport;
ByteBuffer buffer;
- Segment(Descriptor descriptor, SyncedOffsets syncedOffsets, Metadata metadata, KeySupport<K> keySupport)
+ Segment(Descriptor descriptor, Metadata metadata, KeySupport<K> keySupport)
{
this.file = descriptor.fileFor(Component.DATA);
this.descriptor = descriptor;
- this.syncedOffsets = syncedOffsets;
this.metadata = metadata;
this.keySupport = keySupport;
}
@@ -98,7 +117,27 @@ public abstract class Segment<K, V> implements Closeable, RefCounted<Segment<K,
}
}
+ @Override
+ public int compareTo(Segment<K, V> that)
+ {
+ return this.descriptor.compareTo(that.descriptor);
+ }
+
abstract boolean read(int offset, int size, EntrySerializer.EntryHolder<K> into);
- abstract void release();
+ abstract void close(Journal<K, V> journal);
+
+ void release(Journal<K, V> journal)
+ {
+ Ref<Segment<K, V>> selfRef = selfRef();
+ Tidier tidier = (Tidier) selfRef.tidier();
+ if (journal != null)
+ {
+ // permitted to be null ONLY for tests
+ tidier.await = journal.readOrder.newBarrier();
+ tidier.await.issue();
+ tidier.executor = journal.releaser;
+ }
+ selfRef.release();
+ }
}
diff --git a/src/java/org/apache/cassandra/journal/SegmentWriter.java b/src/java/org/apache/cassandra/journal/SegmentWriter.java
deleted file mode 100644
index 09797c363e..0000000000
--- a/src/java/org/apache/cassandra/journal/SegmentWriter.java
+++ /dev/null
@@ -1,114 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.cassandra.journal;
-
-import java.io.Closeable;
-import java.io.IOException;
-import java.nio.ByteBuffer;
-import java.util.Set;
-
-import com.google.common.primitives.Ints;
-
-import org.apache.cassandra.io.util.File;
-import org.apache.cassandra.io.util.FileOutputStreamPlus;
-import org.apache.cassandra.io.util.TrackedDataOutputPlus;
-
-final class SegmentWriter<K> implements Closeable
-{
- private final Descriptor descriptor;
- private final KeySupport<K> keySupport;
-
- private final InMemoryIndex<K> index;
- private final Metadata metadata;
-
- private final File file;
- private FileOutputStreamPlus untrackedOut;
- private TrackedDataOutputPlus trackedOut;
-
- private SegmentWriter(Descriptor descriptor, KeySupport<K> keySupport)
- {
- this.descriptor = descriptor;
- this.keySupport = keySupport;
-
- index = InMemoryIndex.create(keySupport);
- metadata = Metadata.create();
-
- file = descriptor.fileFor(Component.DATA);
- try
- {
- untrackedOut = new FileOutputStreamPlus(file);
- }
- catch (IOException e)
- {
- throw new JournalWriteError(descriptor, file, e);
- }
- trackedOut = TrackedDataOutputPlus.wrap(untrackedOut);
- }
-
- static <K> SegmentWriter<K> create(Descriptor descriptor, KeySupport<K> keySupport)
- {
- return new SegmentWriter<>(descriptor, keySupport);
- }
-
- int write(K key, ByteBuffer record, Set<Integer> hosts)
- {
- int position = position();
- try
- {
- EntrySerializer.write(key, record, hosts, keySupport, trackedOut, descriptor.userVersion);
- index.update(key, position, position() - position);
- metadata.update(hosts);
- }
- catch (IOException e)
- {
- throw new JournalWriteError(descriptor, file, e);
- }
- return position;
- }
-
- int position()
- {
- return Ints.checkedCast(trackedOut.position());
- }
-
- @Override
- public void close()
- {
- try
- {
- untrackedOut.flush();
- untrackedOut.sync();
- untrackedOut.close();
- }
- catch (IOException e)
- {
- throw new JournalWriteError(descriptor, file, e);
- }
-
- try (SyncedOffsets syncedOffsets = SyncedOffsets.active(descriptor))
- {
- syncedOffsets.mark(position(), true);
- }
-
- index.persist(descriptor);
- metadata.persist(descriptor);
-
- untrackedOut = null;
- trackedOut = null;
- }
-}
diff --git a/src/java/org/apache/cassandra/journal/Segments.java b/src/java/org/apache/cassandra/journal/Segments.java
index cc98750fc4..6601d75549 100644
--- a/src/java/org/apache/cassandra/journal/Segments.java
+++ b/src/java/org/apache/cassandra/journal/Segments.java
@@ -17,15 +17,13 @@
*/
package org.apache.cassandra.journal;
-import java.util.ArrayList;
import java.util.Collection;
-import java.util.Comparator;
import java.util.List;
import java.util.function.Predicate;
import accord.utils.Invariants;
+import accord.utils.SortedArrays.SortedArrayList;
import org.agrona.collections.Long2ObjectHashMap;
-import org.apache.cassandra.utils.concurrent.Ref;
import org.apache.cassandra.utils.concurrent.Refs;
/**
@@ -36,6 +34,7 @@ import org.apache.cassandra.utils.concurrent.Refs;
class Segments<K, V>
{
private final Long2ObjectHashMap<Segment<K, V>> segments;
+ private SortedArrayList<Segment<K, V>> sorted;
Segments(Long2ObjectHashMap<Segment<K, V>> segments)
{
@@ -108,10 +107,9 @@ class Segments<K, V>
*/
List<Segment<K, V>> allSorted(boolean asc)
{
- List<Segment<K, V>> segments = new ArrayList<>(this.segments.values());
- if (asc) segments.sort(Comparator.comparing(s -> s.descriptor));
- else segments.sort((o1, o2) -> -o1.descriptor.compareTo(o2.descriptor));
- return segments;
+ if (sorted == null)
+ sorted = SortedArrayList.<Segment<K, V>>copyUnsorted(segments.values(), Segment[]::new);
+ return asc ? sorted : sorted.reverse();
}
void selectActive(long maxTimestamp, Collection<ActiveSegment<K, V>> into)
@@ -132,12 +130,14 @@ class Segments<K, V>
ActiveSegment<K, V> oldestActive()
{
- Segment<K, V> oldest = null;
- for (Segment<K, V> segment : segments.values())
- if (segment.isActive() && (oldest == null || segment.descriptor.timestamp <= oldest.descriptor.timestamp))
- oldest = segment;
-
- return oldest == null ? null : oldest.asActive();
+ List<Segment<K, V>> sorted = allSorted(true);
+ for (int i = 0 ; i < sorted.size() ; ++i)
+ {
+ Segment<K, V> segment = sorted.get(i);
+ if (segment.isActive())
+ return segment.asActive();
+ }
+ return null;
}
Segment<K, V> get(long timestamp)
@@ -158,8 +158,26 @@ class Segments<K, V>
*
* @return a subset of segments with references to them, or {@code null} if failed to grab the refs
*/
- @SuppressWarnings("resource")
ReferencedSegments<K, V> selectAndReference(Predicate<Segment<K, V>> test)
+ {
+ Long2ObjectHashMap<Segment<K, V>> selectedSegments = select(test).segments;
+ Refs<Segment<K, V>> refs = null;
+ if (!selectedSegments.isEmpty())
+ {
+ refs = Refs.tryRef(selectedSegments.values());
+ if (null == refs)
+ return null;
+ }
+ return new ReferencedSegments<>(selectedSegments, refs);
+ }
+
+ /**
+ * Select segments that could potentially have an entry with the specified ids and
+ * attempt to grab references to them all.
+ *
+ * @return a subset of segments with references to them, or {@code null} if failed to grab the refs
+ */
+ Segments<K, V> select(Predicate<Segment<K, V>> test)
{
Long2ObjectHashMap<Segment<K, V>> selectedSegments = null;
for (Segment<K, V> segment : segments.values())
@@ -175,14 +193,7 @@ class Segments<K, V>
if (null == selectedSegments)
selectedSegments = emptyMap();
- Refs<Segment<K, V>> refs = null;
- if (!selectedSegments.isEmpty())
- {
- refs = Refs.tryRef(selectedSegments.values());
- if (null == refs)
- return null;
- }
- return new ReferencedSegments<>(selectedSegments, refs);
+ return new Segments<>(selectedSegments);
}
static class ReferencedSegments<K, V> extends Segments<K, V> implements AutoCloseable
@@ -203,49 +214,6 @@ class Segments<K, V>
}
}
- boolean isFlushed(RecordPointer recordPointer)
- {
- Segment<K, V> segment = segments.get(recordPointer.segment);
- if (null == segment)
- throw new IllegalArgumentException("Can not reference segment " + recordPointer.segment);
- return segment.isFlushed(recordPointer.position);
- }
-
- ReferencedSegment<K, V> selectAndReference(long segmentTimestamp)
- {
- Segment<K, V> segment = segments.get(segmentTimestamp);
- if (null == segment)
- return new ReferencedSegment<>(null, null);
- Ref<Segment<K, V>> ref = segment.tryRef();
- if (null == ref)
- return null;
- return new ReferencedSegment<>(segment, ref);
- }
-
- static class ReferencedSegment<K, V> implements AutoCloseable
- {
- private final Segment<K, V> segment;
- private final Ref<Segment<K, V>> ref;
-
- ReferencedSegment(Segment<K, V> segment, Ref<Segment<K, V>> ref)
- {
- this.segment = segment;
- this.ref = ref;
- }
-
- Segment<K, V> segment()
- {
- return segment;
- }
-
- @Override
- public void close()
- {
- if (null != ref)
- ref.release();
- }
- }
-
private static final Long2ObjectHashMap<?> EMPTY_MAP = new Long2ObjectHashMap<>();
@SuppressWarnings("unchecked")
@@ -256,6 +224,6 @@ class Segments<K, V>
private static <K> Long2ObjectHashMap<K> newMap(int expectedSize)
{
- return new Long2ObjectHashMap<>(0, 0.65f, false);
+ return new Long2ObjectHashMap<>(expectedSize, 0.65f, false);
}
}
diff --git a/src/java/org/apache/cassandra/journal/StaticSegment.java b/src/java/org/apache/cassandra/journal/StaticSegment.java
index bf46ca8ecd..5f83a6eb1b 100644
--- a/src/java/org/apache/cassandra/journal/StaticSegment.java
+++ b/src/java/org/apache/cassandra/journal/StaticSegment.java
@@ -26,14 +26,12 @@ import java.nio.file.StandardOpenOption;
import java.util.ArrayList;
import java.util.Collection;
import java.util.List;
-import java.util.concurrent.locks.LockSupport;
-import accord.utils.Invariants;
import org.agrona.collections.IntHashSet;
-import org.apache.cassandra.io.util.DataInputBuffer;
import org.apache.cassandra.io.util.File;
import org.apache.cassandra.io.util.FileUtils;
import org.apache.cassandra.utils.Closeable;
+import org.apache.cassandra.utils.Throwables;
import org.apache.cassandra.utils.concurrent.Ref;
/**
@@ -45,6 +43,7 @@ import org.apache.cassandra.utils.concurrent.Ref;
public final class StaticSegment<K, V> extends Segment<K, V>
{
final FileChannel channel;
+ final int fsyncLimit;
private final Ref<Segment<K, V>> selfRef;
@@ -53,15 +52,15 @@ public final class StaticSegment<K, V> extends Segment<K, V>
private StaticSegment(Descriptor descriptor,
FileChannel channel,
MappedByteBuffer buffer,
- SyncedOffsets syncedOffsets,
OnDiskIndex<K> index,
Metadata metadata,
KeySupport<K> keySupport)
{
- super(descriptor, syncedOffsets, metadata, keySupport);
+ super(descriptor, metadata, keySupport);
this.index = index;
this.channel = channel;
+ this.fsyncLimit = metadata.fsyncLimit();
this.buffer = buffer;
selfRef = new Ref<>(this, new Tidier<>(descriptor, channel, buffer, index));
@@ -93,21 +92,17 @@ public final class StaticSegment<K, V> extends Segment<K, V>
if (!Component.DATA.existsFor(descriptor))
throw new IllegalArgumentException("Data file for segment " + descriptor + " doesn't exist");
- SyncedOffsets syncedOffsets = Component.SYNCED_OFFSETS.existsFor(descriptor)
- ? SyncedOffsets.load(descriptor)
- : SyncedOffsets.absent();
-
Metadata metadata = Component.METADATA.existsFor(descriptor)
? Metadata.load(descriptor)
- : Metadata.rebuildAndPersist(descriptor, keySupport, syncedOffsets.syncedOffset());
+ : Metadata.rebuildAndPersist(descriptor, keySupport);
OnDiskIndex<K> index = Component.INDEX.existsFor(descriptor)
? OnDiskIndex.open(descriptor, keySupport)
- : OnDiskIndex.rebuildAndPersist(descriptor, keySupport, syncedOffsets.syncedOffset());
+ : OnDiskIndex.rebuildAndPersist(descriptor, keySupport, metadata.fsyncLimit());
try
{
- return internalOpen(descriptor, syncedOffsets, index, metadata, keySupport);
+ return internalOpen(descriptor, index, metadata, keySupport);
}
catch (IOException e)
{
@@ -116,55 +111,27 @@ public final class StaticSegment<K, V> extends Segment<K, V>
}
private static <K, V> StaticSegment<K, V> internalOpen(
- Descriptor descriptor, SyncedOffsets syncedOffsets, OnDiskIndex<K> index, Metadata metadata, KeySupport<K> keySupport)
+ Descriptor descriptor, OnDiskIndex<K> index, Metadata metadata, KeySupport<K> keySupport)
throws IOException
{
File file = descriptor.fileFor(Component.DATA);
FileChannel channel = FileChannel.open(file.toPath(), StandardOpenOption.READ);
MappedByteBuffer buffer = channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size());
- return new StaticSegment<>(descriptor, channel, buffer, syncedOffsets, index, metadata, keySupport);
+ return new StaticSegment<>(descriptor, channel, buffer, index, metadata, keySupport);
}
- @Override
- public void close()
+ public void close(Journal<K, V> journal)
{
- try
- {
- channel.close();
- }
- catch (IOException e)
- {
- throw new RuntimeException("Could not close static segment " + descriptor, e);
- }
-
- release();
+ release(journal);
}
/**
* Waits until this segment is unreferenced, closes it, and deltes all files associated with it.
*/
- void discard()
+ void discard(Journal<K, V> journal)
{
- // TODO: consider moving deletion logic to Tidier instead of busy-looping here
- waitUntilUnreferenced();
- close();
- for (Component component : Component.values())
- {
- File file = descriptor.fileFor(component);
- if (file.exists())
- file.delete();
- }
- }
-
- public void waitUntilUnreferenced()
- {
- while (true)
- {
- if (selfRef.globalCount() == 1)
- return;
-
- LockSupport.parkNanos(100);
- }
+ ((Tidier)selfRef.tidier()).discard = true;
+ close(journal);
}
@Override
@@ -180,23 +147,24 @@ public final class StaticSegment<K, V> extends Segment<K, V>
}
@Override
- void release()
+ public String toString()
{
- selfRef.release();
+ return "StaticSegment{" + descriptor + '}';
}
@Override
- public String toString()
+ public Ref<Segment<K, V>> selfRef()
{
- return "StaticSegment{" + descriptor + '}';
+ return selfRef;
}
- private static final class Tidier<K> implements Tidy
+ private static final class Tidier<K> extends Segment.Tidier implements Tidy
{
private final Descriptor descriptor;
private final FileChannel channel;
private final ByteBuffer buffer;
private final Index<K> index;
+ boolean discard;
Tidier(Descriptor descriptor, FileChannel channel, ByteBuffer buffer, Index<K> index)
{
@@ -207,11 +175,21 @@ public final class StaticSegment<K, V> extends Segment<K, V>
}
@Override
- public void tidy()
+ void onUnreferenced()
{
FileUtils.clean(buffer);
FileUtils.closeQuietly(channel);
index.close();
+ if (discard)
+ {
+ Throwable fail = null;
+ for (Component component : Component.VALUES)
+ {
+ try { descriptor.fileFor(component).deleteIfExists(); }
+ catch (Throwable t) { fail = Throwables.merge(fail, t); }
+ }
+ Throwables.maybeFail(fail);
+ }
}
@Override
@@ -259,13 +237,9 @@ public final class StaticSegment<K, V> extends Segment<K, V>
boolean read(int offset, int size, EntrySerializer.EntryHolder<K> into)
{
ByteBuffer duplicate = buffer.duplicate().position(offset).limit(offset + size);
- try (DataInputBuffer in = new DataInputBuffer(duplicate, false))
+ try
{
- if (!EntrySerializer.tryRead(into, keySupport, duplicate, in, syncedOffsets.syncedOffset(), descriptor.userVersion))
- return false;
-
- Invariants.checkState(in.available() == 0);
- return true;
+ return 0 <= EntrySerializer.tryRead(into, keySupport, duplicate, fsyncLimit, descriptor.userVersion);
}
catch (IOException e)
{
@@ -278,7 +252,7 @@ public final class StaticSegment<K, V> extends Segment<K, V>
*/
void forEachRecord(RecordConsumer<K> consumer)
{
- try (SequentialReader<K> reader = sequentialReader(descriptor, keySupport, syncedOffsets.syncedOffset()))
+ try (SequentialReader<K> reader = sequentialReader(descriptor, keySupport, fsyncLimit))
{
while (reader.advance())
{
@@ -389,13 +363,11 @@ public final class StaticSegment<K, V> extends Segment<K, V>
static final class SequentialReader<K> extends Reader<K>
{
private final int fsyncedLimit; // exclusive
- private final DataInputBuffer in;
SequentialReader(Descriptor descriptor, KeySupport<K> keySupport, int fsyncedLimit)
{
super(descriptor, keySupport);
this.fsyncedLimit = fsyncedLimit;
- in = new DataInputBuffer(buffer, false);
}
@Override
@@ -413,8 +385,10 @@ public final class StaticSegment<K, V> extends Segment<K, V>
offset = buffer.position();
try
{
- if (!EntrySerializer.tryRead(holder, keySupport, buffer, in, fsyncedLimit, descriptor.userVersion))
+ int length = EntrySerializer.tryRead(holder, keySupport, buffer.duplicate(), fsyncedLimit, descriptor.userVersion);
+ if (length < 0)
return eof();
+ buffer.position(offset + length);
}
catch (IOException e)
{
diff --git a/src/java/org/apache/cassandra/journal/SyncedOffsets.java b/src/java/org/apache/cassandra/journal/SyncedOffsets.java
deleted file mode 100644
index 8a89f72ad7..0000000000
--- a/src/java/org/apache/cassandra/journal/SyncedOffsets.java
+++ /dev/null
@@ -1,256 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.cassandra.journal;
-
-import java.io.IOException;
-import java.io.UncheckedIOException;
-import java.nio.file.NoSuchFileException;
-import java.util.zip.CRC32;
-
-import org.apache.cassandra.io.FSWriteError;
-import org.apache.cassandra.io.util.File;
-import org.apache.cassandra.io.util.FileOutputStreamPlus;
-import org.apache.cassandra.io.util.RandomAccessReader;
-import org.apache.cassandra.utils.Closeable;
-import org.apache.cassandra.utils.Crc;
-
-import static org.apache.cassandra.utils.FBUtilities.updateChecksumInt;
-
-/**
- * Keeps track of fsynced limits of a data file. Enables us to treat invalid
- * records that are known to have been fsynced to disk differently from those
- * that aren't.
- * <p/>
- * On disk representation is a sequence of 2-int tuples of {synced offset, CRC32(synced offset)}
- */
-interface SyncedOffsets extends Closeable
-{
- /**
- * @return furthest known synced offset
- */
- int syncedOffset();
-
- /**
- * Record an offset as synced to disk.
- *
- * @param offset the offset into datafile, up to which contents have been fsynced (exclusive)
- */
- void mark(int offset, boolean fsync);
-
- void fsync();
-
- @Override
- default void close()
- {
- }
-
- /**
- * @return a disk-backed synced offset tracker for a new {@link ActiveSegment}
- */
- static Active active(Descriptor descriptor)
- {
- return new Active(descriptor);
- }
-
- /**
- * Load an existing log of synced offsets from disk into an immutable instance.
- */
- static Static load(Descriptor descriptor)
- {
- return Static.load(descriptor);
- }
-
- /**
- * @return a placeholder instance in case this component is missing
- */
- static Absent absent()
- {
- return Absent.INSTANCE;
- }
-
- /**
- * Single-threaded, file-based list of synced offsets.
- */
- final class Active implements SyncedOffsets
- {
- private final Descriptor descriptor;
-
- private final FileOutputStreamPlus output;
- private volatile int syncedOffset;
-
- private Active(Descriptor descriptor)
- {
- this.descriptor = descriptor;
-
- File file = descriptor.fileFor(Component.SYNCED_OFFSETS);
- if (file.exists())
- throw new IllegalArgumentException("Synced offsets file " + file + " already exists");
-
- try
- {
- output = file.newOutputStream(File.WriteMode.OVERWRITE);
- }
- catch (UncheckedIOException | FSWriteError e)
- {
- // extract original cause and throw as JournalWriteError
- throw new JournalWriteError(descriptor, file, e.getCause());
- }
- catch (NoSuchFileException e)
- {
- throw new AssertionError(); // unreachable
- }
- }
-
- @Override
- public int syncedOffset()
- {
- return syncedOffset;
- }
-
- @Override
- public void mark(int offset, boolean fsync)
- {
- if (offset < syncedOffset)
- throw new IllegalArgumentException("offset " + offset + " is smaller than previous mark " + offset);
-
- CRC32 crc = Crc.crc32();
- updateChecksumInt(crc, offset);
-
- try
- {
- output.writeInt(offset);
- output.writeInt((int) crc.getValue());
- output.flush();
- }
- catch (IOException e)
- {
- throw new JournalWriteError(descriptor, Component.SYNCED_OFFSETS, e);
- }
-
- syncedOffset = offset;
- if (fsync) fsync();
- }
-
- public void fsync()
- {
- try
- {
- output.sync();
- }
- catch (IOException e)
- {
- throw new JournalWriteError(descriptor, Component.SYNCED_OFFSETS, e);
- }
- }
-
- @Override
- public void close()
- {
- fsync();
-
- try
- {
- output.close();
- }
- catch (IOException e)
- {
- throw new JournalWriteError(descriptor, Component.SYNCED_OFFSETS, e);
- }
- }
- }
-
- final class Static implements SyncedOffsets
- {
- private final int syncedOffset;
-
- static Static load(Descriptor descriptor)
- {
- File file = descriptor.fileFor(Component.SYNCED_OFFSETS);
- if (!file.exists())
- throw new IllegalArgumentException("Synced offsets file " + file + " doesn't exist");
-
- int syncedOffset = 0;
- try (RandomAccessReader reader = RandomAccessReader.open(file))
- {
- CRC32 crc = Crc.crc32();
- while (reader.bytesRemaining() >= 8)
- {
- int offset = reader.readInt();
- updateChecksumInt(crc, offset);
- int readCrc = reader.readInt();
- if (readCrc != (int) crc.getValue())
- break;
- syncedOffset = offset;
- Crc.initialize(crc);
- }
- }
- catch (Throwable t)
- {
- throw new JournalReadError(descriptor, file, t);
- }
-
- return new Static(syncedOffset);
- }
-
- Static(int offset)
- {
- this.syncedOffset = offset;
- }
-
- @Override
- public int syncedOffset()
- {
- return syncedOffset;
- }
-
- @Override
- public void mark(int offset, boolean fsync)
- {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void fsync()
- {
- throw new UnsupportedOperationException();
- }
- }
-
- final class Absent implements SyncedOffsets
- {
- static final Absent INSTANCE = new Absent();
-
- @Override
- public int syncedOffset()
- {
- return 0;
- }
-
- @Override
- public void mark(int offset, boolean fsync)
- {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public void fsync()
- {
- throw new UnsupportedOperationException();
- }
- }
-}
diff --git a/src/java/org/apache/cassandra/metrics/AccordStateCacheMetrics.java b/src/java/org/apache/cassandra/metrics/AccordCacheMetrics.java
similarity index 94%
rename from src/java/org/apache/cassandra/metrics/AccordStateCacheMetrics.java
rename to src/java/org/apache/cassandra/metrics/AccordCacheMetrics.java
index b00793087e..1250c34751 100644
--- a/src/java/org/apache/cassandra/metrics/AccordStateCacheMetrics.java
+++ b/src/java/org/apache/cassandra/metrics/AccordCacheMetrics.java
@@ -26,7 +26,7 @@ import com.codahale.metrics.Histogram;
import static org.apache.cassandra.metrics.CacheMetrics.TYPE_NAME;
import static org.apache.cassandra.metrics.CassandraMetricsRegistry.Metrics;
-public class AccordStateCacheMetrics extends CacheAccessMetrics
+public class AccordCacheMetrics extends CacheAccessMetrics
{
public static final String OBJECT_SIZE = "ObjectSize";
@@ -36,7 +36,7 @@ public class AccordStateCacheMetrics extends CacheAccessMetrics
private final String scope;
- public AccordStateCacheMetrics(String scope)
+ public AccordCacheMetrics(String scope)
{
super(new DefaultNameFactory(TYPE_NAME, scope));
objectSize = Metrics.histogram(factory.createMetricName(OBJECT_SIZE), false);
diff --git a/src/java/org/apache/cassandra/metrics/AccordMetrics.java b/src/java/org/apache/cassandra/metrics/AccordMetrics.java
index 8064dee66a..8195d46f45 100644
--- a/src/java/org/apache/cassandra/metrics/AccordMetrics.java
+++ b/src/java/org/apache/cassandra/metrics/AccordMetrics.java
@@ -49,6 +49,7 @@ public class AccordMetrics
public static final String PROGRESS_LOG_SIZE = "ProgressLogSize";
public static final String DEPENDENCIES = "Dependencies";
+ public static final String EPHEMERAL = "Ephemeral";
public static final String FAST_PATHS = "FastPaths";
public static final String SLOW_PATHS = "SlowPaths";
public static final String PREEMPTS = "Preempts";
diff --git a/src/java/org/apache/cassandra/schema/TableId.java b/src/java/org/apache/cassandra/schema/TableId.java
index 5cdaf1c2b6..dbdf6fcc8d 100644
--- a/src/java/org/apache/cassandra/schema/TableId.java
+++ b/src/java/org/apache/cassandra/schema/TableId.java
@@ -27,6 +27,7 @@ import java.util.concurrent.TimeUnit;
import javax.annotation.Nullable;
+import org.agrona.collections.Hashing;
import org.apache.cassandra.concurrent.ScheduledExecutors;
import org.apache.cassandra.tcm.ClusterMetadata;
import org.apache.commons.lang3.ArrayUtils;
@@ -41,6 +42,7 @@ import org.apache.cassandra.tcm.serialization.Version;
import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.utils.ObjectSizes;
import org.apache.cassandra.utils.Pair;
+import org.apache.cassandra.utils.UUIDGen;
import static java.nio.charset.StandardCharsets.UTF_8;
import static org.apache.cassandra.utils.TimeUUID.Generator.nextTimeUUID;
@@ -51,18 +53,24 @@ import static org.apache.cassandra.utils.TimeUUID.Generator.nextTimeUUID;
* This is essentially a UUID, but we wrap it as it's used quite a bit in the code and having a nicely named class make
* the code more readable.
*/
-public class TableId implements Comparable<TableId>
+public final class TableId implements Comparable<TableId>
{
public static final long MAGIC = 1956074401491665062L;
public static final long EMPTY_SIZE = ObjectSizes.measureDeep(new UUID(0, 0));
private static final ConcurrentHashMap<TableId, TableId> internCache = new ConcurrentHashMap<>();
- private final UUID id;
+ final long msb, lsb;
private TableId(UUID id)
{
- this.id = id;
+ this(id.getMostSignificantBits(), id.getLeastSignificantBits());
+ }
+
+ private TableId(long msb, long lsb)
+ {
+ this.msb = msb;
+ this.lsb = lsb;
}
public static TableId fromUUID(UUID id)
@@ -70,6 +78,11 @@ public class TableId implements Comparable<TableId>
return new TableId(id);
}
+ public static TableId fromRaw(long msb, long lsb)
+ {
+ return new TableId(msb, lsb);
+ }
+
// TODO: should we be using UUID.randomUUID()?
public static TableId generate()
{
@@ -145,43 +158,46 @@ public class TableId implements Comparable<TableId>
public String toHexString()
{
- return ByteBufferUtil.bytesToHex(ByteBufferUtil.bytes(id));
+ return ByteBufferUtil.bytesToHex(ByteBuffer.wrap(UUIDGen.decompose(msb, lsb)));
}
public UUID asUUID()
{
- return id;
+ return new UUID(msb, lsb);
}
@Override
- public final int hashCode()
+ public int hashCode()
{
- return id.hashCode();
+ return Hashing.hash(msb ^ lsb);
}
@Override
public final boolean equals(Object o)
{
- return this == o || (o instanceof TableId && this.id.equals(((TableId) o).id));
+ if (o == this | o == null) return o == this;
+ if (o.getClass() != TableId.class) return false;
+ TableId that = (TableId) o;
+ return this.msb == that.msb && this.lsb == that.lsb;
}
@Override
public String toString()
{
- return id.toString();
+ return new UUID(msb, lsb).toString();
}
public void serialize(DataOutput out) throws IOException
{
- out.writeLong(id.getMostSignificantBits());
- out.writeLong(id.getLeastSignificantBits());
+ out.writeLong(msb);
+ out.writeLong(lsb);
}
public <V> int serialize(V dst, ValueAccessor<V> accessor, int offset)
{
int position = offset;
- position += accessor.putLong(dst, position, id.getMostSignificantBits());
- position += accessor.putLong(dst, position, id.getLeastSignificantBits());
+ position += accessor.putLong(dst, position, msb);
+ position += accessor.putLong(dst, position, lsb);
return position - offset;
}
@@ -197,12 +213,12 @@ public class TableId implements Comparable<TableId>
public static TableId deserialize(DataInput in) throws IOException
{
- return new TableId(new UUID(in.readLong(), in.readLong()));
+ return new TableId(in.readLong(), in.readLong());
}
public static <V> TableId deserialize(V src, ValueAccessor<V> accessor, int offset)
{
- return new TableId(new UUID(accessor.getLong(src, offset), accessor.getLong(src, offset + TypeSizes.LONG_SIZE)));
+ return new TableId(accessor.getLong(src, offset), accessor.getLong(src, offset + TypeSizes.LONG_SIZE));
}
public TableId intern()
@@ -212,9 +228,10 @@ public class TableId implements Comparable<TableId>
}
@Override
- public int compareTo(TableId o)
+ public int compareTo(TableId that)
{
- return id.compareTo(o.id);
+ int c = Long.compare(this.msb, that.msb);
+ return c != 0 ? c : Long.compare(this.lsb, that.lsb);
}
public static final IVersionedSerializer<TableId> serializer = new IVersionedSerializer<>()
@@ -263,5 +280,4 @@ public class TableId implements Comparable<TableId>
{
ScheduledExecutors.scheduledFastTasks.scheduleSelfRecurring(internCache::clear, 1, TimeUnit.HOURS);
}
-
}
diff --git a/src/java/org/apache/cassandra/service/StorageProxy.java b/src/java/org/apache/cassandra/service/StorageProxy.java
index 4dac4a6d9e..ac2e81e2d4 100644
--- a/src/java/org/apache/cassandra/service/StorageProxy.java
+++ b/src/java/org/apache/cassandra/service/StorageProxy.java
@@ -179,6 +179,7 @@ import static com.google.common.base.Preconditions.checkState;
import static com.google.common.collect.Iterables.concat;
import static java.util.concurrent.TimeUnit.MILLISECONDS;
import static java.util.concurrent.TimeUnit.NANOSECONDS;
+import static org.apache.cassandra.config.DatabaseDescriptor.getAccordEphemeralReadEnabledEnabled;
import static org.apache.cassandra.db.ConsistencyLevel.SERIAL;
import static org.apache.cassandra.metrics.ClientRequestsMetricsHolder.casReadMetrics;
import static org.apache.cassandra.metrics.ClientRequestsMetricsHolder.casWriteMetrics;
@@ -205,6 +206,7 @@ import static org.apache.cassandra.service.consensus.migration.ConsensusMigratio
import static org.apache.cassandra.service.consensus.migration.ConsensusMigrationMutationHelper.splitMutationsIntoAccordAndNormal;
import static org.apache.cassandra.service.consensus.migration.ConsensusRequestRouter.ConsensusRoutingDecision;
import static org.apache.cassandra.service.consensus.migration.ConsensusRequestRouter.getTableMetadata;
+import static org.apache.cassandra.service.consensus.migration.TransactionalMigrationFromMode.none;
import static org.apache.cassandra.service.paxos.Ballot.Flag.GLOBAL;
import static org.apache.cassandra.service.paxos.Ballot.Flag.LOCAL;
import static org.apache.cassandra.service.paxos.BallotGenerator.Global.nextBallot;
@@ -2182,7 +2184,7 @@ public class StorageProxy implements StorageProxyMBean
consistencyLevel = consistencyLevelForAccordRead(cm, group, consistencyLevel);
TxnRead read = TxnRead.createSerialRead(group.queries, consistencyLevel);
Txn.Kind kind = Read;
- if (transactionalMode == TransactionalMode.full && DatabaseDescriptor.getAccordEphemeralReadEnabledEnabled() && group.queries.size() == 1)
+ if (transactionalMode == TransactionalMode.full && getAccordEphemeralReadEnabledEnabled() && group.queries.size() == 1 && group.metadata().params.transactionalMigrationFrom == none)
kind = EphemeralRead;
Txn txn = new Txn.InMemory(kind, read.keys(), read, TxnQuery.ALL, null);
AsyncTxnResult asyncTxnResult = AccordService.instance().coordinateAsync(txn, consistencyLevel, requestTime);
diff --git a/src/java/org/apache/cassandra/service/accord/AccordCache.java b/src/java/org/apache/cassandra/service/accord/AccordCache.java
new file mode 100644
index 0000000000..f504b61739
--- /dev/null
+++ b/src/java/org/apache/cassandra/service/accord/AccordCache.java
@@ -0,0 +1,1290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.service.accord;
+
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.IdentityHashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.concurrent.TimeUnit;
+import java.util.function.BiConsumer;
+import java.util.function.BiFunction;
+import java.util.function.Function;
+import java.util.function.ToLongFunction;
+import java.util.stream.Stream;
+
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import accord.api.RoutingKey;
+import accord.local.Command;
+import accord.local.cfk.CommandsForKey;
+import accord.primitives.Routable;
+import accord.primitives.SaveStatus;
+import accord.primitives.Txn;
+import accord.primitives.TxnId;
+import accord.utils.IntrusiveLinkedList;
+import accord.utils.Invariants;
+import accord.utils.QuadFunction;
+import accord.utils.TriFunction;
+import accord.utils.async.Cancellable;
+import org.agrona.collections.Object2ObjectHashMap;
+import org.apache.cassandra.cache.CacheSize;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.exceptions.UnknownTableException;
+import org.apache.cassandra.io.util.DataInputBuffer;
+import org.apache.cassandra.metrics.AccordCacheMetrics;
+import org.apache.cassandra.metrics.CacheAccessMetrics;
+import org.apache.cassandra.service.accord.AccordCacheEntry.Status;
+import org.apache.cassandra.service.accord.events.CacheEvents;
+import org.apache.cassandra.service.accord.serializers.CommandsForKeySerializer;
+import org.apache.cassandra.utils.NoSpamLogger;
+import org.apache.cassandra.utils.NoSpamLogger.NoSpamLogStatement;
+import org.apache.cassandra.utils.ObjectSizes;
+
+import static accord.utils.Invariants.checkState;
+import static org.apache.cassandra.net.MessagingService.current_version;
+import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.EVICTED;
+import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.LOADED;
+import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.MODIFIED;
+
+/**
+ * Cache for AccordCommand and AccordCommandsForKey, available memory is shared between the two object types.
+ * </p>
+ * Supports dynamic object sizes. After each acquire/free cycle, the cacheable objects size is recomputed to
+ * account for data added/removed during txn processing if it's modified flag is set
+ *
+ * TODO (required): we only iterate over unreferenced entries
+ */
+public class AccordCache implements CacheSize
+{
+ private static final Logger logger = LoggerFactory.getLogger(AccordCache.class);
+ private static final NoSpamLogStatement evictNoEvict = NoSpamLogger.getStatement(logger, "Found and expired {} marked no evict, with age {}, exceeding its expected max age of {}", 1L, TimeUnit.MINUTES);
+
+ // Debug mode to verify that loading from journal + system tables results in
+ // functionally identical (or superceding) command to the one we've just evicted.
+ private static boolean VALIDATE_LOAD_ON_EVICT = false;
+
+ @VisibleForTesting
+ public static void validateLoadOnEvict(boolean value)
+ {
+ VALIDATE_LOAD_ON_EVICT = value;
+ }
+
+ public interface Adapter<K, V, S>
+ {
+ @Nullable V load(AccordCommandStore commandStore, K key);
+ @Nullable Runnable save(AccordCommandStore commandStore, K key, @Nullable V value, @Nullable Object shrunk);
+ // a result of null means we can immediately evict, without saving
+ @Nullable V quickShrink(V value);
+ // a result of null means we cannot shrink, and should save/evict as appropriate
+ @Nullable Object fullShrink(K key, V value);
+ @Nullable V inflate(K key, Object shrunk);
+ long estimateHeapSize(V value);
+ long estimateShrunkHeapSize(Object shrunk);
+ boolean validate(AccordCommandStore commandStore, K key, V value);
+ S safeRef(AccordCacheEntry<K, V> node);
+
+ default AccordCacheEntry<K, V> newEntry(K key, AccordCache.Type<K, V, ?>.Instance owner)
+ {
+ return AccordCacheEntry.createReadyToLoad(key, owner);
+ }
+ }
+
+ static class Stats
+ {
+ long queries;
+ long hits;
+ long misses;
+ }
+
+ public static final class ImmutableStats
+ {
+ public final long queries;
+ public final long hits;
+ public final long misses;
+
+ public ImmutableStats(Stats stats)
+ {
+ queries = stats.queries;
+ hits = stats.hits;
+ misses = stats.misses;
+ }
+ }
+
+ private final List<Type<?, ?, ?>> types = new CopyOnWriteArrayList<>();
+ private final Function<Runnable, Cancellable> saveExecutor;
+ private final AccordCacheEntry.OnSaved onSaved;
+ // TODO (required): monitor this queue and periodically clean up entries, or implement an eviction deadline system
+ private final IntrusiveLinkedList<AccordCacheEntry<?,?>> evictQueue = new IntrusiveLinkedList<>();
+ private final IntrusiveLinkedList<AccordCacheEntry<?,?>> noEvictQueue = new IntrusiveLinkedList<>();
+
+ private int unreferencedBytes;
+ private int unreferenced;
+ private long maxSizeInBytes;
+ private long bytesCached;
+ private int noEvictGeneration;
+ private boolean shrinkingOn = true;
+
+ @VisibleForTesting
+ final AccordCacheMetrics metrics;
+ final Stats stats = new Stats();
+
+ public AccordCache(Function<Runnable, Cancellable> saveExecutor, AccordCacheEntry.OnSaved onSaved, long maxSizeInBytes, AccordCacheMetrics metrics)
+ {
+ this.saveExecutor = saveExecutor;
+ this.onSaved = onSaved;
+ this.maxSizeInBytes = maxSizeInBytes;
+ this.metrics = metrics;
+ }
+
+ @Override
+ public void setCapacity(long sizeInBytes)
+ {
+ maxSizeInBytes = sizeInBytes;
+ maybeShrinkOrEvictSomeNodes();
+ }
+
+ public void setShrinkingOn(boolean shrinkingOn)
+ {
+ this.shrinkingOn = shrinkingOn;
+ }
+
+ @Override
+ public long capacity()
+ {
+ return maxSizeInBytes;
+ }
+
+ /**
+ * Make sure we don't have any items lingering too long in the no evict queue, to avoid cache memory leaks
+ */
+ void processNoEvictQueue()
+ {
+ noEvictGeneration = (noEvictGeneration + 1) & 0xffff;
+ if (noEvictQueue.isEmpty())
+ return;
+
+ Iterator<AccordCacheEntry<?, ?>> iter = noEvictQueue.iterator();
+ int skipCount = 3;
+ while (skipCount > 0 && iter.hasNext())
+ {
+ AccordCacheEntry<?, ?> entry = iter.next();
+ int age = (noEvictGeneration - entry.noEvictGeneration()) & 0xffff;
+ if (age >= entry.noEvictMaxAge())
+ {
+ evictNoEvict.warn(entry, age, entry.noEvictMaxAge());
+ evict(entry, true);
+ }
+ else
+ {
+ --skipCount;
+ }
+ }
+ }
+
+ /*
+ * Roughly respects LRU semantics when evicting. Might consider prioritising keeping MODIFIED nodes around
+ * for longer to maximise the chances of hitting system tables fewer times (or not at all).
+ */
+ private void maybeShrinkOrEvictSomeNodes()
+ {
+ while (bytesCached > maxSizeInBytes && !evictQueue.isEmpty())
+ {
+ AccordCacheEntry<?, ?> node = evictQueue.peek();
+ shrinkOrEvict(node);
+ }
+ }
+
+ @VisibleForTesting
+ private <K, V> void shrinkOrEvict(AccordCacheEntry<K, V> node)
+ {
+ checkState(node.references() == 0);
+
+ if (shrinkingOn && node.tryShrink())
+ {
+ IntrusiveLinkedList<AccordCacheEntry<?,?>> queue;
+ queue = node.isNoEvict() ? noEvictQueue : evictQueue;
+ node.unlink();
+ queue.addLast(node);
+ }
+ else
+ {
+ tryEvict(node);
+ }
+ }
+
+ @VisibleForTesting
+ public <K, V> void tryEvict(AccordCacheEntry<K, V> node)
+ {
+ checkState(node.references() == 0);
+
+ if (node.isNoEvict())
+ {
+ node.unlink();
+ noEvictQueue.addLast(node);
+ return;
+ }
+
+ Status status = node.status();
+ switch (status)
+ {
+ default: throw new IllegalStateException("Unhandled status " + status);
+ case LOADING:
+ node.loading().loading.cancel();
+ case WAITING_TO_LOAD:
+ Invariants.paranoid(node.loadingOrWaiting().waiters == null);
+ case LOADED:
+ node.unlink();
+ evict(node, true);
+ break;
+ case MODIFIED:
+ Type<K, V, ?> parent = node.owner.parent();
+ node.save(saveExecutor, parent.adapter, onSaved);
+ boolean evict = node.status() == LOADED;
+ node.unlink();
+ if (evict) evict(node, true);
+ }
+ }
+
+ private void evict(AccordCacheEntry<?, ?> node, boolean updateUnreferenced)
+ {
+ if (logger.isTraceEnabled())
+ logger.trace("Evicting {}", node);
+
+ checkState(node.isUnqueued());
+
+ if (updateUnreferenced)
+ {
+ unreferencedBytes -= node.sizeOnHeap;
+ --unreferenced;
+ }
+ bytesCached -= node.sizeOnHeap;
+ Type<?, ?, ?>.Instance owner = node.owner;
+ Type<?, ?, ?> parent = owner.parent();
+ parent.bytesCached -= node.sizeOnHeap;
+ --parent.size;
+
+ // TODO (expected): use listeners
+ if (node.status() == LOADED && VALIDATE_LOAD_ON_EVICT)
+ owner.validateLoadEvicted(node);
+
+ AccordCacheEntry<?, ?> self = node.owner.cache.remove(node.key());
+ Invariants.checkState(self.references() == 0);
+ checkState(self == node, "Leaked node detected; was attempting to remove %s but cache had %s", node, self);
+ node.notifyListeners(Listener::onEvict);
+ node.evicted();
+ }
+
+ <P, K, V> Collection<AccordTask<?>> load(BiFunction<P, Runnable, Cancellable> loadExecutor, P param, AccordCacheEntry<K, V> node, AccordCacheEntry.OnLoaded onLoaded)
+ {
+ Type<K, V, ?> parent = node.owner.parent();
+ return node.load(loadExecutor, param, parent.adapter, onLoaded).waiters();
+ }
+
+ <K, V> void loaded(AccordCacheEntry<K, V> node, V value)
+ {
+ node.loaded(value);
+ node.notifyListeners(Listener::onUpdate);
+ }
+
+ <K, V> void failedToLoad(AccordCacheEntry<K, V> node)
+ {
+ Invariants.checkState(node.references() == 0);
+ if (node.isUnqueued())
+ {
+ Invariants.checkState(node.status() == EVICTED);
+ return;
+ }
+ node.unlink();
+ node.failedToLoad();
+ evict(node, true);
+ }
+
+ <K, V> void saved(AccordCacheEntry<K, V> node, Object identity, Throwable fail)
+ {
+ if (node.saved(identity, fail) && node.references() == 0)
+ evictQueue.addFirst(node); // add to front since we have just saved, so we were eligible for eviction
+ }
+
+ public <K, V, S extends AccordSafeState<K, V>> void release(S safeRef, AccordTask<?> owner)
+ {
+ safeRef.global().owner.release(safeRef, owner);
+ }
+
+ public ImmutableStats stats()
+ {
+ return new ImmutableStats(stats);
+ }
+
+ public <K, V, S extends AccordSafeState<K, V>> Type<K, V, S> newType(Class<K> keyClass, Adapter<K, V, S> adapter)
+ {
+ Type<K, V, S> instance = new Type<>(keyClass, adapter);
+ types.add(instance);
+ return instance;
+ }
+
+ public <K, V, S extends AccordSafeState<K, V>> Type<K, V, S> newType(
+ Class<K> keyClass,
+ BiFunction<AccordCommandStore, K, V> loadFunction,
+ QuadFunction<AccordCommandStore, K, V, Object, Runnable> saveFunction,
+ Function<V, V> quickShrink,
+ TriFunction<AccordCommandStore, K, V, Boolean> validateFunction,
+ ToLongFunction<V> heapEstimator,
+ Function<AccordCacheEntry<K, V>, S> safeRefFactory)
+ {
+ return newType(keyClass, loadFunction, saveFunction, quickShrink, (i, j) -> j, (i, j) -> (V)j, validateFunction, heapEstimator, i -> 0, safeRefFactory);
+ }
+
+ public <K, V, S extends AccordSafeState<K, V>> Type<K, V, S> newType(
+ Class<K> keyClass,
+ BiFunction<AccordCommandStore, K, V> loadFunction,
+ QuadFunction<AccordCommandStore, K, V, Object, Runnable> saveFunction,
+ Function<V, V> quickShrink,
+ BiFunction<K, V, Object> fullShrink,
+ BiFunction<K, Object, V> inflate,
+ TriFunction<AccordCommandStore, K, V, Boolean> validateFunction,
+ ToLongFunction<V> heapEstimator,
+ ToLongFunction<Object> shrunkHeapEstimator,
+ Function<AccordCacheEntry<K, V>, S> safeRefFactory)
+ {
+ return newType(keyClass, new FunctionalAdapter<>(loadFunction, saveFunction, quickShrink,
+ fullShrink, inflate,
+ validateFunction, heapEstimator, shrunkHeapEstimator,
+ safeRefFactory, AccordCacheEntry::createReadyToLoad));
+ }
+
+ public Collection<Type<?, ? ,? >> types()
+ {
+ return types;
+ }
+
+ public interface Listener<K, V>
+ {
+ default void onAdd(AccordCacheEntry<K, V> state) {}
+ default void onUpdate(AccordCacheEntry<K, V> state) {}
+ default void onEvict(AccordCacheEntry<K, V> state) {}
+ }
+
+ public class Type<K, V, S extends AccordSafeState<K, V>> implements CacheSize
+ {
+ public class Instance implements Iterable<AccordCacheEntry<K, V>>
+ {
+ final AccordCommandStore commandStore;
+ // TODO (desired): don't need to store key separately as stored in node; ideally use a hash set that allows us to get the current entry
+ private final Map<K, AccordCacheEntry<K, V>> cache = new Object2ObjectHashMap<>();
+ private List<Listener<K, V>> listeners = null;
+
+ public Instance(AccordCommandStore commandStore)
+ {
+ this.commandStore = commandStore;
+ }
+
+ public S acquire(K key)
+ {
+ AccordCacheEntry<K, V> node = acquire(key, false);
+ return adapter.safeRef(node);
+ }
+
+ public S acquireIfLoaded(K key)
+ {
+ AccordCacheEntry<K, V> node = acquire(key, true);
+ if (node == null)
+ return null;
+ return adapter.safeRef(node);
+ }
+
+ public S acquire(AccordCacheEntry<K, V> node)
+ {
+ Invariants.checkState(node.owner == this);
+ acquireExisting(node, false);
+ return adapter.safeRef(node);
+ }
+
+ public void recordPreAcquired(AccordSafeState<K, V> ref)
+ {
+ Invariants.checkState(ref.global().owner == this);
+ incrementCacheHits();
+ }
+
+ private AccordCacheEntry<K, V> acquire(K key, boolean onlyIfLoaded)
+ {
+ incrementCacheQueries();
+ @SuppressWarnings("unchecked")
+ AccordCacheEntry<K, V> node = cache.get(key);
+ return node == null
+ ? acquireAbsent(key, onlyIfLoaded)
+ : acquireExisting(node, onlyIfLoaded);
+ }
+
+ /*
+ * Can only return a LOADING Node (or null)
+ */
+ private AccordCacheEntry<K, V> acquireAbsent(K key, boolean onlyIfLoaded)
+ {
+ incrementCacheMisses();
+ if (onlyIfLoaded)
+ return null;
+ AccordCacheEntry<K, V> node = adapter.newEntry(key, this);
+ node.increment();
+
+ Object prev = cache.put(key, node);
+ node.initSize(parent());
+ Invariants.checkState(prev == null, "%s not absent from cache: %s already present", key, node);
+ ++size;
+ node.notifyListeners(Listener::onAdd);
+ maybeShrinkOrEvictSomeNodes();
+ return node;
+ }
+
+ /*
+ * Can't return EVICTED or INITIALIZED
+ */
+ private AccordCacheEntry<K, V> acquireExisting(AccordCacheEntry<K, V> node, boolean onlyIfLoaded)
+ {
+ boolean isLoaded = node.isLoaded();
+ if (isLoaded)
+ incrementCacheHits();
+ else
+ incrementCacheMisses();
+
+ if (onlyIfLoaded && !isLoaded)
+ return null;
+
+ if (node.increment() == 1)
+ {
+ --unreferenced;
+ unreferencedBytes -= node.sizeOnHeap;
+ node.unlink();
+ }
+
+ return node;
+ }
+
+ public void release(AccordSafeState<K, V> safeRef, AccordTask<?> owner)
+ {
+ K key = safeRef.global().key();
+ logger.trace("Releasing resources for {}: {}", key, safeRef);
+
+ AccordCacheEntry<K, V> node = cache.get(key);
+
+ checkState(!safeRef.invalidated());
+ checkState(safeRef.global() != null, "safeRef node is null for %s", key);
+ checkState(safeRef.global() == node, "safeRef node not in map: %s != %s", safeRef.global(), node);
+ checkState(node.references() > 0, "references (%d) are zero for %s (%s)", node.references(), key, node);
+ checkState(node.isUnqueued());
+
+ boolean evict = false;
+ if (safeRef.hasUpdate())
+ {
+ V update = safeRef.current();
+ if (update != null)
+ update = adapter.quickShrink(update);
+ node.setExclusive(update);
+ if (update == null)
+ {
+ if (node.is(MODIFIED))
+ node.saved();
+ evict = true;
+ }
+ node.notifyListeners(Listener::onUpdate);
+ }
+ else if (node.isLoadingOrWaiting())
+ {
+ node.loadingOrWaiting().remove(owner);
+ }
+ else
+ {
+ evict = node.is(LOADED) && node.isNull();
+ }
+ safeRef.invalidate();
+
+ if (node.decrement() == 0)
+ {
+ if (evict)
+ {
+ evict(node, false);
+ return;
+ }
+
+ ++unreferenced;
+ unreferencedBytes += node.sizeOnHeap;
+ Status status = node.status(); // status() completes
+ switch (status)
+ {
+ default: throw new IllegalStateException("Unhandled status " + status);
+ case WAITING_TO_LOAD:
+ case LOADING:
+ case LOADED:
+ case MODIFIED:
+ logger.trace("Moving {} with status {} to eviction queue", key, status);
+ evictQueue.addLast(node);
+
+ case SAVING:
+ case FAILED_TO_SAVE:
+ break; // can never evict, so no point in adding to eviction queue either
+ }
+ }
+
+ maybeShrinkOrEvictSomeNodes();
+ }
+
+ public Stream<AccordCacheEntry<K, V>> stream()
+ {
+ return cache.values().stream();
+ }
+
+ Type<K, V, S> parent()
+ {
+ return Type.this;
+ }
+
+ @Override
+ public Iterator<AccordCacheEntry<K, V>> iterator()
+ {
+ return stream().iterator();
+ }
+
+ void validateLoadEvicted(AccordCacheEntry<?, ?> node)
+ {
+ @SuppressWarnings("unchecked")
+ AccordCacheEntry<K, V> state = (AccordCacheEntry<K, V>) node;
+ K key = state.key();
+ V evicted = state.tryGetFull();
+ if (evicted == null)
+ {
+ try
+ {
+ Object shrunk = state.tryGetShrunk();
+ if (shrunk != null)
+ evicted = adapter.inflate(key, shrunk);
+ }
+ catch (RuntimeException rte)
+ {
+ if (rte.getCause() instanceof UnknownTableException)
+ return;
+ throw rte;
+ }
+ }
+ if (!adapter.validate(node.owner.commandStore, key, evicted))
+ throw new IllegalStateException("Reloaded value for key " + key + " is not equal to or fuller than evicted value " + evicted);
+ }
+
+ @VisibleForTesting
+ public AccordCacheEntry<K, V> getUnsafe(K key)
+ {
+ return cache.get(key);
+ }
+
+ public Set<K> keySet()
+ {
+ return cache.keySet();
+ }
+
+ @VisibleForTesting
+ public boolean isReferenced(K key)
+ {
+ AccordCacheEntry<K, V> node = cache.get(key);
+ return node != null && node.references() > 0;
+ }
+
+ @VisibleForTesting
+ boolean keyIsReferenced(Object key, Class<? extends AccordSafeState<?, ?>> valClass)
+ {
+ AccordCacheEntry<?, ?> node = cache.get(key);
+ return node != null && node.references() > 0;
+ }
+
+ @VisibleForTesting
+ boolean keyIsCached(Object key, Class<? extends AccordSafeState<?, ?>> valClass)
+ {
+ AccordCacheEntry<?, ?> node = cache.get(key);
+ return node != null;
+ }
+
+ @VisibleForTesting
+ int references(Object key, Class<? extends AccordSafeState<?, ?>> valClass)
+ {
+ AccordCacheEntry<?, ?> node = cache.get(key);
+ return node != null ? node.references() : 0;
+ }
+
+ void notifyListeners(BiConsumer<Listener<K, V>, AccordCacheEntry<K, V>> notify, AccordCacheEntry<K, V> node)
+ {
+ notifyListeners(listeners, notify, node);
+ notifyListeners(typeListeners, notify, node);
+ }
+
+ void notifyListeners(List<Listener<K, V>> listeners, BiConsumer<Listener<K, V>, AccordCacheEntry<K, V>> notify, AccordCacheEntry<K, V> node)
+ {
+ if (listeners != null)
+ {
+ for (int i = 0, size = listeners.size() ; i < size ; ++i)
+ notify.accept(listeners.get(i), node);
+
+ }
+ }
+
+ public void register(Listener<K, V> l)
+ {
+ if (listeners == null)
+ listeners = new ArrayList<>();
+ listeners.add(l);
+ }
+
+ public void unregister(Listener<K, V> l)
+ {
+ if (!tryUnregister(l))
+ throw new AssertionError("Listener was not registered");
+ }
+
+ public boolean tryUnregister(Listener<K, V> l)
+ {
+ if (listeners == null || !listeners.remove(l))
+ return false;
+ if (listeners.isEmpty())
+ listeners = null;
+ return true;
+ }
+
+ }
+
+ private final Class<K> keyClass;
+ private Adapter<K, V, S> adapter;
+ private long bytesCached;
+ private int size;
+
+ @VisibleForTesting
+ final CacheAccessMetrics typeMetrics;
+ private final Stats stats = new Stats();
+ private List<Listener<K, V>> typeListeners = null;
+
+ public Type(
+ Class<K> keyClass,
+ Adapter<K, V, S> adapter)
+ {
+ this.keyClass = keyClass;
+ this.adapter = adapter;
+ this.typeMetrics = metrics.forInstance(keyClass);
+ }
+
+ void updateSize(long newSize, long delta, boolean isUnreferenced, boolean updateHistogram)
+ {
+ // TODO (expected): deprecate this in favour of a histogram snapshot of any point in time
+ bytesCached += delta;
+ AccordCache.this.bytesCached += delta;
+ if (updateHistogram) metrics.objectSize.update(newSize);
+ if (isUnreferenced) AccordCache.this.unreferencedBytes += delta;
+ }
+
+ // can be safely garbage collected if empty
+ Instance newInstance(AccordCommandStore commandStore)
+ {
+ return new Instance(commandStore);
+ }
+
+ private void incrementCacheQueries()
+ {
+ typeMetrics.requests.mark();
+ metrics.requests.mark();
+ stats.queries++;
+ AccordCache.this.stats.queries++;
+ }
+
+ private void incrementCacheHits()
+ {
+ typeMetrics.hits.mark();
+ metrics.hits.mark();
+ stats.hits++;
+ AccordCache.this.stats.hits++;
+ }
+
+ private void incrementCacheMisses()
+ {
+ typeMetrics.misses.mark();
+ metrics.misses.mark();
+ stats.misses++;
+ AccordCache.this.stats.misses++;
+ }
+
+ AccordCache parent()
+ {
+ return AccordCache.this;
+ }
+
+ public Stats stats()
+ {
+ return stats;
+ }
+
+ public ImmutableStats statsSnapshot()
+ {
+ return new ImmutableStats(stats);
+ }
+
+ public Stats globalStats()
+ {
+ return AccordCache.this.stats;
+ }
+
+ @VisibleForTesting
+ public void unsafeSetLoadFunction(BiFunction<AccordCommandStore, K, V> loadFunction)
+ {
+ if (adapter.getClass() != SettableWrapper.class)
+ adapter = new SettableWrapper<>(adapter);
+ ((SettableWrapper<K, V, S>)adapter).load = loadFunction;
+ }
+
+ public BiFunction<AccordCommandStore, K, V> unsafeGetLoadFunction()
+ {
+ if (adapter.getClass() != SettableWrapper.class)
+ adapter = new SettableWrapper<>(adapter);
+ return ((SettableWrapper<K, V, S>)adapter).load;
+ }
+
+ Adapter<K, V, S> adapter()
+ {
+ return adapter;
+ }
+
+ @Override
+ public long capacity()
+ {
+ return AccordCache.this.capacity();
+ }
+
+ @Override
+ public void setCapacity(long capacity)
+ {
+ throw new UnsupportedOperationException("Capacity is shared between all instances. Please set the capacity on the global cache");
+ }
+
+ @Override
+ public int size()
+ {
+ return size;
+ }
+
+ @Override
+ public long weightedSize()
+ {
+ return bytesCached;
+ }
+
+ public long globalAllocated()
+ {
+ return AccordCache.this.bytesCached;
+ }
+
+ public int globalReferencedEntries()
+ {
+ return AccordCache.this.numReferencedEntries();
+ }
+
+ public int globalUnreferencedEntries()
+ {
+ return AccordCache.this.numUnreferencedEntries();
+ }
+
+ public void register(Listener<K, V> l)
+ {
+ if (typeListeners == null)
+ typeListeners = new ArrayList<>();
+ typeListeners.add(l);
+ }
+
+ public void unregister(Listener<K, V> l)
+ {
+ if (typeListeners == null)
+ throw new AssertionError("No listeners exist");
+ if (!typeListeners.remove(l))
+ throw new AssertionError("Listener was not registered");
+ if (typeListeners.isEmpty())
+ typeListeners = null;
+ }
+
+ @Override
+ public String toString()
+ {
+ return "Instance{" +
+ ", keyClass=" + keyClass +
+ '}';
+ }
+ }
+
+ @VisibleForTesting
+ AccordCacheEntry<?, ?> head()
+ {
+ Iterator<AccordCacheEntry<?, ?>> iter = evictQueue.iterator();
+ return iter.hasNext() ? iter.next() : null;
+ }
+
+ @VisibleForTesting
+ AccordCacheEntry<?, ?> tail()
+ {
+ AccordCacheEntry<?,?> last = null;
+ Iterator<AccordCacheEntry<?, ?>> iter = evictQueue.iterator();
+ while (iter.hasNext())
+ last = iter.next();
+ return last;
+ }
+
+ public boolean isEmpty()
+ {
+ return size() == 0;
+ }
+
+ Iterable<AccordCacheEntry<?, ?>> evictionQueue()
+ {
+ return evictQueue::iterator;
+ }
+
+ private int cacheSize()
+ {
+ int size = 0;
+ for (Type<?, ?, ?> type : types)
+ size += type.size();
+ return size;
+ }
+
+ @VisibleForTesting
+ int numReferencedEntries()
+ {
+ return cacheSize() - unreferenced;
+ }
+
+ @VisibleForTesting
+ int numUnreferencedEntries()
+ {
+ return unreferenced;
+ }
+
+ @VisibleForTesting
+ int unreferencedBytes()
+ {
+ return unreferencedBytes;
+ }
+
+ @Override
+ public int size()
+ {
+ return cacheSize();
+ }
+
+ @Override
+ public long weightedSize()
+ {
+ return bytesCached;
+ }
+
+ static <K, V> void registerJfrListener(int shardId, AccordCache.Type<K, V, ?> type, String name)
+ {
+ if (!DatabaseDescriptor.getAccordStateCacheListenerJFREnabled())
+ return;
+
+ type.register(new AccordCache.Listener<>() {
+ private final IdentityHashMap<AccordCacheEntry<?, ?>, CacheEvents.Evict> pendingEvicts = new IdentityHashMap<>();
+
+ @Override
+ public void onAdd(AccordCacheEntry<K, V> state)
+ {
+ CacheEvents.Add add = new CacheEvents.Add();
+ CacheEvents.Evict evict = new CacheEvents.Evict();
+ if (!add.isEnabled())
+ return;
+ add.begin();
+ evict.begin();
+ add.shard = evict.shard = shardId;
+ add.instance = evict.instance = name;
+ add.key = evict.key = state.key().toString();
+ updateMutable(type, state, add);
+ add.commit();
+ pendingEvicts.put(state, evict);
+ }
+
+ @Override
+ public void onEvict(AccordCacheEntry<K, V> state)
+ {
+ CacheEvents.Evict event = pendingEvicts.remove(state);
+ if (event == null) return;
+ updateMutable(type, state, event);
+ event.commit();
+ }
+ });
+ }
+
+ private static void updateMutable(AccordCache.Type<?, ?, ?> type, AccordCacheEntry<?, ?> state, CacheEvents event)
+ {
+ event.status = state.status().name();
+
+ event.lastQueriedEstimatedSizeOnHeap = state.sizeOnHeap();
+
+ event.instanceAllocated = type.weightedSize();
+ AccordCache.Stats stats = type.stats();
+ event.instanceStatsQueries = stats.queries;
+ event.instanceStatsHits = stats.hits;
+ event.instanceStatsMisses = stats.misses;
+
+ event.globalSize = type.size();
+ event.globalReferenced = type.globalReferencedEntries();
+ event.globalUnreferenced = type.globalUnreferencedEntries();
+ event.globalCapacity = type.capacity();
+ event.globalAllocated = type.globalAllocated();
+
+ stats = type.globalStats();
+ event.globalStatsQueries = stats.queries;
+ event.globalStatsHits = stats.hits;
+ event.globalStatsMisses = stats.misses;
+
+ event.update();
+ }
+
+ static class FunctionalAdapter<K, V, S> implements Adapter<K, V, S>
+ {
+ final BiFunction<AccordCommandStore, K, V> load;
+ final QuadFunction<AccordCommandStore, K, V, Object, Runnable> save;
+ final Function<V, V> quickShrink;
+ final BiFunction<K, V, Object> shrink;
+ final BiFunction<K, Object, V> inflate;
+ final TriFunction<AccordCommandStore, K, V, Boolean> validate;
+ final ToLongFunction<V> estimateHeapSize;
+ final ToLongFunction<Object> estimateShrunkHeapSize;
+ final Function<AccordCacheEntry<K, V>, S> newSafeRef;
+ final BiFunction<K, AccordCache.Type<K, V, ?>.Instance, AccordCacheEntry<K, V>> newNode;
+
+ FunctionalAdapter(BiFunction<AccordCommandStore, K, V> load,
+ QuadFunction<AccordCommandStore, K, V, Object, Runnable> save,
+ Function<V, V> quickShrink, BiFunction<K, V, Object> shrink,
+ BiFunction<K, Object, V> inflate,
+ TriFunction<AccordCommandStore, K, V, Boolean> validate,
+ ToLongFunction<V> estimateHeapSize,
+ ToLongFunction<Object> estimateShrunkHeapSize,
+ Function<AccordCacheEntry<K, V>, S> newSafeRef,
+ BiFunction<K, Type<K, V, ?>.Instance, AccordCacheEntry<K, V>> newNode)
+ {
+ this.load = load;
+ this.save = save;
+ this.shrink = shrink;
+ this.quickShrink = quickShrink;
+ this.inflate = inflate;
+ this.validate = validate;
+ this.estimateHeapSize = estimateHeapSize;
+ this.estimateShrunkHeapSize = estimateShrunkHeapSize;
+ this.newSafeRef = newSafeRef;
+ this.newNode = newNode;
+ }
+
+ FunctionalAdapter(Adapter<K, V, S> wrap)
+ {
+ this(wrap::load, wrap::save, wrap::quickShrink, wrap::fullShrink, wrap::inflate, wrap::validate, wrap::estimateHeapSize, wrap::estimateShrunkHeapSize, wrap::safeRef, wrap::newEntry);
+ }
+
+ @Override
+ public V load(AccordCommandStore commandStore, K key)
+ {
+ return load.apply(commandStore, key);
+ }
+
+ @Override
+ public Runnable save(AccordCommandStore commandStore, K key, @Nullable V value, @Nullable Object shrunk)
+ {
+ return save.apply(commandStore, key, value, shrunk);
+ }
+
+ @Override
+ public V quickShrink(V value)
+ {
+ return quickShrink.apply(value);
+ }
+
+ @Override
+ public Object fullShrink(K key, V value)
+ {
+ return shrink.apply(key, value);
+ }
+
+ @Override
+ public V inflate(K key, Object shrunk)
+ {
+ return inflate.apply(key, shrunk);
+ }
+
+ @Override
+ public boolean validate(AccordCommandStore commandStore, K key, V value)
+ {
+ return validate.apply(commandStore, key, value);
+ }
+
+ @Override
+ public long estimateHeapSize(V value)
+ {
+ return estimateHeapSize.applyAsLong(value);
+ }
+
+ @Override
+ public long estimateShrunkHeapSize(Object shrunk)
+ {
+ return estimateShrunkHeapSize.applyAsLong(shrunk);
+ }
+
+ @Override
+ public S safeRef(AccordCacheEntry<K, V> node)
+ {
+ return newSafeRef.apply(node);
+ }
+
+ @Override
+ public AccordCacheEntry<K, V> newEntry(K key, Type<K, V, ?>.Instance owner)
+ {
+ return newNode.apply(key, owner);
+ }
+ }
+
+ static class SettableWrapper<K, V, S> extends FunctionalAdapter<K, V, S>
+ {
+ volatile BiFunction<AccordCommandStore, K, V> load;
+
+ SettableWrapper(Adapter<K, V, S> wrapper)
+ {
+ super(wrapper);
+ this.load = super.load;
+ }
+
+ public static <K, V> Adapter<K, V, ?> loadOnly(BiFunction<AccordCommandStore, K, V> load)
+ {
+ SettableWrapper<K, V, ?> result = new SettableWrapper<>(new NoOpAdapter<>());
+ result.load = load;
+ return result;
+ }
+
+ @Override
+ public V load(AccordCommandStore commandStore, K key)
+ {
+ return load.apply(commandStore, key);
+ }
+ }
+
+ static class NoOpAdapter<K, V, S> implements Adapter<K, V, S>
+ {
+ @Override public V load(AccordCommandStore commandStore, K key) { return null; }
+ @Override public Runnable save(AccordCommandStore commandStore, K key, @Nullable V value, @Nullable Object shrunk) { return null; }
+ @Override public V quickShrink(V value) { return null; }
+ @Override public Object fullShrink(K key, V value) { return null; }
+ @Override public V inflate(K key, Object shrunk) { return null; }
+ @Override public long estimateHeapSize(V value) { return 0; }
+ @Override public long estimateShrunkHeapSize(Object shrunk) { return 0; }
+ @Override public boolean validate(AccordCommandStore commandStore, K key, V value) { return false; }
+ @Override public S safeRef(AccordCacheEntry<K, V> node) { return null; }
+ }
+
+ public static class CommandsForKeyAdapter implements Adapter<RoutingKey, CommandsForKey, AccordSafeCommandsForKey>
+ {
+ public static final CommandsForKeyAdapter CFK_ADAPTER = new CommandsForKeyAdapter();
+ private CommandsForKeyAdapter() {}
+
+ @Override
+ public CommandsForKey load(AccordCommandStore commandStore, RoutingKey key)
+ {
+ return commandStore.loadCommandsForKey(key);
+ }
+
+ @Override
+ public Runnable save(AccordCommandStore commandStore, RoutingKey key, @Nullable CommandsForKey value, @Nullable Object serialized)
+ {
+ return commandStore.saveCommandsForKey(key, value, serialized);
+ }
+
+ @Override
+ public CommandsForKey quickShrink(CommandsForKey value)
+ {
+ return value;
+ }
+
+ @Override
+ public Object fullShrink(RoutingKey key, CommandsForKey value)
+ {
+ if (value.isEmpty())
+ return null;
+
+ return CommandsForKeySerializer.toBytesWithoutKey(value.maximalPrune());
+ }
+
+ @Override
+ public CommandsForKey inflate(RoutingKey key, Object shrunk)
+ {
+ return CommandsForKeySerializer.fromBytes(key, (ByteBuffer)shrunk);
+ }
+
+ @Override
+ public long estimateHeapSize(CommandsForKey value)
+ {
+ return AccordObjectSizes.commandsForKey(value);
+ }
+
+ @Override
+ public long estimateShrunkHeapSize(Object shrunk)
+ {
+ return ObjectSizes.sizeOnHeapOf((ByteBuffer) shrunk);
+ }
+
+ @Override
+ public boolean validate(AccordCommandStore commandStore, RoutingKey key, CommandsForKey value)
+ {
+ return commandStore.validateCommandsForKey(key, value);
+ }
+
+ @Override
+ public AccordSafeCommandsForKey safeRef(AccordCacheEntry<RoutingKey, CommandsForKey> node)
+ {
+ return new AccordSafeCommandsForKey(node);
+ }
+ }
+
+ public static class CommandAdapter implements Adapter<TxnId, Command, AccordSafeCommand>
+ {
+ public static final CommandAdapter COMMAND_ADAPTER = new CommandAdapter();
+ private CommandAdapter() {}
+
+ @Override
+ public Command load(AccordCommandStore commandStore, TxnId txnId)
+ {
+ Invariants.checkState(!txnId.is(Txn.Kind.EphemeralRead));
+ return commandStore.loadCommand(txnId);
+ }
+
+ @Override
+ public Runnable save(AccordCommandStore commandStore, TxnId txnId, @Nullable Command value, @Nullable Object serialized)
+ {
+ if (txnId.is(Routable.Domain.Key))
+ return null;
+
+ if (value == null)
+ {
+ value = inflate(txnId, serialized);
+ if (value == null)
+ return null;
+ }
+
+ return commandStore.appendToKeyspace(txnId, value);
+ }
+
+ @Override
+ public Command quickShrink(Command value)
+ {
+ if (value.saveStatus() == SaveStatus.Uninitialised)
+ return null;
+ if (value.txnId().is(Txn.Kind.EphemeralRead) && value.saveStatus().compareTo(SaveStatus.ReadyToExecute) >= 0)
+ return null; // TODO (expected): should we manage this with the waiting callback? more work, but maybe cleaner/clearer/safer
+ return AccordCommandStore.prepareToCache(value);
+ }
+
+ @Override
+ public Object fullShrink(TxnId txnId, Command value)
+ {
+ if (txnId.is(Txn.Kind.EphemeralRead))
+ Invariants.checkState(value.saveStatus().compareTo(SaveStatus.ReadyToExecute) < 0);
+
+ try
+ {
+ return SavedCommand.asSerializedDiff(null, value, current_version);
+ }
+ catch (IOException e)
+ {
+ logger.warn("Failed to serialize {}", value, e);
+ return null;
+ }
+ }
+
+ @Override
+ public @Nullable Command inflate(TxnId key, Object serialized)
+ {
+ SavedCommand.Builder builder = new SavedCommand.Builder(key);
+ ByteBuffer buffer = (ByteBuffer) serialized;
+ buffer.mark();
+ try (DataInputBuffer buf = new DataInputBuffer(buffer, false))
+ {
+ builder.deserializeNext(buf, current_version);
+ return builder.construct();
+ }
+ catch (UnknownTableException e)
+ {
+ // TODO (required): log, and make sure callers correctly handle null
+ return null;
+ }
+ catch (IOException e)
+ {
+ // TODO (required): test and make sure recover safely from exceptions OR log and return null
+ throw new RuntimeException(e);
+ }
+ finally
+ {
+ buffer.reset();
+ }
+ }
+
+ @Override
+ public long estimateHeapSize(Command value)
+ {
+ return AccordObjectSizes.command(value);
+ }
+
+ @Override
+ public long estimateShrunkHeapSize(Object shrunk)
+ {
+ return ObjectSizes.sizeOnHeapOf((ByteBuffer) shrunk);
+ }
+
+ @Override
+ public boolean validate(AccordCommandStore commandStore, TxnId key, Command value)
+ {
+ return commandStore.validateCommand(key, value);
+ }
+
+ @Override
+ public AccordSafeCommand safeRef(AccordCacheEntry<TxnId, Command> node)
+ {
+ return new AccordSafeCommand(node);
+ }
+
+ @Override
+ public AccordCacheEntry<TxnId, Command> newEntry(TxnId txnId, Type<TxnId, Command, ?>.Instance owner)
+ {
+ AccordCacheEntry<TxnId, Command> node = new AccordCacheEntry<>(txnId, owner);
+ if (txnId.is(Txn.Kind.EphemeralRead))
+ {
+ node.initialize(null);
+ int maxAge = (int)Math.min(0xff, 1 + DatabaseDescriptor.getReadRpcTimeout(TimeUnit.SECONDS));
+ node.markNoEvict(owner.parent().parent().noEvictGeneration, maxAge);
+ }
+ else
+ {
+ node.readyToLoad();
+ }
+ return node;
+ }
+ }
+}
diff --git a/src/java/org/apache/cassandra/service/accord/AccordCacheEntry.java b/src/java/org/apache/cassandra/service/accord/AccordCacheEntry.java
new file mode 100644
index 0000000000..011c1a37ba
--- /dev/null
+++ b/src/java/org/apache/cassandra/service/accord/AccordCacheEntry.java
@@ -0,0 +1,664 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.service.accord;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
+import java.util.function.BiConsumer;
+import java.util.function.BiFunction;
+import java.util.function.Function;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.primitives.Ints;
+
+import accord.utils.ArrayBuffers.BufferList;
+import accord.utils.IntrusiveLinkedListNode;
+import accord.utils.Invariants;
+import accord.utils.async.Cancellable;
+import org.apache.cassandra.service.accord.AccordCache.Adapter;
+import org.apache.cassandra.utils.ObjectSizes;
+
+import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.EVICTED;
+import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.FAILED_TO_LOAD;
+import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.FAILED_TO_SAVE;
+import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.LOADED;
+import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.LOADING;
+import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.MODIFIED;
+import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.SAVING;
+import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.WAITING_TO_LOAD;
+
+/**
+ * Global (per CommandStore) state of a cached entity (Command or CommandsForKey).
+ */
+public class AccordCacheEntry<K, V> extends IntrusiveLinkedListNode
+{
+ public enum Status
+ {
+ UNINITIALIZED,
+ WAITING_TO_LOAD(UNINITIALIZED),
+ LOADING(WAITING_TO_LOAD),
+ /**
+ * Consumers should never see this state
+ */
+ FAILED_TO_LOAD(LOADING),
+
+ LOADED(true, false, UNINITIALIZED, LOADING),
+ MODIFIED(true, false, LOADED),
+ SAVING(true, true, MODIFIED),
+
+ /**
+ * Attempted to save but failed. Shouldn't normally happen unless we have a bug in serialization,
+ * or commit log has been stopped.
+ */
+ FAILED_TO_SAVE(true, true, SAVING),
+
+ UNUSED, // spacing to permit easier bit masks
+
+ EVICTED(WAITING_TO_LOAD, LOADING, LOADED, FAILED_TO_LOAD),
+ ;
+
+ static final Status[] VALUES = values();
+ static
+ {
+ MODIFIED.permittedFrom |= 1 << MODIFIED.ordinal();
+ MODIFIED.permittedFrom |= 1 << SAVING.ordinal();
+ MODIFIED.permittedFrom |= 1 << FAILED_TO_SAVE.ordinal();
+ LOADED.permittedFrom |= 1 << SAVING.ordinal();
+ LOADED.permittedFrom |= 1 << MODIFIED.ordinal();
+ for (Status status : VALUES)
+ {
+ Invariants.checkState((status.ordinal() & IS_LOADED) != 0 == status.loaded);
+ Invariants.checkState(((status.ordinal() & IS_LOADED) != 0 && (status.ordinal() & IS_NESTED) != 0) == status.nested);
+ }
+ }
+
+ final boolean loaded;
+ final boolean nested;
+ int permittedFrom;
+
+ Status(Status ... statuses)
+ {
+ this(false, false, statuses);
+ }
+
+ Status(boolean loaded, boolean nested, Status ... statuses)
+ {
+ this.loaded = loaded;
+ this.nested = nested;
+ for (Status status : statuses)
+ permittedFrom |= 1 << status.ordinal();
+ }
+ }
+
+ static final int STATUS_MASK = 0x0000001F;
+ static final int SHRUNK = 0x00000040;
+ static final int NO_EVICT = 0x00000020;
+ static final int IS_LOADED = 0x4;
+ static final int IS_NESTED = 0x2; // only valid to test if already tested NORMAL
+ static final int IS_LOADING_OR_WAITING_MASK = 0x6; // only valid to test if already tested NORMAL
+ static final int IS_LOADING_OR_WAITING = 0x2; // only valid to test if already tested NORMAL
+ static final long EMPTY_SIZE = ObjectSizes.measure(new AccordCacheEntry<>(null, null));
+
+ private final K key;
+ final AccordCache.Type<K, V, ?>.Instance owner;
+
+ private Object state;
+ private int status;
+ int sizeOnHeap;
+ private volatile int references;
+ private static final AtomicIntegerFieldUpdater<AccordCacheEntry> referencesUpdater = AtomicIntegerFieldUpdater.newUpdater(AccordCacheEntry.class, "references");
+
+ AccordCacheEntry(K key, AccordCache.Type<K, V, ?>.Instance owner)
+ {
+ this.key = key;
+ this.owner = owner;
+ }
+
+ void unlink()
+ {
+ remove();
+ }
+
+ boolean isUnqueued()
+ {
+ return isFree();
+ }
+
+ public K key()
+ {
+ return key;
+ }
+
+ public int references()
+ {
+ return references;
+ }
+
+ public int increment()
+ {
+ return referencesUpdater.incrementAndGet(this);
+ }
+
+ public int decrement()
+ {
+ return referencesUpdater.decrementAndGet(this);
+ }
+
+ boolean isLoaded()
+ {
+ return (status & IS_LOADED) != 0;
+ }
+
+ boolean isNested()
+ {
+ Invariants.checkState(isLoaded());
+ return (status & IS_NESTED) != 0;
+ }
+
+ boolean isShrunk()
+ {
+ return (status & SHRUNK) != 0;
+ }
+
+ public boolean is(Status status)
+ {
+ return (this.status & STATUS_MASK) == status.ordinal();
+ }
+
+ boolean isLoadingOrWaiting()
+ {
+ return (status & IS_LOADING_OR_WAITING_MASK) == IS_LOADING_OR_WAITING;
+ }
+
+ public boolean isComplete()
+ {
+ return !is(LOADING) && !is(SAVING);
+ }
+
+ int noEvictGeneration()
+ {
+ Invariants.checkState(isNoEvict());
+ return (status >>> 8) & 0xffff;
+ }
+
+ int noEvictMaxAge()
+ {
+ Invariants.checkState(isNoEvict());
+ return status >>> 24;
+ }
+
+ boolean isNoEvict()
+ {
+ return (status & NO_EVICT) != 0;
+ }
+
+ int sizeOnHeap()
+ {
+ return sizeOnHeap;
+ }
+
+ void updateSize(AccordCache.Type<K, V, ?> parent)
+ {
+ // TODO (expected): we aren't weighing the keys
+ int newSizeOnHeap = Ints.saturatedCast(EMPTY_SIZE + estimateOnHeapSize(parent.adapter()));
+ parent.updateSize(newSizeOnHeap, newSizeOnHeap - sizeOnHeap, references == 0, true);
+ sizeOnHeap = newSizeOnHeap;
+ }
+
+ void initSize(AccordCache.Type<K, V, ?> parent)
+ {
+ // TODO (expected): we aren't weighing the keys
+ sizeOnHeap = Ints.saturatedCast(EMPTY_SIZE);
+ parent.updateSize(sizeOnHeap, sizeOnHeap, false, false);
+ }
+
+ @Override
+ public String toString()
+ {
+ return "Node{" + status() +
+ ", key=" + key() +
+ ", references=" + references +
+ "}@" + Integer.toHexString(System.identityHashCode(this));
+ }
+
+ public Status status()
+ {
+ return Status.VALUES[(status & STATUS_MASK)];
+ }
+
+ private void setStatus(Status newStatus)
+ {
+ Invariants.checkState((newStatus.permittedFrom & (1 << (status & STATUS_MASK))) != 0, "%s not permitted from %s", newStatus, status());
+ status &= ~STATUS_MASK;
+ status |= newStatus.ordinal();
+ Invariants.checkState(status() == newStatus);
+ }
+
+ public void initialize(V value)
+ {
+ Invariants.checkState(state == null);
+ setStatus(LOADED);
+ state = value;
+ }
+
+ public void readyToLoad()
+ {
+ Invariants.checkState(state == null);
+ setStatus(WAITING_TO_LOAD);
+ state = new WaitingToLoad();
+ }
+
+ public void markNoEvict(int generation, int maxAge)
+ {
+ Invariants.checkState((maxAge & ~0xff) == 0);
+ Invariants.checkState((generation & ~0xffff) == 0);
+ status |= NO_EVICT;
+ status |= generation << 8;
+ status |= maxAge << 24;
+ }
+
+ public LoadingOrWaiting loadingOrWaiting()
+ {
+ return (LoadingOrWaiting)state;
+ }
+
+ void notifyListeners(BiConsumer<AccordCache.Listener<K, V>, AccordCacheEntry<K, V>> notify)
+ {
+ owner.notifyListeners(notify, this);
+ }
+
+ public interface OnLoaded
+ {
+ <K, V> void onLoaded(AccordCacheEntry<K, V> state, V value, Throwable fail);
+
+ static OnLoaded immediate()
+ {
+ return new OnLoaded()
+ {
+ @Override
+ public <K, V> void onLoaded(AccordCacheEntry<K, V> state, V value, Throwable fail)
+ {
+ if (fail == null) state.loaded(value);
+ else state.failedToLoad();
+ }
+ };
+ }
+ }
+
+ public interface OnSaved
+ {
+ <K, V> void onSaved(AccordCacheEntry<K, V> state, Object identity, Throwable fail);
+
+ static OnSaved immediate()
+ {
+ return new OnSaved()
+ {
+ @Override
+ public <K, V> void onSaved(AccordCacheEntry<K, V> state, Object identity, Throwable fail)
+ {
+ state.saved(identity, fail);
+ }
+ };
+ }
+ }
+
+ public <P> Loading load(BiFunction<P, Runnable, Cancellable> loadExecutor, P param, Adapter<K, V, ?> adapter, OnLoaded onLoaded)
+ {
+ Invariants.checkState(is(WAITING_TO_LOAD), "%s", this);
+ Loading loading = ((WaitingToLoad)state).load(loadExecutor.apply(param, () -> {
+ V result;
+ try
+ {
+ result = adapter.load(owner.commandStore, key);
+ }
+ catch (Throwable t)
+ {
+ onLoaded.onLoaded(this, null, t);
+ throw t;
+ }
+ onLoaded.onLoaded(this, result, null);
+ }));
+ setStatus(LOADING);
+ state = loading;
+ return loading;
+ }
+
+ public Loading testLoad()
+ {
+ Invariants.checkState(is(WAITING_TO_LOAD));
+ Loading loading = ((WaitingToLoad)state).load(() -> {});
+ setStatus(LOADING);
+ state = loading;
+ return loading;
+ }
+
+ public Loading loading()
+ {
+ Invariants.checkState(is(LOADING), "%s", this);
+ return (Loading) state;
+ }
+
+ // must own the cache's lock when invoked. this is true of most methods in the class,
+ // but this one is less obvious so named as to draw attention
+ public V getExclusive()
+ {
+ Invariants.checkState(owner == null || owner.commandStore == null || owner.commandStore.executor().isOwningThread());
+ Invariants.checkState(isLoaded(), "%s", this);
+ if (isShrunk())
+ {
+ AccordCache.Type<K, V, ?> parent = owner.parent();
+ inflate(key, parent.adapter());
+ updateSize(parent);
+ }
+
+ return (V)unwrap();
+ }
+
+ private Object unwrap()
+ {
+ return isNested() ? ((Nested)state).state : state;
+ }
+
+ // must own the cache's lock when invoked
+ void setExclusive(V value)
+ {
+ if (value == state)
+ return;
+
+ Saving cancel = is(SAVING) ? ((Saving)state) : null;
+ setStatus(MODIFIED);
+ state = value;
+ updateSize(owner.parent());
+ // TODO (expected): do we want to always cancel in-progress saving?
+ if (cancel != null)
+ cancel.saving.cancel();
+ }
+
+ public void loaded(V value)
+ {
+ setStatus(LOADED);
+ state = value;
+ updateSize(owner.parent());
+ }
+
+ public void testLoaded(V value)
+ {
+ setStatus(LOADED);
+ state = value;
+ }
+
+ public void failedToLoad()
+ {
+ setStatus(FAILED_TO_LOAD);
+ state = null;
+ }
+
+ boolean tryShrink()
+ {
+ if (!isLoaded())
+ return false;
+
+ AccordCache.Type<K, V, ?> parent = owner.parent();
+ if (!tryShrink(key, parent.adapter()))
+ return false;
+ updateSize(parent);
+ return true;
+ }
+
+ V tryGetFull()
+ {
+ return isShrunk() ? null : (V)unwrap();
+ }
+
+ Object tryGetShrunk()
+ {
+ return isShrunk() ? unwrap() : null;
+ }
+
+ boolean isNull()
+ {
+ return state == null;
+ }
+
+ /**
+ * Submits a save runnable to the specified executor. When the runnable
+ * has completed, the state save will have either completed or failed.
+ */
+ @VisibleForTesting
+ void save(Function<Runnable, Cancellable> saveExecutor, Adapter<K, V, ?> adapter, OnSaved onSaved)
+ {
+ V full = isShrunk() ? null : (V)state;
+ Object shrunk = isShrunk() ? state : null;
+ Runnable save = adapter.save(owner.commandStore, key, full, shrunk);
+ if (null == save) // null mutation -> null Runnable -> no change on disk
+ {
+ setStatus(LOADED);
+ }
+ else
+ {
+ setStatus(SAVING);
+ Object identity = new Object();
+ Cancellable saving = saveExecutor.apply(() -> {
+ try
+ {
+ save.run();
+ }
+ catch (Throwable t)
+ {
+ onSaved.onSaved(this, identity, t);
+ throw t;
+ }
+ onSaved.onSaved(this, identity, null);
+ });
+ state = new Saving(saving, identity, state);
+ }
+ }
+
+ boolean saved(Object identity, Throwable fail)
+ {
+ if (!is(SAVING))
+ return false;
+
+ Saving saving = (Saving) state;
+ if (saving.identity != identity)
+ return false;
+
+ if (fail != null)
+ {
+ setStatus(FAILED_TO_SAVE);
+ state = new FailedToSave(fail, ((Saving)state).state);
+ return false;
+ }
+ else
+ {
+ setStatus(LOADED);
+ state = saving.state;
+ return true;
+ }
+ }
+
+ protected void saved()
+ {
+ Invariants.checkState(is(MODIFIED));
+ setStatus(LOADED);
+ }
+
+ public Cancellable saving()
+ {
+ return ((Saving)state).saving;
+ }
+
+ public AccordCacheEntry<K, V> evicted()
+ {
+ setStatus(EVICTED);
+ state = null;
+ return this;
+ }
+
+ public Throwable failure()
+ {
+ return ((FailedToSave)state).cause;
+ }
+
+ private boolean tryShrink(K key, Adapter<K, V, ?> adapter)
+ {
+ Invariants.checkState(!isNested());
+ if (isShrunk() || state == null)
+ return false;
+
+ Object update = adapter.fullShrink(key, (V)state);
+ if (update == null || update == state)
+ return false;
+
+ state = update;
+ status |= SHRUNK;
+ return true;
+ }
+
+ private void inflate(K key, Adapter<K, V, ?> adapter)
+ {
+ Invariants.checkState(isShrunk());
+ if (isNested())
+ {
+ Nested nested = (Nested) state;
+ nested.state = adapter.inflate(key, nested.state);
+ }
+ else
+ {
+ state = adapter.inflate(key, state);
+ }
+ status &= ~SHRUNK;
+ }
+
+ private long estimateOnHeapSize(Adapter<K, V, ?> adapter)
+ {
+ Object current = unwrap();
+ if (current == null) return 0;
+ else if (isShrunk()) return adapter.estimateShrunkHeapSize(current);
+ return adapter.estimateHeapSize((V)current);
+ }
+
+ public static abstract class LoadingOrWaiting
+ {
+ Collection<AccordTask<?>> waiters;
+
+ public LoadingOrWaiting()
+ {
+ }
+
+ public LoadingOrWaiting(Collection<AccordTask<?>> waiters)
+ {
+ this.waiters = waiters;
+ }
+
+ public Collection<AccordTask<?>> waiters()
+ {
+ return waiters != null ? waiters : Collections.emptyList();
+ }
+
+ public BufferList<AccordTask<?>> copyWaiters()
+ {
+ BufferList<AccordTask<?>> list = new BufferList<>();
+ if (waiters != null)
+ list.addAll(waiters);
+ return list;
+ }
+
+ public void add(AccordTask<?> waiter)
+ {
+ if (waiters == null)
+ waiters = new ArrayList<>();
+ waiters.add(waiter);
+ }
+
+ public void remove(AccordTask<?> waiter)
+ {
+ if (waiters != null)
+ {
+ waiters.remove(waiter);
+ if (waiters.isEmpty())
+ waiters = null;
+ }
+ }
+ }
+
+ static class WaitingToLoad extends LoadingOrWaiting
+ {
+ public Loading load(Cancellable loading)
+ {
+ Invariants.paranoid(waiters == null || !waiters.isEmpty());
+ Loading result = new Loading(waiters, loading);
+ waiters = Collections.emptyList();
+ return result;
+ }
+ }
+
+ static class Loading extends LoadingOrWaiting
+ {
+ public final Cancellable loading;
+
+ public Loading(Collection<AccordTask<?>> waiters, Cancellable loading)
+ {
+ super(waiters);
+ this.loading = loading;
+ }
+ }
+
+ static class Nested
+ {
+ Object state;
+ }
+
+ static class Saving extends Nested
+ {
+ final Cancellable saving;
+ final Object identity;
+
+ Saving(Cancellable saving, Object identity, Object state)
+ {
+ this.saving = saving;
+ this.identity = identity;
+ this.state = state;
+ }
+ }
+
+ static class FailedToSave extends Nested
+ {
+ final Throwable cause;
+
+ FailedToSave(Throwable cause, Object state)
+ {
+ this.cause = cause;
+ this.state = state;
+ }
+
+ public Throwable failure()
+ {
+ return cause;
+ }
+ }
+
+ public static <K, V> AccordCacheEntry<K, V> createReadyToLoad(K key, AccordCache.Type<K, V, ?>.Instance owner)
+ {
+ AccordCacheEntry<K, V> node = new AccordCacheEntry<>(key, owner);
+ node.readyToLoad();
+ return node;
+ }
+}
diff --git a/src/java/org/apache/cassandra/service/accord/AccordCachingState.java b/src/java/org/apache/cassandra/service/accord/AccordCachingState.java
deleted file mode 100644
index 52d6b7c090..0000000000
--- a/src/java/org/apache/cassandra/service/accord/AccordCachingState.java
+++ /dev/null
@@ -1,651 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.cassandra.service.accord;
-
-import java.util.concurrent.Callable;
-import java.util.function.Function;
-import java.util.function.ToLongFunction;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.primitives.Ints;
-
-import accord.utils.IntrusiveLinkedListNode;
-import accord.utils.Invariants;
-import accord.utils.async.AsyncChain;
-import accord.utils.async.AsyncResults.RunnableResult;
-import org.apache.cassandra.concurrent.ExecutorPlus;
-import org.apache.cassandra.utils.ObjectSizes;
-
-import static java.lang.String.format;
-import static org.apache.cassandra.service.accord.AccordCachingState.Status.EVICTED;
-import static org.apache.cassandra.service.accord.AccordCachingState.Status.FAILED_TO_LOAD;
-import static org.apache.cassandra.service.accord.AccordCachingState.Status.FAILED_TO_SAVE;
-import static org.apache.cassandra.service.accord.AccordCachingState.Status.LOADED;
-import static org.apache.cassandra.service.accord.AccordCachingState.Status.LOADING;
-import static org.apache.cassandra.service.accord.AccordCachingState.Status.MODIFIED;
-import static org.apache.cassandra.service.accord.AccordCachingState.Status.SAVING;
-import static org.apache.cassandra.service.accord.AccordCachingState.Status.UNINITIALIZED;
-
-/**
- * Global (per CommandStore) state of a cached entity (Command or CommandsForKey).
- */
-public class AccordCachingState<K, V> extends IntrusiveLinkedListNode
-{
- static final long EMPTY_SIZE = ObjectSizes.measure(new AccordCachingState<>(null, 0, null));
-
- public interface Factory<K, V>
- {
- AccordCachingState<K, V> create(K key, int index);
- }
-
- static <K, V> Factory<K, V> defaultFactory()
- {
- return AccordCachingState::new;
- }
-
- private final K key;
- private State<K, V> state;
-
- int references = 0;
- int lastQueriedEstimatedSizeOnHeap = 0;
- final int index;
- private boolean shouldUpdateSize;
-
- AccordCachingState(K key, int index)
- {
- this.key = key;
- Invariants.checkArgument(index >= 0);
- this.index = index;
- //noinspection unchecked
- this.state = (State<K, V>) Uninitialized.instance;
- }
-
- private AccordCachingState(K key, int index, State<K, V> state)
- {
- this.key = key;
- this.index = index;
- this.state = state;
- }
-
- void unlink()
- {
- remove();
- }
-
- boolean isLinked()
- {
- return !isFree();
- }
-
- public K key()
- {
- return key;
- }
-
- public int referenceCount()
- {
- return references;
- }
-
- boolean isLoaded()
- {
- return status().isLoaded();
- }
-
- public boolean isComplete()
- {
- return status().isComplete();
- }
-
- int lastQueriedEstimatedSizeOnHeap()
- {
- return lastQueriedEstimatedSizeOnHeap;
- }
-
- int estimatedSizeOnHeap(ToLongFunction<V> estimator)
- {
- shouldUpdateSize = false; // TODO (expected): probably not the safest place to clear need to compute size
- return lastQueriedEstimatedSizeOnHeap = Ints.checkedCast(EMPTY_SIZE + estimateStateOnHeapSize(estimator));
- }
-
- long estimatedSizeOnHeapDelta(ToLongFunction<V> estimator)
- {
- long prevSize = lastQueriedEstimatedSizeOnHeap;
- return estimatedSizeOnHeap(estimator) - prevSize;
- }
-
- boolean shouldUpdateSize()
- {
- return shouldUpdateSize;
- }
-
- @Override
- public String toString()
- {
- return "Node{" + state.status() +
- ", key=" + key() +
- ", references=" + references +
- "}@" + Integer.toHexString(System.identityHashCode(this));
- }
-
- public Status status()
- {
- return complete().status();
- }
-
- State<K, V> complete()
- {
- return state.isCompleteable() ? state(state.complete()) : state;
- }
-
- /**
- * Submits a load runnable to the specified executor. When the runnable
- * has completed, the state load will have either completed or failed.
- */
- public AsyncChain<V> load(ExecutorPlus executor, Function<K, V> loadFunction)
- {
- Loading<K, V> loading = state.load(key, loadFunction);
- executor.submit(loading);
- state(loading);
- return loading;
- }
-
- public void initialize(V value)
- {
- state(state.initialize(value));
- }
-
- protected State<K, V> state(State<K, V> next)
- {
- State<K, V> prev = state;
- if (prev != next) // TODO (expected): we change state to transition the cache state machine but often keep payload the same - so shouldn't recompute
- shouldUpdateSize = true;
- return state = next;
- }
-
- @VisibleForTesting
- protected State<K, V> state()
- {
- return state;
- }
-
- public AsyncChain<V> loading()
- {
- // do *not* attempt to complete, to prevent races where the caller found a pending load, attempts
- // to register a callback, but gets an exception because the load completed in the meantime
- return state.loading();
- }
-
- public V get()
- {
- return complete().get();
- }
-
- public void set(V value)
- {
- shouldUpdateSize = true;
- state(complete().set(value));
- }
-
- /**
- * Submits a save runnable to the specified executor. When the runnable
- * has completed, the state save will have either completed or failed.
- */
- @VisibleForTesting
- public void save(ExecutorPlus executor, Function<?, Runnable> saveFunction)
- {
- @SuppressWarnings("unchecked")
- State<K, V> savingOrLoaded = state.save((Function<V, Runnable>) saveFunction);
- if (savingOrLoaded.status() == SAVING)
- executor.submit(savingOrLoaded.saving());
- state(savingOrLoaded);
- }
-
- public AsyncChain<Void> saving()
- {
- // do *not* attempt to complete, to prevent races where the caller found a pending save, attempts
- // to register a callback, but gets an exception because the save completed in the meantime
- return state.saving();
- }
-
- public AccordCachingState<K, V> reset()
- {
- state(state.reset());
- return this;
- }
-
- public Throwable failure()
- {
- return complete().failure();
- }
-
- public void markEvicted()
- {
- state(complete().evict());
- lastQueriedEstimatedSizeOnHeap = 0;
- shouldUpdateSize = false;
- }
-
- long estimateStateOnHeapSize(ToLongFunction<V> estimateFunction)
- {
- return state.estimateOnHeapSize(estimateFunction);
- }
-
- public enum Status
- {
- UNINITIALIZED,
- LOADING,
- LOADED,
- FAILED_TO_LOAD,
- MODIFIED,
- SAVING,
-
- /**
- * Attempted to save but failed. Shouldn't normally happen unless we have a bug in serialization,
- * or commit log has been stopped.
- */
- FAILED_TO_SAVE,
-
- /**
- * Entry has been successfully evicted, but there were transient listeners present, so we kept the
- * Node around (transient listeners must survive cache eviction).
- */
- EVICTED,
- ;
-
- boolean isLoaded()
- {
- return this == LOADED || this == MODIFIED || this == FAILED_TO_SAVE;
- }
-
- boolean isComplete()
- {
- return !(this == LOADING || this == SAVING);
- }
- }
-
- interface State<K, V>
- {
- Status status();
-
- default boolean isCompleteable()
- {
- return false;
- }
-
- default State<K, V> complete()
- {
- throw illegalState(this, "complete()");
- }
-
- default Loading<K, V> load(K key, Function<K, V> loadFunction)
- {
- throw illegalState(this, "load(key, loadFunction)");
- }
-
- default Loaded<K, V> initialize(V value)
- {
- throw illegalState(this, "initialize(value)");
- }
-
- default RunnableResult<V> loading()
- {
- throw illegalState(this, "loading()");
- }
-
- default V get()
- {
- throw illegalState(this, "get()");
- }
-
- default State<K, V> set(V value)
- {
- throw illegalState(this, "set(value)");
- }
-
- default State<K, V> save(Function<V, Runnable> saveFunction)
- {
- throw illegalState(this, "save(saveFunction)");
- }
-
- default RunnableResult<Void> saving()
- {
- throw illegalState(this, "saving()");
- }
-
- default Throwable failure()
- {
- throw illegalState(this, "failure()");
- }
-
- default Uninitialized<K, V> reset()
- {
- throw illegalState(this, "reset()");
- }
-
- default Evicted<K, V> evict()
- {
- throw illegalState(this, "evict()");
- }
-
- default long estimateOnHeapSize(ToLongFunction<V> estimateFunction)
- {
- return 0;
- }
- }
-
- private static IllegalStateException illegalState(State<?, ?> state, String method)
- {
- return new IllegalStateException(format("%s invoked on %s", method, state.status()));
- }
-
- static class Uninitialized<K, V> implements State<K, V>
- {
- static final Uninitialized<?, ?> instance = new Uninitialized<>();
-
- @SuppressWarnings("unchecked")
- static <K, V> Uninitialized<K, V> instance()
- {
- return (Uninitialized<K, V>) instance;
- }
-
- @Override
- public Status status()
- {
- return UNINITIALIZED;
- }
-
- @Override
- public Loading<K, V> load(K key, Function<K, V> loadFunction)
- {
- return new Loading<>(() -> loadFunction.apply(key));
- }
-
- public Loaded<K, V> initialize(V value)
- {
- return new Loaded<>(value);
- }
-
- @Override
- public Evicted<K, V> evict()
- {
- return Evicted.instance();
- }
- }
-
- static class Loading<K, V> extends RunnableResult<V> implements State<K, V>
- {
- Loading(Callable<V> callable)
- {
- super(callable);
- }
-
- @Override
- public Status status()
- {
- return LOADING;
- }
-
- @Override
- public boolean isCompleteable()
- {
- return isDone();
- }
-
- @Override
- public State<K, V> complete()
- {
- if (!isDone()) return this;
- else if (isSuccess()) return new Loaded<>(result());
- else return new FailedToLoad<>(failure());
- }
-
- @Override
- public RunnableResult<V> loading()
- {
- return this;
- }
- }
-
- static class Loaded<K, V> implements State<K, V>
- {
- final V original;
-
- Loaded(V original)
- {
- this.original = original;
- }
-
- @Override
- public Status status()
- {
- return LOADED;
- }
-
- @Override
- public V get()
- {
- return original;
- }
-
- @Override
- public State<K, V> set(V value)
- {
- return value == original ? this : new Modified<>(value);
- }
-
- @Override
- public Evicted<K, V> evict()
- {
- return Evicted.instance();
- }
-
- @Override
- public long estimateOnHeapSize(ToLongFunction<V> estimateFunction)
- {
- return null == original ? 0 : estimateFunction.applyAsLong(original);
- }
- }
-
- static class FailedToLoad<K, V> implements State<K, V>
- {
- final Throwable cause;
-
- FailedToLoad(Throwable cause)
- {
- this.cause = cause;
- }
-
- @Override
- public Status status()
- {
- return FAILED_TO_LOAD;
- }
-
- @Override
- public Throwable failure()
- {
- return cause;
- }
-
- @Override
- public Uninitialized<K, V> reset()
- {
- return Uninitialized.instance();
- }
-
- @Override
- public Evicted<K, V> evict()
- {
- return Evicted.instance();
- }
- }
-
- static class Modified<K, V> implements State<K, V>
- {
- V current;
-
- Modified(V current)
- {
- this.current = current;
- }
-
- @Override
- public Status status()
- {
- return MODIFIED;
- }
-
- @Override
- public V get()
- {
- return current;
- }
-
- @Override
- public State<K, V> set(V value)
- {
- current = value;
- return this;
- }
-
- @Override
- public State<K, V> save(Function<V, Runnable> saveFunction)
- {
- Runnable runnable = saveFunction.apply(current);
- if (null == runnable) // null mutation -> null Runnable -> no change on disk
- return new Loaded<>(current);
- else
- return new Saving<>(current, runnable);
- }
-
- @Override
- public long estimateOnHeapSize(ToLongFunction<V> estimateFunction)
- {
- return (null == current ? 0 : estimateFunction.applyAsLong(current));
- }
- }
-
- static class Saving<K, V> extends RunnableResult<Void> implements State<K, V>
- {
- V current;
-
- Saving(V current, Runnable saveRunnable)
- {
- this(current, () -> { saveRunnable.run(); return null; });
- }
-
- Saving(V current, Callable<Void> saveCallable)
- {
- super(saveCallable);
- this.current = current;
- }
-
- @Override
- public Status status()
- {
- return SAVING;
- }
-
- @Override
- public boolean isCompleteable()
- {
- return isDone();
- }
-
- @Override
- public V get()
- {
- return current;
- }
-
- @Override
- public State<K, V> complete()
- {
- if (!isDone()) return this;
- else if (isSuccess()) return new Loaded<>(current);
- else return new FailedToSave<>(current, failure());
- }
-
- @Override
- public RunnableResult<Void> saving()
- {
- return this;
- }
- }
-
- static class FailedToSave<K, V> implements State<K, V>
- {
- V current;
- final Throwable cause;
-
- FailedToSave(V current, Throwable cause)
- {
- this.current = current;
- this.cause = cause;
- }
-
- @Override
- public Status status()
- {
- return FAILED_TO_SAVE;
- }
-
- @Override
- public V get()
- {
- return current;
- }
-
- @Override
- public State<K, V> set(V value)
- {
- current = value;
- return this;
- }
-
- @Override
- public Throwable failure()
- {
- return cause;
- }
- }
-
- static class Evicted<K, V> implements State<K, V>
- {
- static final Evicted<?, ?> instance = new Evicted<>();
-
- @SuppressWarnings("unchecked")
- static <K, V> Evicted<K, V> instance()
- {
- return (Evicted<K, V>) instance;
- }
-
- @Override
- public Status status()
- {
- return EVICTED;
- }
-
- @Override
- public Uninitialized<K, V> reset()
- {
- return Uninitialized.instance();
- }
- }
-}
diff --git a/src/java/org/apache/cassandra/service/accord/AccordCommandStore.java b/src/java/org/apache/cassandra/service/accord/AccordCommandStore.java
index c7ee8aa6cd..4394fbb29a 100644
--- a/src/java/org/apache/cassandra/service/accord/AccordCommandStore.java
+++ b/src/java/org/apache/cassandra/service/accord/AccordCommandStore.java
@@ -18,24 +18,20 @@
package org.apache.cassandra.service.accord;
-import java.util.IdentityHashMap;
import java.util.List;
-import java.util.Map;
import java.util.NavigableMap;
import java.util.Objects;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Future;
+import java.util.concurrent.Executor;
import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.Lock;
import java.util.function.Consumer;
import java.util.function.Function;
import java.util.function.IntFunction;
import javax.annotation.Nullable;
import com.google.common.annotations.VisibleForTesting;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
import accord.api.Agent;
import accord.api.DataStore;
@@ -55,6 +51,7 @@ import accord.local.RedundantBefore;
import accord.local.SafeCommand;
import accord.local.SafeCommandStore;
import accord.local.cfk.CommandsForKey;
+import accord.primitives.PartialTxn;
import accord.primitives.Participants;
import accord.primitives.RangeDeps;
import accord.primitives.Ranges;
@@ -66,22 +63,17 @@ import accord.primitives.TxnId;
import accord.utils.Invariants;
import accord.utils.async.AsyncChain;
import accord.utils.async.AsyncChains;
-import org.apache.cassandra.cache.CacheSize;
-import org.apache.cassandra.concurrent.SequentialExecutorPlus;
-import org.apache.cassandra.config.CassandraRelevantProperties;
-import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.Mutation;
import org.apache.cassandra.service.accord.SavedCommand.MinimalCommand;
import org.apache.cassandra.service.accord.api.AccordRoutingKey.TokenKey;
-import org.apache.cassandra.service.accord.async.AsyncOperation;
-import org.apache.cassandra.service.accord.events.CacheEvents;
+import org.apache.cassandra.service.accord.txn.TxnRead;
import org.apache.cassandra.utils.Clock;
import org.apache.cassandra.utils.concurrent.AsyncPromise;
import org.apache.cassandra.utils.concurrent.Promise;
import org.apache.cassandra.utils.concurrent.UncheckedInterruptedException;
-import static accord.local.KeyHistory.COMMANDS;
import static accord.primitives.SaveStatus.Applying;
+import static accord.local.KeyHistory.SYNC;
import static accord.primitives.Status.Committed;
import static accord.primitives.Status.Invalidated;
import static accord.primitives.Status.Truncated;
@@ -90,87 +82,70 @@ import static org.apache.cassandra.service.accord.SavedCommand.Load.MINIMAL;
public class AccordCommandStore extends CommandStore
{
- private static final Logger logger = LoggerFactory.getLogger(AccordCommandStore.class);
- private static final boolean CHECK_THREADS = CassandraRelevantProperties.TEST_ACCORD_STORE_THREAD_CHECKS_ENABLED.getBoolean();
-
- public final String loggingId;
- private final IJournal journal;
-
- private final CommandStoreExecutor executor;
- private final AccordStateCache.Instance<TxnId, Command, AccordSafeCommand> commandCache;
- private final AccordStateCache.Instance<RoutingKey, TimestampsForKey, AccordSafeTimestampsForKey> timestampsForKeyCache;
- private final AccordStateCache.Instance<RoutingKey, CommandsForKey, AccordSafeCommandsForKey> commandsForKeyCache;
- private AsyncOperation<?> currentOperation = null;
- private AccordSafeCommandStore current = null;
- private long lastSystemTimestampMicros = Long.MIN_VALUE;
- private final CommandsForRangesLoader commandsForRangesLoader;
-
- private static <K, V> void registerJfrListener(int id, AccordStateCache.Instance<K, V, ?> instance, String name)
+ // TODO (required): track this via a PhantomReference, so that if we remove a CommandStore without clearing the caches we can be sure to release them
+ public static class Caches
{
- if (!DatabaseDescriptor.getAccordStateCacheListenerJFREnabled())
- return;
- instance.register(new AccordStateCache.Listener<>() {
- private final IdentityHashMap<AccordCachingState<?, ?>, CacheEvents.Evict> pendingEvicts = new IdentityHashMap<>();
+ private final AccordCache global;
+ private final AccordCache.Type<TxnId, Command, AccordSafeCommand>.Instance commands;
+ private final AccordCache.Type<RoutingKey, TimestampsForKey, AccordSafeTimestampsForKey>.Instance timestampsForKeys;
+ private final AccordCache.Type<RoutingKey, CommandsForKey, AccordSafeCommandsForKey>.Instance commandsForKeys;
- @Override
- public void onAdd(AccordCachingState<K, V> state)
- {
- CacheEvents.Add add = new CacheEvents.Add();
- CacheEvents.Evict evict = new CacheEvents.Evict();
- if (!add.isEnabled())
- return;
- add.begin();
- evict.begin();
- add.store = evict.store = id;
- add.instance = evict.instance = name;
- add.key = evict.key = state.key().toString();
- updateMutable(instance, state, add);
- add.commit();
- pendingEvicts.put(state, evict);
- }
+ Caches(AccordCache global, AccordCache.Type<TxnId, Command, AccordSafeCommand>.Instance commandCache, AccordCache.Type<RoutingKey, TimestampsForKey, AccordSafeTimestampsForKey>.Instance timestampsForKeyCache, AccordCache.Type<RoutingKey, CommandsForKey, AccordSafeCommandsForKey>.Instance commandsForKeyCache)
+ {
+ this.global = global;
+ this.commands = commandCache;
+ this.timestampsForKeys = timestampsForKeyCache;
+ this.commandsForKeys = commandsForKeyCache;
+ }
- @Override
- public void onRelease(AccordCachingState<K, V> state)
- {
+ public final AccordCache global()
+ {
+ return global;
+ }
- }
+ public final AccordCache.Type<TxnId, Command, AccordSafeCommand>.Instance commands()
+ {
+ return commands;
+ }
- @Override
- public void onEvict(AccordCachingState<K, V> state)
- {
- CacheEvents.Evict event = pendingEvicts.remove(state);
- if (event == null) return;
- updateMutable(instance, state, event);
- event.commit();
- }
- });
+ public final AccordCache.Type<RoutingKey, TimestampsForKey, AccordSafeTimestampsForKey>.Instance timestampsForKeys()
+ {
+ return timestampsForKeys;
+ }
+
+ public final AccordCache.Type<RoutingKey, CommandsForKey, AccordSafeCommandsForKey>.Instance commandsForKeys()
+ {
+ return commandsForKeys;
+ }
}
- private static void updateMutable(AccordStateCache.Instance<?, ?, ?> instance, AccordCachingState<?, ?> state, CacheEvents event)
+ public static final class ExclusiveCaches extends Caches implements AutoCloseable
{
- event.status = state.state().status().name();
-
- event.lastQueriedEstimatedSizeOnHeap = state.lastQueriedEstimatedSizeOnHeap();
+ private final Lock lock;
- event.instanceAllocated = instance.weightedSize();
- AccordStateCache.Stats stats = instance.stats();
- event.instanceStatsQueries = stats.queries;
- event.instanceStatsHits = stats.hits;
- event.instanceStatsMisses = stats.misses;
+ public ExclusiveCaches(Lock lock, AccordCache global, AccordCache.Type<TxnId, Command, AccordSafeCommand>.Instance commands, AccordCache.Type<RoutingKey, TimestampsForKey, AccordSafeTimestampsForKey>.Instance timestampsForKeys, AccordCache.Type<RoutingKey, CommandsForKey, AccordSafeCommandsForKey>.Instance commandsForKeys)
+ {
+ super(global, commands, timestampsForKeys, commandsForKeys);
+ this.lock = lock;
+ }
- event.globalSize = instance.size();
- event.globalReferenced = instance.globalReferencedEntries();
- event.globalUnreferenced = instance.globalUnreferencedEntries();
- event.globalCapacity = instance.capacity();
- event.globalAllocated = instance.globalAllocated();
+ @Override
+ public void close()
+ {
+ lock.unlock();
+ }
+ }
- stats = instance.globalStats();
- event.globalStatsQueries = stats.queries;
- event.globalStatsHits = stats.hits;
- event.globalStatsMisses = stats.misses;
+ public final String loggingId;
+ private final IJournal journal;
+ private final AccordExecutor executor;
+ private final Executor taskExecutor;
+ private final ExclusiveCaches caches;
+ private long lastSystemTimestampMicros = Long.MIN_VALUE;
+ private final CommandsForRangesLoader commandsForRangesLoader;
- event.update();
- }
+ private AccordSafeCommandStore current;
+ private Thread currentThread;
public AccordCommandStore(int id,
NodeCommandStoreService node,
@@ -180,53 +155,33 @@ public class AccordCommandStore extends CommandStore
LocalListeners.Factory listenerFactory,
EpochUpdateHolder epochUpdateHolder,
IJournal journal,
- CommandStoreExecutor commandStoreExecutor)
+ AccordExecutor executor)
{
super(id, node, agent, dataStore, progressLogFactory, listenerFactory, epochUpdateHolder);
- this.journal = journal;
loggingId = String.format("[%s]", id);
- executor = commandStoreExecutor;
- AccordStateCache stateCache = executor.stateCache;
- commandCache =
- stateCache.instance(TxnId.class,
- AccordSafeCommand.class,
- AccordSafeCommand.safeRefFactory(),
- this::loadCommand,
- this::appendToKeyspace,
- this::validateCommand,
- AccordObjectSizes::command);
- registerJfrListener(id, commandCache, "Command");
- timestampsForKeyCache =
- stateCache.instance(RoutingKey.class,
- AccordSafeTimestampsForKey.class,
- AccordSafeTimestampsForKey::new,
- this::loadTimestampsForKey,
- this::saveTimestampsForKey,
- this::validateTimestampsForKey,
- AccordObjectSizes::timestampsForKey);
- registerJfrListener(id, timestampsForKeyCache, "TimestampsForKey");
- commandsForKeyCache =
- stateCache.instance(RoutingKey.class,
- AccordSafeCommandsForKey.class,
- AccordSafeCommandsForKey::new,
- this::loadCommandsForKey,
- this::saveCommandsForKey,
- this::validateCommandsForKey,
- AccordObjectSizes::commandsForKey,
- AccordCachingState::new);
- registerJfrListener(id, commandsForKeyCache, "CommandsForKey");
+ this.journal = journal;
+ this.executor = executor;
- this.commandsForRangesLoader = new CommandsForRangesLoader(this);
+ final AccordCache.Type<TxnId, Command, AccordSafeCommand>.Instance commands;
+ final AccordCache.Type<RoutingKey, TimestampsForKey, AccordSafeTimestampsForKey>.Instance timestampsForKey;
+ final AccordCache.Type<RoutingKey, CommandsForKey, AccordSafeCommandsForKey>.Instance commandsForKey;
+ try (AccordExecutor.ExclusiveGlobalCaches exclusive = executor.lockCaches())
+ {
+ commands = exclusive.commands.newInstance(this);
+ timestampsForKey = exclusive.timestampsForKey.newInstance(this);
+ commandsForKey = exclusive.commandsForKey.newInstance(this);
+ this.caches = new ExclusiveCaches(executor.lock, exclusive.global, commands, timestampsForKey, commandsForKey);
+ }
+ this.taskExecutor = executor.executor(this);
+ this.commandsForRangesLoader = new CommandsForRangesLoader(this);
loadRedundantBefore(journal.loadRedundantBefore(id()));
loadBootstrapBeganAt(journal.loadBootstrapBeganAt(id()));
loadSafeToRead(journal.loadSafeToRead(id()));
loadRangesForEpoch(journal.loadRangesForEpoch(id()));
-
- executor.execute(() -> CommandStore.register(this));
}
- static Factory factory(AccordJournal journal, IntFunction<CommandStoreExecutor> executorFactory)
+ static Factory factory(AccordJournal journal, IntFunction<AccordExecutor> executorFactory)
{
return (id, node, agent, dataStore, progressLogFactory, listenerFactory, rangesForEpoch) ->
new AccordCommandStore(id, node, agent, dataStore, progressLogFactory, listenerFactory, rangesForEpoch, journal, executorFactory.apply(id));
@@ -247,61 +202,65 @@ public class AccordCommandStore extends CommandStore
@Override
public boolean inStore()
{
- return executor.isInThread();
+ return currentThread == Thread.currentThread();
}
- public void checkInStoreThread()
+ void tryPreSetup(AccordTask<?> task)
{
- checkState(inStore());
+ if (inStore() && current != null)
+ task.presetup(current.task);
}
- public void checkNotInStoreThread()
+ public AccordExecutor executor()
{
- if (!CHECK_THREADS)
- return;
- checkState(!inStore());
+ return executor;
}
- public ExecutorService executor()
+ // TODO (desired): we use this for executing callbacks with mutual exclusivity,
+ // but we don't need to block the actual CommandStore - could quite easily
+ // inflate a separate queue dynamically in AccordExecutor
+ public Executor taskExecutor()
{
- return executor.delegate();
+ return taskExecutor;
}
- /**
- * Note that this cache is shared with other commandStores!
- */
- public AccordStateCache cache()
+ public ExclusiveCaches lockCaches()
{
- return executor.cache();
+ //noinspection LockAcquiredButNotSafelyReleased
+ caches.lock.lock();
+ return caches;
}
- public AccordStateCache.Instance<TxnId, Command, AccordSafeCommand> commandCache()
+ public ExclusiveCaches tryLockCaches()
{
- return commandCache;
+ if (caches.lock.tryLock())
+ return caches;
+ return null;
}
- public AccordStateCache.Instance<RoutingKey, TimestampsForKey, AccordSafeTimestampsForKey> timestampsForKeyCache()
+ public Caches cachesExclusive()
{
- return timestampsForKeyCache;
+ Invariants.checkState(executor.isOwningThread());
+ return caches;
}
- public AccordStateCache.Instance<RoutingKey, CommandsForKey, AccordSafeCommandsForKey> commandsForKeyCache()
+ public Caches cachesUnsafe()
{
- return commandsForKeyCache;
+ return caches;
}
@VisibleForTesting
@Override
- protected void unsafeSetRangesForEpoch(CommandStores.RangesForEpoch newRangesForEpoch)
+ public void unsafeSetRangesForEpoch(CommandStores.RangesForEpoch newRangesForEpoch)
{
super.unsafeSetRangesForEpoch(newRangesForEpoch);
}
@Nullable
@VisibleForTesting
- public Runnable appendToKeyspace(Command after)
+ public Runnable appendToKeyspace(TxnId txnId, Command after)
{
- if (after.txnId().is(Routable.Domain.Key))
+ if (txnId.is(Routable.Domain.Key))
return null;
Mutation mutation = AccordKeyspace.getCommandMutation(this.id, after, nextSystemTimestampMicros());
@@ -345,18 +304,18 @@ public class AccordCommandStore extends CommandStore
if (!Invariants.isParanoid())
return true;
- TimestampsForKey reloaded = AccordKeyspace.unsafeLoadTimestampsForKey(this, (TokenKey) key);
+ TimestampsForKey reloaded = AccordKeyspace.unsafeLoadTimestampsForKey(id, (TokenKey) key);
return Objects.equals(evicting, reloaded);
}
TimestampsForKey loadTimestampsForKey(RoutableKey key)
{
- return AccordKeyspace.loadTimestampsForKey(this, (TokenKey) key);
+ return AccordKeyspace.loadTimestampsForKey(id, (TokenKey) key);
}
CommandsForKey loadCommandsForKey(RoutableKey key)
{
- return AccordKeyspace.loadCommandsForKey(this, (TokenKey) key);
+ return AccordKeyspace.loadCommandsForKey(id, (TokenKey) key);
}
boolean validateCommandsForKey(RoutableKey key, CommandsForKey evicting)
@@ -364,40 +323,20 @@ public class AccordCommandStore extends CommandStore
if (!Invariants.isParanoid())
return true;
- CommandsForKey reloaded = AccordKeyspace.loadCommandsForKey(this, (TokenKey) key);
+ CommandsForKey reloaded = AccordKeyspace.loadCommandsForKey(id, (TokenKey) key);
return Objects.equals(evicting, reloaded);
}
@Nullable
- private Runnable saveTimestampsForKey(TimestampsForKey after)
+ Runnable saveTimestampsForKey(RoutingKey key, TimestampsForKey after, Object serialized)
{
- Mutation mutation = AccordKeyspace.getTimestampsForKeyMutation(id, after, nextSystemTimestampMicros());
- return null != mutation ? mutation::applyUnsafe : null;
+ return AccordKeyspace.getTimestampsForKeyUpdater(this, after, nextSystemTimestampMicros());
}
@Nullable
- private Runnable saveCommandsForKey(CommandsForKey after)
- {
- Mutation mutation = AccordKeyspace.getCommandsForKeyMutation(id, after, nextSystemTimestampMicros());
- return null != mutation ? mutation::applyUnsafe : null;
- }
-
- public void setCurrentOperation(AsyncOperation<?> operation)
- {
- checkState(currentOperation == null);
- currentOperation = operation;
- }
-
- public AsyncOperation<?> getContext()
- {
- checkState(currentOperation != null);
- return currentOperation;
- }
-
- public void unsetCurrentOperation(AsyncOperation<?> operation)
+ Runnable saveCommandsForKey(RoutingKey key, CommandsForKey after, Object serialized)
{
- checkState(currentOperation == operation);
- currentOperation = null;
+ return AccordKeyspace.getCommandsForKeyUpdater(id, (TokenKey) key, after, serialized, nextSystemTimestampMicros());
}
public long nextSystemTimestampMicros()
@@ -408,13 +347,13 @@ public class AccordCommandStore extends CommandStore
@Override
public <T> AsyncChain<T> submit(PreLoadContext loadCtx, Function<? super SafeCommandStore, T> function)
{
- return AsyncOperation.create(this, loadCtx, function);
+ return AccordTask.create(this, loadCtx, function).chain();
}
@Override
public <T> AsyncChain<T> submit(Callable<T> task)
{
- return AsyncChains.ofCallable(executor.delegate(), task);
+ return AsyncChains.ofCallable(taskExecutor(), task);
}
public DataStore dataStore()
@@ -435,7 +374,7 @@ public class AccordCommandStore extends CommandStore
@Override
public AsyncChain<Void> execute(PreLoadContext preLoadContext, Consumer<? super SafeCommandStore> consumer)
{
- return AsyncOperation.create(this, preLoadContext, consumer);
+ return AccordTask.create(this, preLoadContext, consumer).chain();
}
public void executeBlocking(Runnable runnable)
@@ -454,43 +393,38 @@ public class AccordCommandStore extends CommandStore
}
}
- public AccordSafeCommandStore beginOperation(PreLoadContext preLoadContext,
- Map<TxnId, AccordSafeCommand> commands,
- Map<RoutingKey, AccordSafeTimestampsForKey> timestampsForKeys,
- Map<RoutingKey, AccordSafeCommandsForKey> commandsForKeys,
- @Nullable AccordSafeCommandsForRanges commandsForRanges)
+ public AccordSafeCommandStore begin(AccordTask<?> operation,
+ @Nullable CommandsForRanges commandsForRanges)
{
checkState(current == null);
- commands.values().forEach(AccordSafeState::preExecute);
- commandsForKeys.values().forEach(AccordSafeState::preExecute);
- timestampsForKeys.values().forEach(AccordSafeState::preExecute);
- if (commandsForRanges != null)
- commandsForRanges.preExecute();
-
- current = AccordSafeCommandStore.create(preLoadContext, commands, timestampsForKeys, commandsForKeys, commandsForRanges, this);
+ current = AccordSafeCommandStore.create(operation, commandsForRanges, this);
return current;
}
+ void setOwner(Thread thread, Thread self)
+ {
+ Invariants.checkState(thread == null ? currentThread == self : currentThread == null);
+ currentThread = thread;
+ if (thread != null) CommandStore.register(this);
+
+ }
+
public boolean hasSafeStore()
{
return current != null;
}
- public void completeOperation(AccordSafeCommandStore store)
+ public void complete(AccordSafeCommandStore store)
{
checkState(current == store);
- try
- {
- current.postExecute();
- }
- finally
- {
- current = null;
- }
+ current.postExecute();
+ current = null;
}
- public void abortCurrentOperation()
+ public void abort(AccordSafeCommandStore store)
{
+ checkInStore();
+ Invariants.checkState(store == current);
current = null;
}
@@ -511,34 +445,37 @@ public class AccordCommandStore extends CommandStore
Ranges allRanges = safeStore.ranges().all();
Ranges coordinateRanges = Ranges.EMPTY;
long coordinateEpoch = -1;
- for (int i = 0; i < rangeDeps.txnIdCount(); i++)
+ try (ExclusiveCaches caches = lockCaches())
{
- TxnId txnId = rangeDeps.txnId(i);
- AccordCachingState<TxnId, Command> state = commandCache.getUnsafe(txnId);
- if (state != null && state.isLoaded() && state.get() != null && state.get().known().isDefinitionKnown())
- continue;
+ for (int i = 0; i < rangeDeps.txnIdCount(); i++)
+ {
+ TxnId txnId = rangeDeps.txnId(i);
+ AccordCacheEntry<TxnId, Command> state = caches.commands().getUnsafe(txnId);
+ if (state != null && state.isLoaded() && state.getExclusive() != null && state.getExclusive().known().isDefinitionKnown())
+ continue;
- Ranges addRanges = rangeDeps.ranges(i).slice(allRanges);
- if (addRanges.isEmpty()) continue;
+ Ranges addRanges = rangeDeps.ranges(i).slice(allRanges);
+ if (addRanges.isEmpty()) continue;
- if (coordinateEpoch != txnId.epoch())
- {
- coordinateEpoch = txnId.epoch();
- coordinateRanges = ranges.allAt(txnId.epoch());
+ if (coordinateEpoch != txnId.epoch())
+ {
+ coordinateEpoch = txnId.epoch();
+ coordinateRanges = ranges.allAt(txnId.epoch());
+ }
+ if (addRanges.intersects(coordinateRanges)) continue;
+ addRanges = redundantBefore.removeShardRedundant(txnId, txnId, addRanges);
+ if (addRanges.isEmpty()) continue;
+ diskCommandsForRanges().mergeTransitive(txnId, addRanges, Ranges::with);
}
- if (addRanges.intersects(coordinateRanges)) continue;
- addRanges = redundantBefore.removeShardRedundant(txnId, txnId, addRanges);
- if (addRanges.isEmpty()) continue;
- diskCommandsForRanges().mergeTransitive(txnId, addRanges, Ranges::with);
}
}
- public void appendCommands(List<SavedCommand.DiffWriter> diffs, Runnable onFlush)
+ public void appendCommands(List<SavedCommand.Writer> diffs, Runnable onFlush)
{
for (int i = 0; i < diffs.size(); i++)
{
boolean isLast = i == diffs.size() - 1;
- SavedCommand.DiffWriter writer = diffs.get(i);
+ SavedCommand.Writer writer = diffs.get(i);
journal.appendCommand(id, writer, isLast ? onFlush : null);
}
}
@@ -549,6 +486,21 @@ public class AccordCommandStore extends CommandStore
return journal.loadCommand(id, txnId, unsafeGetRedundantBefore(), durableBefore());
}
+ public static Command prepareToCache(Command command)
+ {
+ // TODO (required): validate we don't have duplicate objects
+ if (command != null)
+ {
+ PartialTxn txn = command.partialTxn();
+ if (txn != null)
+ {
+ TxnRead read = (TxnRead) txn.read();
+ read.unmemoize();
+ }
+ }
+ return command;
+ }
+
public MinimalCommand loadMinimal(TxnId txnId)
{
return journal.loadMinimal(id, txnId, MINIMAL, unsafeGetRedundantBefore(), durableBefore());
@@ -568,22 +520,13 @@ public class AccordCommandStore extends CommandStore
{
TxnId txnId = command.txnId();
Participants<?> keys = null;
- List<TxnId> deps = null;
if (CommandsForKey.manages(txnId))
keys = command.hasBeen(Committed) ? command.participants().hasTouched() : command.participants().touches();
else if (!CommandsForKey.managesExecution(txnId) && command.hasBeen(Status.Stable) && !command.hasBeen(Status.Truncated))
keys = command.asCommitted().waitingOn.keys;
- if (command.partialDeps() != null)
- deps = command.partialDeps().txnIds();
-
if (keys != null)
- {
- if (deps != null)
- return PreLoadContext.contextFor(txnId, deps, keys, keyHistory);
-
return PreLoadContext.contextFor(txnId, keys, keyHistory);
- }
return PreLoadContext.contextFor(txnId);
}
@@ -593,12 +536,12 @@ public class AccordCommandStore extends CommandStore
TxnId txnId = command.txnId();
AsyncPromise<?> future = new AsyncPromise<>();
- execute(context(command, COMMANDS),
+ execute(context(command, SYNC),
safeStore -> {
Command local = command;
if (local.status() != Truncated && local.status() != Invalidated)
{
- Cleanup cleanup = Cleanup.shouldCleanup(local, unsafeGetRedundantBefore(), durableBefore());
+ Cleanup cleanup = Cleanup.shouldCleanup(agent, local, unsafeGetRedundantBefore(), durableBefore());
switch (cleanup)
{
case NO:
@@ -678,116 +621,4 @@ public class AccordCommandStore extends CommandStore
if (rangesForEpoch != null)
unsafeSetRangesForEpoch(new CommandStores.RangesForEpoch(rangesForEpoch.epochs, rangesForEpoch.ranges, this));
}
-
- public static class CommandStoreExecutor implements CacheSize
- {
- final AccordStateCache stateCache;
- final SequentialExecutorPlus delegate;
- final long threadId;
-
- CommandStoreExecutor(AccordStateCache stateCache, SequentialExecutorPlus delegate, long threadId)
- {
- this.stateCache = stateCache;
- this.delegate = delegate;
- this.threadId = threadId;
- }
-
- public boolean hasTasks()
- {
- return delegate.getPendingTaskCount() > 0 || delegate.getActiveTaskCount() > 0;
- }
-
- CommandStoreExecutor(AccordStateCache stateCache, SequentialExecutorPlus delegate)
- {
- this.stateCache = stateCache;
- this.delegate = delegate;
- this.threadId = getThreadId();
- }
-
- public boolean isInThread()
- {
- if (!CHECK_THREADS)
- return true;
-
- return threadId == Thread.currentThread().getId();
- }
-
- public void shutdown()
- {
- delegate.shutdown();
- }
-
- public boolean awaitTermination(long timeout, TimeUnit unit) throws InterruptedException
- {
- return delegate.awaitTermination(timeout, unit);
- }
-
- public Future<?> submit(Runnable task)
- {
- return delegate.submit(task);
- }
-
- public ExecutorService delegate()
- {
- return delegate;
- }
-
- public void execute(Runnable command)
- {
- delegate.submit(command);
- }
-
- private long getThreadId()
- {
- try
- {
- return delegate.submit(() -> Thread.currentThread().getId()).get();
- }
- catch (InterruptedException e)
- {
- throw new AssertionError(e);
- }
- catch (ExecutionException e)
- {
- throw new RuntimeException(e);
- }
- }
-
- @VisibleForTesting
- public AccordStateCache cache()
- {
- return stateCache;
- }
-
- @VisibleForTesting
- public void unsafeClearCache()
- {
- stateCache.unsafeClear();
- }
-
- @Override
- public void setCapacity(long bytes)
- {
- Invariants.checkState(isInThread());
- stateCache.setCapacity(bytes);
- }
-
- @Override
- public long capacity()
- {
- return stateCache.capacity();
- }
-
- @Override
- public int size()
- {
- return stateCache.size();
- }
-
- @Override
- public long weightedSize()
- {
- return stateCache.weightedSize();
- }
- }
}
diff --git a/src/java/org/apache/cassandra/service/accord/AccordCommandStores.java b/src/java/org/apache/cassandra/service/accord/AccordCommandStores.java
index d51223243d..0a530bf4d3 100644
--- a/src/java/org/apache/cassandra/service/accord/AccordCommandStores.java
+++ b/src/java/org/apache/cassandra/service/accord/AccordCommandStores.java
@@ -18,6 +18,7 @@
package org.apache.cassandra.service.accord;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.List;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;
@@ -27,7 +28,6 @@ import accord.api.Agent;
import accord.api.DataStore;
import accord.api.LocalListeners;
import accord.api.ProgressLog;
-import accord.local.CommandStore;
import accord.local.CommandStores;
import accord.local.Node;
import accord.local.NodeCommandStoreService;
@@ -36,50 +36,87 @@ import accord.primitives.Range;
import accord.topology.Topology;
import accord.utils.RandomSource;
import org.apache.cassandra.cache.CacheSize;
-import org.apache.cassandra.concurrent.ExecutorPlus;
import org.apache.cassandra.concurrent.Stage;
+import org.apache.cassandra.config.AccordSpec.QueueShardModel;
import org.apache.cassandra.config.DatabaseDescriptor;
-import org.apache.cassandra.metrics.AccordStateCacheMetrics;
+import org.apache.cassandra.metrics.AccordCacheMetrics;
import org.apache.cassandra.metrics.CacheSizeMetrics;
import org.apache.cassandra.schema.TableId;
-import org.apache.cassandra.service.accord.AccordCommandStore.CommandStoreExecutor;
+import org.apache.cassandra.service.accord.AccordExecutor.AccordExecutorFactory;
import org.apache.cassandra.service.accord.api.AccordRoutingKey;
import org.apache.cassandra.utils.concurrent.UncheckedInterruptedException;
-import static org.apache.cassandra.concurrent.ExecutorFactory.Global.executorFactory;
-import static org.apache.cassandra.concurrent.Stage.ACCORD_MIGRATION;
-import static org.apache.cassandra.concurrent.Stage.ACCORD_RANGE_LOADER;
-import static org.apache.cassandra.concurrent.Stage.MUTATION;
-import static org.apache.cassandra.concurrent.Stage.READ;
+import static org.apache.cassandra.config.AccordSpec.QueueShardModel.THREAD_PER_SHARD;
+import static org.apache.cassandra.config.DatabaseDescriptor.getAccordQueueSubmissionModel;
+import static org.apache.cassandra.config.DatabaseDescriptor.getAccordQueueShardCount;
+import static org.apache.cassandra.service.accord.AccordExecutor.Mode.RUN_WITHOUT_LOCK;
+import static org.apache.cassandra.service.accord.AccordExecutor.Mode.RUN_WITH_LOCK;
+import static org.apache.cassandra.service.accord.AccordExecutor.constant;
+import static org.apache.cassandra.service.accord.AccordExecutor.constantFactory;
public class AccordCommandStores extends CommandStores implements CacheSize
{
public static final String ACCORD_STATE_CACHE = "AccordStateCache";
private final CacheSizeMetrics cacheSizeMetrics;
- private final CommandStoreExecutor[] executors;
- private long cacheSize;
+ private final AccordExecutor[] executors;
+ private long cacheSize, workingSetSize;
+ private int maxQueuedLoads, maxQueuedRangeLoads;
+ private boolean shrinkingOn;
AccordCommandStores(NodeCommandStoreService node, Agent agent, DataStore store, RandomSource random,
ShardDistributor shardDistributor, ProgressLog.Factory progressLogFactory, LocalListeners.Factory listenerFactory,
- AccordJournal journal, CommandStoreExecutor[] executors)
+ AccordJournal journal, AccordExecutor[] executors)
{
super(node, agent, store, random, shardDistributor, progressLogFactory, listenerFactory,
AccordCommandStore.factory(journal, id -> executors[id % executors.length]));
- setCapacity(DatabaseDescriptor.getAccordCacheSizeInMiB() << 20);
this.executors = executors;
this.cacheSizeMetrics = new CacheSizeMetrics(ACCORD_STATE_CACHE, this);
+ cacheSize = DatabaseDescriptor.getAccordCacheSizeInMiB() << 20;
+ workingSetSize = DatabaseDescriptor.getAccordWorkingSetSizeInMiB() << 20;
+ maxQueuedLoads = DatabaseDescriptor.getAccordMaxQueuedLoadCount();
+ maxQueuedRangeLoads = DatabaseDescriptor.getAccordMaxQueuedRangeLoadCount();
+ shrinkingOn = DatabaseDescriptor.getAccordCacheShrinkingOn();
+ refreshCapacities();
}
static Factory factory(AccordJournal journal)
{
return (time, agent, store, random, shardDistributor, progressLogFactory, listenerFactory) -> {
- CommandStoreExecutor[] executors = new CommandStoreExecutor[DatabaseDescriptor.getAccordShardCount()];
+ AccordExecutor[] executors = new AccordExecutor[getAccordQueueShardCount()];
+ AccordExecutorFactory factory;
+ int maxThreads = Integer.MAX_VALUE;
+ switch (getAccordQueueSubmissionModel())
+ {
+ default: throw new AssertionError("Unhandled QueueSubmissionModel: " + getAccordQueueSubmissionModel());
+ case SYNC: factory = AccordExecutorSyncSubmit::new; break;
+ case SEMI_SYNC: factory = AccordExecutorSemiSyncSubmit::new; break;
+ case ASYNC: factory = AccordExecutorAsyncSubmit::new; break;
+ case EXEC_ST:
+ factory = AccordExecutorSimple::new;
+ maxThreads = 1;
+ break;
+ }
+
for (int id = 0; id < executors.length; id++)
{
- AccordStateCacheMetrics metrics = new AccordStateCacheMetrics(ACCORD_STATE_CACHE);
- AccordStateCache stateCache = new AccordStateCache(Stage.READ.executor(), Stage.MUTATION.executor(), 8 << 20, metrics);
- executors[id] = new CommandStoreExecutor(stateCache, executorFactory().sequential(CommandStore.class.getSimpleName() + '[' + id + ']'));
+ AccordCacheMetrics metrics = new AccordCacheMetrics(ACCORD_STATE_CACHE);
+ QueueShardModel shardModel = DatabaseDescriptor.getAccordQueueShardModel();
+ String baseName = AccordExecutor.class.getSimpleName() + '[' + id;
+ int threads = Math.min(maxThreads, Math.max(DatabaseDescriptor.getAccordConcurrentOps() / getAccordQueueShardCount(), 1));
+ switch (shardModel)
+ {
+ case THREAD_PER_SHARD:
+ case THREAD_PER_SHARD_SYNC_QUEUE:
+ executors[id] = factory.get(id, shardModel == THREAD_PER_SHARD ? RUN_WITHOUT_LOCK : RUN_WITH_LOCK, 1, constant(baseName + ']'), metrics, constantFactory(Stage.READ.executor()), constantFactory(Stage.MUTATION.executor()), constantFactory(Stage.READ.executor()), agent);
+ break;
+ case THREAD_POOL_PER_SHARD:
+ executors[id] = factory.get(id, RUN_WITHOUT_LOCK, threads, num -> baseName + ',' + num + ']', metrics, AccordExecutor::submitIOToSelf, AccordExecutor::submitIOToSelf, AccordExecutor::submitIOToSelf, agent);
+ break;
+ case THREAD_POOL_PER_SHARD_EXCLUDES_IO:
+ executors[id] = factory.get(id, RUN_WITHOUT_LOCK, threads, num -> baseName + ',' + num + ']', metrics, constantFactory(Stage.READ.executor()), constantFactory(Stage.MUTATION.executor()), constantFactory(Stage.READ.executor()), agent);
+ break;
+ }
}
return new AccordCommandStores(time, agent, store, random, shardDistributor, progressLogFactory, listenerFactory, journal, executors);
@@ -109,7 +146,27 @@ public class AccordCommandStores extends CommandStores implements CacheSize
public synchronized void setCapacity(long bytes)
{
cacheSize = bytes;
- refreshCacheSizes();
+ refreshCapacities();
+ }
+
+ public synchronized void setWorkingSetSize(long bytes)
+ {
+ workingSetSize = bytes;
+ refreshCapacities();
+ }
+
+ public synchronized void setCapacityAndWorkingSetSize(long newCacheSize, long newWorkingSetSize)
+ {
+ cacheSize = newCacheSize;
+ workingSetSize = newWorkingSetSize;
+ refreshCapacities();
+ }
+
+ public synchronized void setMaxQueuedLoads(int total, int range)
+ {
+ maxQueuedLoads = total;
+ maxQueuedRangeLoads = range;
+ refreshCapacities();
}
@Override
@@ -122,7 +179,7 @@ public class AccordCommandStores extends CommandStores implements CacheSize
public int size()
{
int size = 0;
- for (CommandStoreExecutor executor : executors)
+ for (AccordExecutor executor : executors)
size += executor.size();
return size;
}
@@ -131,19 +188,31 @@ public class AccordCommandStores extends CommandStores implements CacheSize
public long weightedSize()
{
long size = 0;
- for (CommandStoreExecutor executor : executors)
+ for (AccordExecutor executor : executors)
size += executor.weightedSize();
return size;
}
- synchronized void refreshCacheSizes()
+ synchronized void refreshCapacities()
+ {
+ long capacityPerExecutor = cacheSize / executors.length;
+ long workingSetPerExecutor = workingSetSize < 0 ? Long.MAX_VALUE : workingSetSize / executors.length;
+ int maxLoadsPerExecutor = (maxQueuedLoads + executors.length - 1) / executors.length;
+ int maxRangeLoadsPerExecutor = (maxQueuedRangeLoads + executors.length - 1) / executors.length;
+ for (AccordExecutor executor : executors)
+ {
+ executor.executeDirectlyWithLock(() -> {
+ executor.setCapacity(capacityPerExecutor);
+ executor.setWorkingSetSize(workingSetPerExecutor);
+ executor.setMaxQueuedLoads(maxLoadsPerExecutor, maxRangeLoadsPerExecutor);
+ executor.cacheExclusive().setShrinkingOn(shrinkingOn);
+ });
+ }
+ }
+
+ public List<AccordExecutor> executors()
{
- if (count() == 0)
- return;
- long perExecutor = cacheSize / executors.length;
- // TODO (low priority, safety): we might transiently breach our limit if we increase one store before decreasing another
- for (CommandStoreExecutor executor : executors)
- executor.execute(() -> executor.setCapacity(perExecutor));
+ return Arrays.asList(executors.clone());
}
public void waitForQuiescense()
@@ -151,23 +220,13 @@ public class AccordCommandStores extends CommandStores implements CacheSize
boolean hadPending;
try
{
- List<ExecutorPlus> executors = new ArrayList<>();
- for (CommandStoreExecutor executor : this.executors)
- executors.add(executor.delegate);
-
- executors.add(READ.executor());
- executors.add(MUTATION.executor());
- executors.add(ACCORD_MIGRATION.executor());
- executors.add(ACCORD_RANGE_LOADER.executor());
-
do
{
hadPending = false;
List<Future<?>> futures = new ArrayList<>();
- for (ExecutorPlus executor : executors)
+ for (AccordExecutor executor : this.executors)
{
- if (!hadPending && (executor.getPendingTaskCount() > 0 || executor.getActiveTaskCount() > 0))
- hadPending = true;
+ hadPending |= executor.hasTasks();
futures.add(executor.submit(() -> {}));
}
for (Future<?> future : futures)
@@ -190,7 +249,7 @@ public class AccordCommandStores extends CommandStores implements CacheSize
public synchronized void shutdown()
{
super.shutdown();
- for (CommandStoreExecutor executor : executors)
+ for (AccordExecutor executor : executors)
{
executor.shutdown();
try
diff --git a/src/java/org/apache/cassandra/service/accord/AccordDataStore.java b/src/java/org/apache/cassandra/service/accord/AccordDataStore.java
index 19c30b0f38..9786de506b 100644
--- a/src/java/org/apache/cassandra/service/accord/AccordDataStore.java
+++ b/src/java/org/apache/cassandra/service/accord/AccordDataStore.java
@@ -38,7 +38,6 @@ import org.apache.cassandra.config.DatabaseDescriptor;
import org.apache.cassandra.db.ColumnFamilyStore;
import org.apache.cassandra.db.DataRange;
import org.apache.cassandra.db.Keyspace;
-import org.apache.cassandra.db.commitlog.CommitLogPosition;
import org.apache.cassandra.db.filter.ColumnFilter;
import org.apache.cassandra.db.lifecycle.View;
import org.apache.cassandra.db.memtable.Memtable;
@@ -68,7 +67,7 @@ public class AccordDataStore implements DataStore
static class SnapshotBounds
{
final List<org.apache.cassandra.dht.Range<Token>> ranges = new ArrayList<>();
- CommitLogPosition position;
+ long id;
}
@Override
@@ -93,7 +92,7 @@ public class AccordDataStore implements DataStore
ColumnFamilyStore cfs = Keyspace.openAndGetStoreIfExists(tableMetadata);
// TODO (required): when we can safely map TxnId.hlc() -> local timestamp, consult Memtable timestamps
Memtable memtable = cfs.getCurrentMemtable();
- e.getValue().position = memtable.getCommitLogLowerBound();
+ e.getValue().id = memtable.getMemtableId();
}
ScheduledExecutors.scheduledTasks.schedule(() -> {
@@ -106,7 +105,7 @@ public class AccordDataStore implements DataStore
View view = cfs.getTracker().getView();
for (Memtable memtable : view.getAllMemtables())
{
- if (memtable.getCommitLogLowerBound().compareTo(bounds.position) > 0) continue;
+ if (memtable.getMemtableId() > bounds.id) continue;
if (!intersects(cfs, memtable, bounds.ranges)) continue;
futures.add(cfs.forceFlush(ACCORD_TXN_GC));
diff --git a/src/java/org/apache/cassandra/service/accord/AccordExecutor.java b/src/java/org/apache/cassandra/service/accord/AccordExecutor.java
new file mode 100644
index 0000000000..ce778dd76a
--- /dev/null
+++ b/src/java/org/apache/cassandra/service/accord/AccordExecutor.java
@@ -0,0 +1,1025 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord;
+
+import java.util.concurrent.Executor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.Lock;
+import java.util.function.BiConsumer;
+import java.util.function.BiFunction;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+
+import accord.api.Agent;
+import accord.api.RoutingKey;
+import accord.impl.TimestampsForKey;
+import accord.local.Command;
+import accord.local.cfk.CommandsForKey;
+import accord.primitives.TxnId;
+import accord.utils.ArrayBuffers.BufferList;
+import accord.utils.IntrusivePriorityHeap;
+import accord.utils.Invariants;
+import accord.utils.QuadConsumer;
+import accord.utils.QuadFunction;
+import accord.utils.QuintConsumer;
+import accord.utils.TriConsumer;
+import accord.utils.TriFunction;
+import accord.utils.async.Cancellable;
+import org.agrona.collections.Object2ObjectHashMap;
+import org.apache.cassandra.cache.CacheSize;
+import org.apache.cassandra.concurrent.ExecutorPlus;
+import org.apache.cassandra.concurrent.ScheduledExecutors;
+import org.apache.cassandra.concurrent.Shutdownable;
+import org.apache.cassandra.metrics.AccordCacheMetrics;
+import org.apache.cassandra.utils.concurrent.AsyncPromise;
+import org.apache.cassandra.utils.concurrent.Future;
+
+import static org.apache.cassandra.service.accord.AccordCacheEntry.Status.EVICTED;
+import static org.apache.cassandra.service.accord.AccordCache.CommandAdapter.COMMAND_ADAPTER;
+import static org.apache.cassandra.service.accord.AccordCache.CommandsForKeyAdapter.CFK_ADAPTER;
+import static org.apache.cassandra.service.accord.AccordCache.registerJfrListener;
+import static org.apache.cassandra.service.accord.AccordTask.State.LOADING;
+import static org.apache.cassandra.service.accord.AccordTask.State.SCANNING_RANGES;
+import static org.apache.cassandra.service.accord.AccordTask.State.WAITING_TO_LOAD;
+import static org.apache.cassandra.service.accord.AccordTask.State.WAITING_TO_RUN;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+public abstract class AccordExecutor implements CacheSize, AccordCacheEntry.OnLoaded, AccordCacheEntry.OnSaved, Shutdownable
+{
+ public interface AccordExecutorFactory
+ {
+ AccordExecutor get(int executorId, Mode mode, int threads, IntFunction<String> name, AccordCacheMetrics metrics, ExecutorFunctionFactory loadExecutor, ExecutorFunctionFactory saveExecutor, ExecutorFunctionFactory rangeLoadExecutor, Agent agent);
+ }
+
+ public enum Mode { RUN_WITH_LOCK, RUN_WITHOUT_LOCK }
+
+ public interface ExecutorFunction extends BiFunction<Task, Runnable, Cancellable> {}
+ public interface ExecutorFunctionFactory extends Function<AccordExecutor, ExecutorFunction> {}
+
+ // WARNING: this is a shared object, so close is NOT idempotent
+ public static final class ExclusiveGlobalCaches extends GlobalCaches implements AutoCloseable
+ {
+ final Lock lock;
+
+ public ExclusiveGlobalCaches(Lock lock, AccordCache global, AccordCache.Type<TxnId, Command, AccordSafeCommand> commands, AccordCache.Type<RoutingKey, TimestampsForKey, AccordSafeTimestampsForKey> timestampsForKey, AccordCache.Type<RoutingKey, CommandsForKey, AccordSafeCommandsForKey> commandsForKey)
+ {
+ super(global, commands, timestampsForKey, commandsForKey);
+ this.lock = lock;
+ }
+
+ @Override
+ public void close()
+ {
+ lock.unlock();
+ }
+ }
+
+ public static class GlobalCaches
+ {
+ public final AccordCache global;
+ public final AccordCache.Type<TxnId, Command, AccordSafeCommand> commands;
+ public final AccordCache.Type<RoutingKey, TimestampsForKey, AccordSafeTimestampsForKey> timestampsForKey;
+ public final AccordCache.Type<RoutingKey, CommandsForKey, AccordSafeCommandsForKey> commandsForKey;
+
+ public GlobalCaches(AccordCache global, AccordCache.Type<TxnId, Command, AccordSafeCommand> commands, AccordCache.Type<RoutingKey, TimestampsForKey, AccordSafeTimestampsForKey> timestampsForKey, AccordCache.Type<RoutingKey, CommandsForKey, AccordSafeCommandsForKey> commandsForKey)
+ {
+ this.global = global;
+ this.commands = commands;
+ this.timestampsForKey = timestampsForKey;
+ this.commandsForKey = commandsForKey;
+ }
+ }
+
+ final Lock lock;
+ final Agent agent;
+ final int executorId;
+ private final AccordCache cache;
+ private final ExecutorFunction loadExecutor;
+ private final ExecutorFunction rangeLoadExecutor;
+
+ private final TaskQueue<AccordTask<?>> scanningRanges = new TaskQueue<>(SCANNING_RANGES); // never queried, just parked here while scanning
+ private final TaskQueue<AccordTask<?>> loading = new TaskQueue<>(LOADING); // never queried, just parked here while loading
+
+ private final TaskQueue<AccordTask<?>> waitingToLoadRangeTxns = new TaskQueue<>(WAITING_TO_LOAD);
+
+ private final TaskQueue<AccordTask<?>> waitingToLoad = new TaskQueue<>(WAITING_TO_LOAD);
+ private final TaskQueue<Task> waitingToRun = new TaskQueue<>(WAITING_TO_RUN);
+ private final Object2ObjectHashMap<AccordCommandStore, CommandStoreQueue> commandStoreQueues = new Object2ObjectHashMap<>();
+
+ private final AccordCacheEntry.OnLoaded onRangeLoaded = this::onRangeLoaded;
+ private final ExclusiveGlobalCaches caches;
+
+ /**
+ * The maximum total number of loads we can queue at once - this includes loads for range transactions,
+ * which are subject to this limit as well as that imposed by {@link #maxQueuedRangeLoads}
+ */
+ private int maxQueuedLoads = 64;
+ /**
+ * The maximum number of loads exclusively for range transactions we can queue at once; the {@link #maxQueuedLoads} limit also applies.
+ */
+ private int maxQueuedRangeLoads = 8;
+
+ private long maxWorkingSetSizeInBytes;
+ private long maxWorkingCapacityInBytes;
+ private int nextPosition;
+ private int activeLoads, activeRangeLoads;
+ private boolean hasPausedLoading;
+ int tasks;
+ int running;
+
+ AccordExecutor(Lock lock, int executorId, AccordCacheMetrics metrics, ExecutorFunctionFactory loadExecutor, ExecutorFunctionFactory saveExecutor, ExecutorFunctionFactory rangeLoadExecutor, Agent agent)
+ {
+ this.lock = lock;
+ this.executorId = executorId;
+ this.cache = new AccordCache(alwaysNullTask(saveExecutor.apply(this)), this, 0, metrics);
+ this.loadExecutor = loadExecutor.apply(this);
+ this.rangeLoadExecutor = rangeLoadExecutor.apply(this);
+ this.agent = agent;
+
+ final AccordCache.Type<TxnId, Command, AccordSafeCommand> commands;
+ final AccordCache.Type<RoutingKey, TimestampsForKey, AccordSafeTimestampsForKey> timestampsForKey;
+ final AccordCache.Type<RoutingKey, CommandsForKey, AccordSafeCommandsForKey> commandsForKey;
+ commands = cache.newType(TxnId.class, COMMAND_ADAPTER);
+ registerJfrListener(executorId, commands, "Command");
+ timestampsForKey = cache.newType(RoutingKey.class,
+ AccordCommandStore::loadTimestampsForKey,
+ AccordCommandStore::saveTimestampsForKey,
+ Function.identity(),
+ AccordCommandStore::validateTimestampsForKey,
+ AccordObjectSizes::timestampsForKey,
+ AccordSafeTimestampsForKey::new);
+ registerJfrListener(executorId, timestampsForKey, "TimestampsForKey");
+ commandsForKey = cache.newType(RoutingKey.class, CFK_ADAPTER);
+ registerJfrListener(executorId, commandsForKey, "CommandsForKey");
+
+ this.caches = new ExclusiveGlobalCaches(lock, cache, commands, timestampsForKey, commandsForKey);
+ ScheduledExecutors.scheduledFastTasks.scheduleAtFixedRate(() -> {
+ executeDirectlyWithLock(cache::processNoEvictQueue);
+ }, 1L, 1L, TimeUnit.SECONDS);
+ }
+
+ public int executorId()
+ {
+ return executorId;
+ }
+
+ public ExclusiveGlobalCaches lockCaches()
+ {
+ //noinspection LockAcquiredButNotSafelyReleased
+ lock.lock();
+ return caches;
+ }
+
+ public AccordCache cacheExclusive()
+ {
+ Invariants.checkState(isOwningThread());
+ return cache;
+ }
+
+ public AccordCache cacheUnsafe()
+ {
+ return cache;
+ }
+
+ boolean hasWaitingToRun()
+ {
+ updateWaitingToRunExclusive();
+ return !waitingToRun.isEmpty();
+ }
+
+ Task pollWaitingToRunExclusive()
+ {
+ updateWaitingToRunExclusive();
+ return waitingToRun.poll();
+ }
+
+ void updateWaitingToRunExclusive()
+ {
+ maybeUnpauseLoading();
+ }
+
+ void maybeUnpauseLoading()
+ {
+ if (!hasPausedLoading)
+ return;
+
+ if (cache.weightedSize() < maxWorkingCapacityInBytes || (loading.isEmpty() && waitingToRun.isEmpty()))
+ {
+ hasPausedLoading = false;
+ enqueueLoadsExclusive();
+ }
+ }
+
+ public abstract boolean hasTasks();
+ abstract boolean isOwningThread();
+
+ private void enqueueLoadsExclusive()
+ {
+ outer: while (true)
+ {
+ TaskQueue<AccordTask<?>> queue = waitingToLoadRangeTxns.isEmpty() || activeRangeLoads >= maxQueuedRangeLoads ? waitingToLoad : waitingToLoadRangeTxns;
+ AccordTask<?> next = queue.peek();
+ if (next == null)
+ return;
+
+ if (hasPausedLoading || cache.weightedSize() >= maxWorkingCapacityInBytes)
+ {
+ // we have too much in memory already, and we have work waiting to run, so let that complete before queueing more
+ if (!loading.isEmpty() || !waitingToRun.isEmpty())
+ {
+ hasPausedLoading = true;
+ return;
+ }
+ }
+
+ switch (next.state())
+ {
+ default:
+ {
+ failExclusive(next, new AssertionError("Unexpected state: " + next.toDescription()));
+ break;
+ }
+ case WAITING_TO_SCAN_RANGES:
+ if (activeRangeLoads >= maxQueuedRangeLoads)
+ {
+ parkRangeLoad(next);
+ }
+ else
+ {
+ ++activeRangeLoads;
+ ++activeLoads;
+ next.rangeScanner().start(rangeLoadExecutor);
+ updateQueue(next);
+ }
+ break;
+
+ case WAITING_TO_LOAD:
+ while (true)
+ {
+ AccordCacheEntry<?, ?> load = next.peekWaitingToLoad();
+ boolean isForRange = isForRange(next, load);
+ if (isForRange && activeRangeLoads >= maxQueuedRangeLoads)
+ {
+ parkRangeLoad(next);
+ continue outer;
+ }
+
+ Invariants.checkState(load != null);
+ AccordCacheEntry.OnLoaded onLoaded = this;
+ ++activeLoads;
+ if (isForRange)
+ {
+ ++activeRangeLoads;
+ onLoaded = onRangeLoaded;
+ }
+
+ for (AccordTask<?> task : cache.load(loadExecutor, next, load, onLoaded))
+ {
+ if (task == next) continue;
+ if (task.onLoading(load))
+ updateQueue(task);
+ }
+ Object prev = next.pollWaitingToLoad();
+ Invariants.checkState(prev == load);
+ if (next.peekWaitingToLoad() == null)
+ break;
+
+ Invariants.checkState(next.state() == WAITING_TO_LOAD, "Invalid state: %s", next);
+ if (activeLoads >= maxQueuedLoads)
+ return;
+ }
+ Invariants.checkState(next.state().compareTo(LOADING) >= 0, "Invalid state: %s", next);
+ updateQueue(next);
+ }
+ }
+ }
+
+ private boolean isForRange(AccordTask<?> task, AccordCacheEntry<?, ?> load)
+ {
+ boolean isForRangeTxn = task.hasRanges();
+ if (!isForRangeTxn)
+ return false;
+
+ for (AccordTask<?> t : load.loadingOrWaiting().waiters())
+ {
+ if (!t.hasRanges())
+ return false;
+ }
+ return true;
+ }
+
+ private void parkRangeLoad(AccordTask<?> task)
+ {
+ if (task.queued() != waitingToLoadRangeTxns)
+ {
+ task.unqueueIfQueued();
+ task.addToQueue(waitingToLoadRangeTxns);
+ }
+ }
+
+ void consumeExclusive(Object object)
+ {
+ try
+ {
+ if (object instanceof AccordTask<?>)
+ loadExclusive((AccordTask<?>) object);
+ else
+ ((SubmitAsync) object).acceptExclusive(this);
+ }
+ catch (Throwable t)
+ {
+ agent.onUncaughtException(t);
+ }
+ }
+
+ private void updateQueue(AccordTask<?> task)
+ {
+ task.unqueueIfQueued();
+ switch (task.state())
+ {
+ default: throw new AssertionError("Unexpected state: " + task.toDescription());
+ case WAITING_TO_SCAN_RANGES:
+ case WAITING_TO_LOAD:
+ task.addToQueue(waitingToLoad);
+ break;
+ case SCANNING_RANGES:
+ task.addToQueue(scanningRanges);
+ break;
+ case LOADING:
+ task.addToQueue(loading);
+ break;
+ case WAITING_TO_RUN:
+ task.runQueuedAt = nanoTime();
+ commandStoreQueues.computeIfAbsent(task.commandStore, CommandStoreQueue::new)
+ .append(task);
+ break;
+ }
+ }
+
+ private void waitingToRun(Task task)
+ {
+ if (task.commandStore == null)
+ {
+ waitingToRun.append(task);
+ }
+ else
+ {
+ commandStoreQueues.computeIfAbsent(task.commandStore, CommandStoreQueue::new)
+ .append(task);
+ }
+ }
+
+ private Cancellable submitIOExclusive(Task parent, Runnable run)
+ {
+ Invariants.checkState(isOwningThread());
+ ++tasks;
+ PlainRunnable task = new PlainRunnable(null, run, null);
+ // TODO (expected): adopt queue position of the submitting task
+ if (parent == null) assignNewQueuePosition(task);
+ else assignQueueSubPosition(parent, task);
+ waitingToRun.append(task);
+ return task;
+ }
+
+ private void assignNewQueuePosition(Task task)
+ {
+ task.queuePosition = (((long)++nextPosition) & 0xffffffffL) << 31;
+ }
+
+ private void assignQueueSubPosition(Task parent, Task task)
+ {
+ task.queuePosition = parent.queuePosition | (++nextPosition & 0x7fffffff);
+ }
+
+ public Executor executor(AccordCommandStore commandStore)
+ {
+ return task -> AccordExecutor.this.submit(task, commandStore);
+ }
+
+ public <R> void submit(AccordTask<R> operation)
+ {
+ submit(AccordExecutor::loadExclusive, Function.identity(), operation);
+ }
+
+ public <R> void cancel(AccordTask<R> operation)
+ {
+ submit(AccordExecutor::cancelExclusive, OnCancel::new, operation);
+ }
+
+ public void onScannedRanges(AccordTask<?> task, Throwable fail)
+ {
+ submit(AccordExecutor::onScannedRangesExclusive, OnScannedRanges::new, task, fail);
+ }
+
+ public <K, V> void onSaved(AccordCacheEntry<K, V> saved, Object identity, Throwable fail)
+ {
+ submit(AccordExecutor::onSavedExclusive, OnSaved::new, saved, identity, fail);
+ }
+
+ @Override
+ public <K, V> void onLoaded(AccordCacheEntry<K, V> loaded, V value, Throwable fail)
+ {
+ submit(AccordExecutor::onLoadedExclusive, OnLoaded::new, loaded, value, fail, false);
+ }
+
+ public <K, V> void onRangeLoaded(AccordCacheEntry<K, V> loaded, V value, Throwable fail)
+ {
+ submit(AccordExecutor::onLoadedExclusive, OnLoaded::new, loaded, value, fail, true);
+ }
+
+ private <P1> void submit(BiConsumer<AccordExecutor, P1> sync, Function<P1, ?> async, P1 p1)
+ {
+ submit((e, c, p1a, p2a, p3) -> c.accept(e, p1a), (f, p1a, p2a, p3) -> f.apply(p1a), sync, async, p1, null, null);
+ }
+
+ private <P1, P2> void submit(TriConsumer<AccordExecutor, P1, P2> sync, BiFunction<P1, P2, ?> async, P1 p1, P2 p2)
+ {
+ submit((e, c, p1a, p2a, p3) -> c.accept(e, p1a, p2a), (f, p1a, p2a, p3) -> f.apply(p1a, p2a), sync, async, p1, p2, null);
+ }
+
+ private <P1, P2, P3> void submit(QuadConsumer<AccordExecutor, P1, P2, P3> sync, TriFunction<P1, P2, P3, ?> async, P1 p1, P2 p2, P3 p3)
+ {
+ submit((e, c, p1a, p2a, p3a) -> c.accept(e, p1a, p2a, p3a), TriFunction::apply, sync, async, p1, p2, p3);
+ }
+
+ private <P1, P2, P3, P4> void submit(QuintConsumer<AccordExecutor, P1, P2, P3, P4> sync, QuadFunction<P1, P2, P3, P4, Object> async, P1 p1, P2 p2, P3 p3, P4 p4)
+ {
+ submit(sync, async, p1, p1, p2, p3, p4);
+ }
+
+ abstract <P1s, P1a, P2, P3, P4> void submit(QuintConsumer<AccordExecutor, P1s, P2, P3, P4> sync, QuadFunction<P1a, P2, P3, P4, Object> async, P1s p1s, P1a p1a, P2 p2, P3 p3, P4 p4);
+
+ private void submitExclusive(AsyncPromise<Void> result, Runnable run, AccordCommandStore commandStore)
+ {
+ ++tasks;
+ PlainRunnable task = new PlainRunnable(result, run, commandStore);
+ task.queuePosition = ++nextPosition;
+ waitingToRun(task);
+ }
+
+ private void submitExclusive(AsyncPromise<Void> result, PlainRunnable task)
+ {
+ ++tasks;
+ task.queuePosition = ++nextPosition;
+ waitingToRun(task);
+ }
+
+ private void loadExclusive(AccordTask<?> task)
+ {
+ ++tasks;
+ assignNewQueuePosition(task);
+ task.setupExclusive();
+ updateQueue(task);
+ enqueueLoadsExclusive();
+ }
+
+ private void cancelExclusive(AccordTask<?> task)
+ {
+ switch (task.state())
+ {
+ default:
+ case INITIALIZED:
+ // we could be cancelled before we even reach the queue
+ task.cancelExclusive();
+ break;
+
+ case LOADING:
+ case WAITING_TO_LOAD:
+ case WAITING_TO_SCAN_RANGES:
+ case SCANNING_RANGES:
+ case WAITING_TO_RUN:
+ --tasks;
+ task.unqueueIfQueued();
+ task.cancelExclusive();
+ break;
+
+ case RUNNING:
+ case PERSISTING:
+ case FINISHED:
+ case CANCELLED:
+ case FAILED:
+ // cannot safely cancel
+ }
+ }
+
+ private void onScannedRangesExclusive(AccordTask<?> task, Throwable fail)
+ {
+ --activeLoads;
+ --activeRangeLoads;
+ // the task may have already been cancelled, in which case we don't need to fail it
+ if (!task.state().isExecuted())
+ {
+ if (fail != null)
+ {
+ failExclusive(task, fail);
+ }
+ else
+ {
+ task.rangeScanner().scannedExclusive();
+ updateQueue(task);
+ }
+ }
+ enqueueLoadsExclusive();
+ }
+
+ private void failExclusive(AccordTask<?> task, Throwable fail)
+ {
+ if (task.state().isExecuted())
+ return;
+
+ --tasks;
+ try { task.failExclusive(fail); }
+ catch (Throwable t) { agent.onUncaughtException(t); }
+ finally
+ {
+ task.unqueueIfQueued();
+ task.cleanupExclusive();
+ }
+ }
+
+ private <K, V> void onSavedExclusive(AccordCacheEntry<K, V> state, Object identity, Throwable fail)
+ {
+ cache.saved(state, identity, fail);
+ }
+
+ private <K, V> void onLoadedExclusive(AccordCacheEntry<K, V> loaded, V value, Throwable fail, boolean isForRange)
+ {
+ --activeLoads;
+ --activeRangeLoads;
+
+ if (loaded.status() != EVICTED)
+ {
+ try (BufferList<AccordTask<?>> tasks = loaded.loading().copyWaiters())
+ {
+ if (fail != null)
+ {
+ for (AccordTask<?> task : tasks)
+ failExclusive(task, fail);
+ cache.failedToLoad(loaded);
+ }
+ else
+ {
+ cache.loaded(loaded, value);
+ for (AccordTask<?> task : tasks)
+ {
+ if (task.onLoad(loaded))
+ {
+ Invariants.checkState(task.queued() == loading);
+ task.unqueue();
+ waitingToRun(task);
+ }
+ }
+ }
+ }
+ }
+
+ enqueueLoadsExclusive();
+ }
+
+ public Future<?> submit(Runnable run)
+ {
+ return submit(run, null);
+ }
+
+ // TODO (expected): offer queue jumping/priorities
+ public Future<?> submit(Runnable run, AccordCommandStore commandStore)
+ {
+ PlainRunnable task = new PlainRunnable(new AsyncPromise<>(), run, commandStore);
+ AsyncPromise<Void> result = new AsyncPromise<>();
+ submit(AccordExecutor::submitExclusive, SubmitPlainRunnable::new, result, run, commandStore);
+ return result;
+ }
+
+ public void execute(Runnable command)
+ {
+ submit(command);
+ }
+
+ public void executeDirectlyWithLock(Runnable command)
+ {
+ lock.lock();
+ try
+ {
+ command.run();
+ }
+ finally
+ {
+ lock.unlock();
+ }
+ }
+
+ public void execute(Runnable command, AccordCommandStore commandStore)
+ {
+ submit(command, commandStore);
+ }
+
+ @Override
+ public void setCapacity(long bytes)
+ {
+ Invariants.checkState(isOwningThread());
+ cache.setCapacity(bytes);
+ maxWorkingCapacityInBytes = cache.capacity() + maxWorkingSetSizeInBytes;
+ }
+
+ public void setWorkingSetSize(long bytes)
+ {
+ Invariants.checkState(isOwningThread());
+ maxWorkingSetSizeInBytes = bytes;
+ maxWorkingCapacityInBytes = cache.capacity() + maxWorkingSetSizeInBytes;
+ if (maxWorkingCapacityInBytes < maxWorkingSetSizeInBytes)
+ maxWorkingCapacityInBytes = Long.MAX_VALUE;
+ }
+
+ public void setMaxQueuedLoads(int total, int range)
+ {
+ Invariants.checkState(isOwningThread());
+ maxQueuedLoads = total;
+ maxQueuedRangeLoads = range;
+ }
+
+ @Override
+ public long capacity()
+ {
+ return cache.capacity();
+ }
+
+ @Override
+ public int size()
+ {
+ return cache.size();
+ }
+
+ @Override
+ public long weightedSize()
+ {
+ return cache.weightedSize();
+ }
+
+ public static abstract class Task extends IntrusivePriorityHeap.Node
+ {
+ final AccordCommandStore commandStore;
+ long queuePosition;
+
+ protected Task(AccordCommandStore commandStore)
+ {
+ this.commandStore = commandStore;
+ }
+
+ /**
+ * Prepare to run while holding the state cache lock
+ */
+ abstract protected void preRunExclusive();
+
+ /**
+ * Run the command; the state cache lock may or may not be held depending on the executor implementation
+ */
+ abstract protected void run();
+ /**
+ * Fail the command; the state cache lock may or may not be held depending on the executor implementation
+ */
+ abstract protected void fail(Throwable fail);
+
+ /**
+ * Cleanup the command while holding the state cache lock
+ */
+ abstract protected void cleanupExclusive();
+
+ abstract protected void addToQueue(TaskQueue queue);
+ }
+
+ class CommandStoreQueue extends Task
+ {
+ final TaskQueue<Task> queue = new TaskQueue<>(WAITING_TO_RUN);
+ Task next;
+
+ CommandStoreQueue(AccordCommandStore commandStore)
+ {
+ super(commandStore);
+ }
+
+ @Override
+ protected void preRunExclusive()
+ {
+ Invariants.checkState(next != null);
+ Thread self = Thread.currentThread();
+ commandStore.setOwner(self, self);
+ next.preRunExclusive();
+ }
+
+ @Override
+ protected void run()
+ {
+ next.run();
+ }
+
+ @Override
+ protected void fail(Throwable t)
+ {
+ next.fail(t);
+ }
+
+ @Override
+ protected void cleanupExclusive()
+ {
+ next.cleanupExclusive();
+ commandStore.setOwner(null, Thread.currentThread());
+ updateNext(queue.poll());
+ }
+
+ @Override
+ protected void addToQueue(TaskQueue queue)
+ {
+ throw new UnsupportedOperationException();
+ }
+
+ void append(Task task)
+ { // TODO (expected): if the new task is higher priority, replace next
+ if (next == null) updateNext(task);
+ else task.addToQueue(queue);
+ }
+
+ void updateNext(Task task)
+ {
+ next = task;
+ if (task != null)
+ {
+ queuePosition = task.queuePosition;
+ waitingToRun.append(this);
+ }
+ }
+ }
+
+ static final class TaskQueue<T extends Task> extends IntrusivePriorityHeap<T>
+ {
+ final AccordTask.State kind;
+
+ TaskQueue(AccordTask.State kind)
+ {
+ this.kind = kind;
+ }
+
+ @Override
+ public int compare(T o1, T o2)
+ {
+ return Long.compare(o1.queuePosition, o2.queuePosition);
+ }
+ public void append(T task)
+ {
+ super.append(task);
+ }
+
+ public T poll()
+ {
+ ensureHeapified();
+ return pollNode();
+ }
+
+ public T peek()
+ {
+ ensureHeapified();
+ return peekNode();
+ }
+
+ public void remove(T remove)
+ {
+ super.remove(remove);
+ }
+
+ public boolean contains(T contains)
+ {
+ return super.contains(contains);
+ }
+ }
+
+ private abstract static class SubmitAsync
+ {
+ abstract void acceptExclusive(AccordExecutor executor);
+ }
+
+ private static class SubmitPlainRunnable extends SubmitAsync
+ {
+ final AsyncPromise<Void> result;
+ final Runnable run;
+ final AccordCommandStore commandStore;
+
+ private SubmitPlainRunnable(AsyncPromise<Void> result, Runnable run, AccordCommandStore commandStore)
+ {
+ this.result = result;
+ this.run = run;
+ this.commandStore = commandStore;
+ }
+
+ @Override
+ void acceptExclusive(AccordExecutor executor)
+ {
+ executor.submitExclusive(result, run, commandStore);
+ }
+ }
+
+ private static class OnLoaded<K, V> extends SubmitAsync
+ {
+ static final int FAIL = 1;
+ static final int RANGE = 2;
+ final AccordCacheEntry<K, V> loaded;
+ final Object result;
+ final int flags;
+
+ OnLoaded(AccordCacheEntry<K, V> loaded, V success, Throwable fail, boolean isForRange)
+ {
+ this.loaded = loaded;
+ int flags = isForRange ? RANGE : 0;
+ if (fail == null)
+ {
+ result = success;
+ }
+ else
+ {
+ result = fail;
+ flags |= FAIL;
+ }
+ this.flags = flags;
+ }
+
+ V success()
+ {
+ return (flags & FAIL) == 0 ? (V) result : null;
+ }
+
+ Throwable fail()
+ {
+ return (flags & FAIL) == 0 ? null : (Throwable) result;
+ }
+
+ boolean isForRange()
+ {
+ return (flags & RANGE) != 0;
+ }
+
+ @Override
+ void acceptExclusive(AccordExecutor executor)
+ {
+ executor.onLoadedExclusive(loaded, success(), fail(), isForRange());
+ }
+ }
+
+ private static class OnScannedRanges extends SubmitAsync
+ {
+ final AccordTask<?> scanned;
+ final Throwable fail;
+
+ private OnScannedRanges(AccordTask<?> scanned, Throwable fail)
+ {
+ this.scanned = scanned;
+ this.fail = fail;
+ }
+
+ @Override
+ void acceptExclusive(AccordExecutor executor)
+ {
+ executor.onScannedRangesExclusive(scanned, fail);
+ }
+ }
+
+ private static class OnSaved<K, V> extends SubmitAsync
+ {
+ final AccordCacheEntry<K, V> state;
+ final Object identity;
+ final Throwable fail;
+
+ private OnSaved(AccordCacheEntry<K, V> state, Object identity, Throwable fail)
+ {
+ this.state = state;
+ this.identity = identity;
+ this.fail = fail;
+ }
+
+ @Override
+ void acceptExclusive(AccordExecutor executor)
+ {
+ executor.onSavedExclusive(state, identity, fail);
+ }
+ }
+
+ private static class OnCancel<R> extends SubmitAsync
+ {
+ final AccordTask<R> cancel;
+
+ private OnCancel(AccordTask<R> cancel)
+ {
+ this.cancel = cancel;
+ }
+
+ @Override
+ void acceptExclusive(AccordExecutor executor)
+ {
+ executor.cancelExclusive(cancel);
+ }
+ }
+
+ static <O> IntFunction<O> constant(O out)
+ {
+ return ignore -> out;
+ }
+
+ static ExecutorFunctionFactory constantFactory(ExecutorFunction exec)
+ {
+ return ignore -> exec;
+ }
+
+ static ExecutorFunctionFactory constantFactory(ExecutorPlus exec)
+ {
+ return ignore -> wrap(exec);
+ }
+
+ static ExecutorFunction wrap(ExecutorPlus exec)
+ {
+ return (t, r) -> wrap(exec.submit(r));
+ }
+
+ static Cancellable wrap(Future<?> f)
+ {
+ return () -> f.cancel(false);
+ }
+
+ public static ExecutorFunction submitIOToSelf(AccordExecutor executor)
+ {
+ return executor::submitIOExclusive;
+ }
+
+ private static Function<Runnable, Cancellable> alwaysNullTask(ExecutorFunction f)
+ {
+ return r -> f.apply(null, r);
+ }
+
+ class PlainRunnable extends Task implements Cancellable
+ { // TODO (expected): support cancellation
+ final AsyncPromise<Void> result;
+ final Runnable run;
+
+ PlainRunnable(AsyncPromise<Void> result, Runnable run, AccordCommandStore commandStore)
+ {
+ super(commandStore);
+ this.result = result;
+ this.run = run;
+ }
+
+ @Override
+ protected void preRunExclusive() {}
+
+ @Override
+ protected void run()
+ {
+ run.run();
+ if (result != null)
+ result.trySuccess(null);
+ }
+
+ @Override
+ protected void fail(Throwable t)
+ {
+ if (result != null)
+ result.tryFailure(t);
+ agent.onUncaughtException(t);
+ }
+
+ @Override
+ protected void cleanupExclusive() {}
+
+ @Override
+ protected void addToQueue(TaskQueue queue)
+ {
+ Invariants.checkState(queue.kind == WAITING_TO_RUN);
+ queue.append(this);
+ }
+
+ @Override
+ public void cancel()
+ {
+ executeDirectlyWithLock(() -> {
+ if (isInHeap())
+ {
+ waitingToRun.remove(this);
+ if (result != null)
+ result.cancel(false);
+ }
+ });
+ }
+ }
+
+}
diff --git a/src/java/org/apache/cassandra/service/accord/AccordExecutorAbstractLockLoop.java b/src/java/org/apache/cassandra/service/accord/AccordExecutorAbstractLockLoop.java
new file mode 100644
index 0000000000..176470f113
--- /dev/null
+++ b/src/java/org/apache/cassandra/service/accord/AccordExecutorAbstractLockLoop.java
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord;
+
+import java.util.concurrent.locks.Lock;
+
+import accord.api.Agent;
+import accord.utils.QuadFunction;
+import accord.utils.QuintConsumer;
+import org.apache.cassandra.concurrent.Interruptible;
+import org.apache.cassandra.metrics.AccordCacheMetrics;
+import org.apache.cassandra.utils.concurrent.ConcurrentLinkedStack;
+
+import static org.apache.cassandra.concurrent.Interruptible.State.NORMAL;
+import static org.apache.cassandra.service.accord.AccordExecutor.Mode.RUN_WITH_LOCK;
+
+abstract class AccordExecutorAbstractLockLoop extends AccordExecutor
+{
+ final ConcurrentLinkedStack<Object> submitted = new ConcurrentLinkedStack<>();
+ boolean isHeldByExecutor;
+
+ AccordExecutorAbstractLockLoop(Lock lock, int executorId, AccordCacheMetrics metrics, ExecutorFunctionFactory loadExecutor, ExecutorFunctionFactory saveExecutor, ExecutorFunctionFactory rangeLoadExecutor, Agent agent)
+ {
+ super(lock, executorId, metrics, loadExecutor, saveExecutor, rangeLoadExecutor, agent);
+ }
+
+ abstract void notifyWorkExclusive();
+ abstract void awaitExclusive() throws InterruptedException;
+ abstract boolean isInLoop();
+ abstract <P1s, P1a, P2, P3, P4> void submitExternal(QuintConsumer<AccordExecutor, P1s, P2, P3, P4> sync, QuadFunction<P1a, P2, P3, P4, Object> async, P1s p1s, P1a p1a, P2 p2, P3 p3, P4 p4);
+
+ <P1s, P1a, P2, P3, P4> void submit(QuintConsumer<AccordExecutor, P1s, P2, P3, P4> sync, QuadFunction<P1a, P2, P3, P4, Object> async, P1s p1s, P1a p1a, P2 p2, P3 p3, P4 p4)
+ {
+ // if we're a loop thread, we will poll the waitingToRun queue when we come around
+ if (isInLoop()) submitted.push(async.apply(p1a, p2, p3, p4));
+ else submitExternal(sync, async, p1s, p1a, p2, p3, p4);
+ }
+
+ <P1s, P1a, P2, P3, P4> void submitExternalExclusive(QuintConsumer<AccordExecutor, P1s, P2, P3, P4> sync, QuadFunction<P1a, P2, P3, P4, Object> async, P1s p1s, P1a p1a, P2 p2, P3 p3, P4 p4)
+ {
+ try
+ {
+ try
+ {
+ drainSubmittedExclusive();
+ }
+ catch (Throwable t)
+ {
+ try { sync.accept(this, p1s, p2, p3, p4); }
+ catch (Throwable t2) { t.addSuppressed(t2); }
+ throw t;
+ }
+ sync.accept(this, p1s, p2, p3, p4);
+ }
+ finally
+ {
+ notifyIfMoreWorkExclusive();
+ }
+ }
+
+ public boolean hasTasks()
+ {
+ if (tasks > 0 || !submitted.isEmpty() || running > 0)
+ return true;
+
+ lock.lock();
+ try
+ {
+ return tasks > 0 || !submitted.isEmpty() || running > 0;
+ }
+ finally
+ {
+ lock.unlock();
+ }
+ }
+
+ void updateWaitingToRunExclusive()
+ {
+ drainSubmittedExclusive();
+ super.updateWaitingToRunExclusive();
+ }
+
+ void drainSubmittedExclusive()
+ {
+ submitted.drain(AccordExecutor::consumeExclusive, this, true);
+ }
+
+ void notifyIfMoreWorkExclusive()
+ {
+ if (hasWaitingToRun())
+ notifyWorkExclusive();
+ }
+
+ private void enterLockExclusive()
+ {
+ isHeldByExecutor = true;
+ }
+
+ private void exitLockExclusive()
+ {
+ isHeldByExecutor = false;
+ notifyIfMoreWorkExclusive();
+ }
+
+ private void pauseExclusive()
+ {
+ --running;
+ }
+
+ private void resumeExclusive()
+ {
+ ++running;
+ }
+
+ Interruptible.Task task(Mode mode)
+ {
+ return mode == RUN_WITH_LOCK ? this::runWithLock : this::runWithoutLock;
+ }
+
+ protected void runWithLock(Interruptible.State state) throws InterruptedException
+ {
+ lock.lockInterruptibly();
+ try
+ {
+ resumeExclusive();
+ enterLockExclusive();
+ while (true)
+ {
+ Task task = pollWaitingToRunExclusive();
+
+ if (task != null)
+ {
+ --tasks;
+ try
+ {
+ task.preRunExclusive();
+ task.run();
+ }
+ catch (Throwable t)
+ {
+ task.fail(t);
+ }
+ finally
+ {
+ task.cleanupExclusive();
+ }
+ }
+ else
+ {
+ if (state != NORMAL)
+ {
+ pauseExclusive();
+ exitLockExclusive();
+ return;
+ }
+
+ pauseExclusive();
+ awaitExclusive();
+ resumeExclusive();
+ }
+ }
+ }
+ catch (Throwable t)
+ {
+ pauseExclusive();
+ exitLockExclusive();
+ throw t;
+ }
+ finally
+ {
+ lock.unlock();
+ }
+ }
+
+ protected void runWithoutLock(Interruptible.State state) throws InterruptedException
+ {
+ Task task = null;
+ while (true)
+ {
+ lock.lock();
+ try
+ {
+ if (task != null) task.cleanupExclusive();
+ else resumeExclusive();
+ enterLockExclusive();
+
+ while (true)
+ {
+ task = pollWaitingToRunExclusive();
+ if (task != null)
+ {
+ exitLockExclusive();
+ break;
+ }
+
+ if (state != NORMAL)
+ {
+ exitLockExclusive();
+ return;
+ }
+
+ pauseExclusive();
+ awaitExclusive();
+ resumeExclusive();
+ }
+ --tasks;
+ task.preRunExclusive();
+ }
+ catch (Throwable t)
+ {
+ if (task != null)
+ {
+ try { task.fail(t); }
+ catch (Throwable t2) { t.addSuppressed(t2); }
+ try { task.cleanupExclusive(); }
+ catch (Throwable t2) { t.addSuppressed(t2); }
+ try { agent.onUncaughtException(t); }
+ catch (Throwable t2) { /* nothing we can sensibly do after already reporting */ }
+ }
+ pauseExclusive();
+ exitLockExclusive();
+ throw t;
+ }
+ finally
+ {
+ lock.unlock();
+ }
+
+ try
+ {
+ task.run();
+ }
+ catch (Throwable t)
+ {
+ try { task.fail(t); }
+ catch (Throwable t2)
+ {
+ try
+ {
+ t2.addSuppressed(t);
+ agent.onUncaughtException(t2);
+ }
+ catch (Throwable t3)
+ {
+ // empty to ensure we definitely loop so we cleanup the task
+ }
+ }
+ }
+ }
+ }
+}
diff --git a/src/java/org/apache/cassandra/service/accord/AccordExecutorAbstractSemiSyncSubmit.java b/src/java/org/apache/cassandra/service/accord/AccordExecutorAbstractSemiSyncSubmit.java
new file mode 100644
index 0000000000..a216e60ae8
--- /dev/null
+++ b/src/java/org/apache/cassandra/service/accord/AccordExecutorAbstractSemiSyncSubmit.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord;
+
+import java.util.concurrent.locks.Lock;
+
+import accord.api.Agent;
+import accord.utils.QuadFunction;
+import accord.utils.QuintConsumer;
+import org.apache.cassandra.concurrent.Interruptible;
+import org.apache.cassandra.metrics.AccordCacheMetrics;
+
+import static org.apache.cassandra.service.accord.AccordExecutor.Mode.RUN_WITH_LOCK;
+
+abstract class AccordExecutorAbstractSemiSyncSubmit extends AccordExecutorAbstractLockLoop
+{
+ AccordExecutorAbstractSemiSyncSubmit(Lock lock, int executorId, AccordCacheMetrics metrics, ExecutorFunctionFactory loadExecutor, ExecutorFunctionFactory saveExecutor, ExecutorFunctionFactory rangeLoadExecutor, Agent agent)
+ {
+ super(lock, executorId, metrics, loadExecutor, saveExecutor, rangeLoadExecutor, agent);
+ }
+
+ abstract void notifyWorkAsync();
+ abstract void awaitExclusive() throws InterruptedException;
+
+ Interruptible.Task task(Mode mode)
+ {
+ return mode == RUN_WITH_LOCK ? this::runWithLock : this::runWithoutLock;
+ }
+
+ <P1s, P1a, P2, P3, P4> void submitExternal(QuintConsumer<AccordExecutor, P1s, P2, P3, P4> sync, QuadFunction<P1a, P2, P3, P4, Object> async, P1s p1s, P1a p1a, P2 p2, P3 p3, P4 p4)
+ {
+ if (!lock.tryLock())
+ {
+ submitted.push(async.apply(p1a, p2, p3, p4));
+ notifyWorkAsync();
+ return;
+ }
+
+ try
+ {
+ submitExternalExclusive(sync, async, p1s, p1a, p2, p3, p4);
+ }
+ finally
+ {
+ lock.unlock();
+ }
+ }
+}
diff --git a/src/java/org/apache/cassandra/service/accord/AccordExecutorAsyncSubmit.java b/src/java/org/apache/cassandra/service/accord/AccordExecutorAsyncSubmit.java
new file mode 100644
index 0000000000..c0f0c5c06a
--- /dev/null
+++ b/src/java/org/apache/cassandra/service/accord/AccordExecutorAsyncSubmit.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord;
+
+import java.util.concurrent.TimeUnit;
+import java.util.function.IntFunction;
+
+import accord.api.Agent;
+import org.apache.cassandra.metrics.AccordCacheMetrics;
+import org.apache.cassandra.utils.concurrent.LockWithAsyncSignal;
+
+// WARNING: experimental - needs more testing
+class AccordExecutorAsyncSubmit extends AccordExecutorAbstractSemiSyncSubmit
+{
+ private final AccordExecutorInfiniteLoops loops;
+ private final LockWithAsyncSignal lock;
+
+ public AccordExecutorAsyncSubmit(int executorId, Mode mode, int threads, IntFunction<String> name, AccordCacheMetrics metrics, ExecutorFunctionFactory loadExecutor, ExecutorFunctionFactory saveExecutor, ExecutorFunctionFactory rangeLoadExecutor, Agent agent)
+ {
+ this(new LockWithAsyncSignal(), executorId, mode, threads, name, metrics, loadExecutor, saveExecutor, rangeLoadExecutor, agent);
+ }
+
+ private AccordExecutorAsyncSubmit(LockWithAsyncSignal lock, int executorId, Mode mode, int threads, IntFunction<String> name, AccordCacheMetrics metrics, ExecutorFunctionFactory loadExecutor, ExecutorFunctionFactory saveExecutor, ExecutorFunctionFactory rangeLoadExecutor, Agent agent)
+ {
+ super(lock, executorId, metrics, loadExecutor, saveExecutor, rangeLoadExecutor, agent);
+ this.lock = lock;
+ this.loops = new AccordExecutorInfiniteLoops(mode, threads, name, this::task);
+ }
+
+ @Override
+ void awaitExclusive() throws InterruptedException
+ {
+ lock.clearSignal();
+ if (submitted.isEmpty())
+ lock.await();
+ }
+
+ @Override
+ boolean isInLoop()
+ {
+ return loops.isInLoop();
+ }
+
+ @Override
+ void notifyWorkAsync()
+ {
+ lock.signal();
+ }
+
+ @Override
+ void notifyWorkExclusive()
+ {
+ lock.signal();
+ }
+
+ @Override
+ boolean isOwningThread()
+ {
+ return lock.isOwner(Thread.currentThread());
+ }
+
+ @Override
+ public void shutdown()
+ {
+ loops.shutdown();
+ }
+
+ @Override
+ public Object shutdownNow()
+ {
+ return loops.shutdownNow();
+ }
+
+ @Override
+ public boolean isTerminated()
+ {
+ return loops.isTerminated();
+ }
+
+ @Override
+ public boolean awaitTermination(long timeout, TimeUnit unit) throws InterruptedException
+ {
+ return loops.awaitTermination(timeout, unit);
+ }
+}
diff --git a/src/java/org/apache/cassandra/service/accord/AccordExecutorInfiniteLoops.java b/src/java/org/apache/cassandra/service/accord/AccordExecutorInfiniteLoops.java
new file mode 100644
index 0000000000..5a19d9aa22
--- /dev/null
+++ b/src/java/org/apache/cassandra/service/accord/AccordExecutorInfiniteLoops.java
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord;
+
+import java.util.concurrent.TimeUnit;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+
+import accord.utils.Invariants;
+import org.agrona.collections.LongHashSet;
+import org.apache.cassandra.concurrent.InfiniteLoopExecutor;
+import org.apache.cassandra.concurrent.Interruptible;
+import org.apache.cassandra.concurrent.Shutdownable;
+import org.apache.cassandra.service.accord.AccordExecutor.Mode;
+
+import static org.apache.cassandra.concurrent.ExecutorFactory.Global.executorFactory;
+import static org.apache.cassandra.concurrent.InfiniteLoopExecutor.Daemon.NON_DAEMON;
+import static org.apache.cassandra.concurrent.InfiniteLoopExecutor.Interrupts.UNSYNCHRONIZED;
+import static org.apache.cassandra.concurrent.InfiniteLoopExecutor.SimulatorSafe.SAFE;
+import static org.apache.cassandra.service.accord.AccordExecutor.Mode.RUN_WITH_LOCK;
+import static org.apache.cassandra.utils.Clock.Global.nanoTime;
+
+class AccordExecutorInfiniteLoops implements Shutdownable
+{
+ private final Interruptible[] loops;
+ private final LongHashSet threadIds;
+
+ public AccordExecutorInfiniteLoops(Mode mode, int threads, IntFunction<String> name, Function<Mode, Interruptible.Task> tasks)
+ {
+ Invariants.checkState(mode == RUN_WITH_LOCK ? threads == 1 : threads >= 1);
+ final LongHashSet threadIds = new LongHashSet(threads, 0.5f);
+ this.loops = new Interruptible[threads];
+ for (int i = 0; i < threads; ++i)
+ {
+ loops[i] = executorFactory().infiniteLoop(name.apply(i), tasks.apply(mode), SAFE, NON_DAEMON, UNSYNCHRONIZED);
+ if (loops[i] instanceof InfiniteLoopExecutor)
+ threadIds.add(((InfiniteLoopExecutor) loops[i]).threadId());
+ }
+ this.threadIds = threadIds;
+ }
+
+ public boolean isInLoop()
+ {
+ return threadIds.contains(Thread.currentThread().getId());
+ }
+
+ @Override
+ public void shutdown()
+ {
+ for (Interruptible loop : loops)
+ loop.shutdown();
+ }
+
+ @Override
+ public Object shutdownNow()
+ {
+ for (Interruptible loop : loops)
+ loop.shutdownNow();
+ return null;
+ }
+
+ @Override
+ public boolean isTerminated()
+ {
+ for (Interruptible loop : loops)
+ {
+ if (!loop.isTerminated())
+ return false;
+ }
+ return true;
+ }
+
+ @Override
+ public boolean awaitTermination(long timeout, TimeUnit unit) throws InterruptedException
+ {
+ long deadline = nanoTime() + unit.toNanos(timeout);
+ for (Interruptible loop : loops)
+ {
+ long wait = deadline - nanoTime();
+ if (!loop.awaitTermination(wait, unit))
+ return false;
+ }
+ return true;
+ }
+}
diff --git a/src/java/org/apache/cassandra/service/accord/AccordExecutorSemiSyncSubmit.java b/src/java/org/apache/cassandra/service/accord/AccordExecutorSemiSyncSubmit.java
new file mode 100644
index 0000000000..3272d52971
--- /dev/null
+++ b/src/java/org/apache/cassandra/service/accord/AccordExecutorSemiSyncSubmit.java
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord;
+
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.IntFunction;
+
+import accord.api.Agent;
+import org.apache.cassandra.metrics.AccordCacheMetrics;
+
+// WARNING: experimental - needs more testing
+class AccordExecutorSemiSyncSubmit extends AccordExecutorAbstractSemiSyncSubmit
+{
+ private final AccordExecutorInfiniteLoops loops;
+ private final ReentrantLock lock;
+ private final Condition hasWork;
+
+ public AccordExecutorSemiSyncSubmit(int executorId, Mode mode, int threads, IntFunction<String> name, AccordCacheMetrics metrics, ExecutorFunctionFactory loadExecutor, ExecutorFunctionFactory saveExecutor, ExecutorFunctionFactory rangeLoadExecutor, Agent agent)
+ {
+ this(new ReentrantLock(), executorId, mode, threads, name, metrics, loadExecutor, saveExecutor, rangeLoadExecutor, agent);
+ }
+
+ private AccordExecutorSemiSyncSubmit(ReentrantLock lock, int executorId, Mode mode, int threads, IntFunction<String> name, AccordCacheMetrics metrics, ExecutorFunctionFactory loadExecutor, ExecutorFunctionFactory saveExecutor, ExecutorFunctionFactory rangeLoadExecutor, Agent agent)
+ {
+ super(lock, executorId, metrics, loadExecutor, saveExecutor, rangeLoadExecutor, agent);
+ this.lock = lock;
+ this.hasWork = lock.newCondition();
+ this.loops = new AccordExecutorInfiniteLoops(mode, threads, name, this::task);
+ }
+
+ @Override
+ void awaitExclusive() throws InterruptedException
+ {
+ if (submitted.isEmpty())
+ hasWork.await();
+ }
+
+ @Override
+ boolean isInLoop()
+ {
+ return loops.isInLoop();
+ }
+
+ @Override
+ void notifyWorkAsync()
+ {
+ // we check running both sides of tryLock for ordering guarantees
+ boolean hadRunning = isHeldByExecutor;
+ if (lock.tryLock())
+ {
+ try { hasWork.signal(); }
+ finally { lock.unlock(); }
+ }
+ else if (!hadRunning || !isHeldByExecutor)
+ {
+ lock.lock();
+ try { hasWork.signal(); }
+ finally { lock.unlock(); }
+ }
+ }
+
+ @Override
+ void notifyWorkExclusive()
+ {
+ hasWork.signal();
+ }
+
+ @Override
+ boolean isOwningThread()
+ {
+ return lock.isHeldByCurrentThread();
+ }
+
+ @Override
+ public void shutdown()
+ {
+ loops.shutdown();
+ }
+
+ @Override
+ public Object shutdownNow()
+ {
+ return loops.shutdownNow();
+ }
+
+ @Override
+ public boolean isTerminated()
+ {
+ return loops.isTerminated();
+ }
+
+ @Override
+ public boolean awaitTermination(long timeout, TimeUnit unit) throws InterruptedException
+ {
+ return loops.awaitTermination(timeout, unit);
+ }
+}
diff --git a/src/java/org/apache/cassandra/service/accord/AccordExecutorSimple.java b/src/java/org/apache/cassandra/service/accord/AccordExecutorSimple.java
new file mode 100644
index 0000000000..3e8e76eb98
--- /dev/null
+++ b/src/java/org/apache/cassandra/service/accord/AccordExecutorSimple.java
@@ -0,0 +1,160 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord;
+
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.IntFunction;
+
+import accord.api.Agent;
+import accord.utils.Invariants;
+import accord.utils.QuadFunction;
+import accord.utils.QuintConsumer;
+import org.apache.cassandra.concurrent.ExecutorPlus;
+import org.apache.cassandra.concurrent.Stage;
+import org.apache.cassandra.metrics.AccordCacheMetrics;
+
+import static org.apache.cassandra.concurrent.ExecutorFactory.Global.executorFactory;
+
+class AccordExecutorSimple extends AccordExecutor
+{
+ final ExecutorPlus executor;
+ final ReentrantLock lock;
+
+ public AccordExecutorSimple(int executorId, String name, AccordCacheMetrics metrics, Agent agent)
+ {
+ this(executorId, name, metrics, Stage.READ.executor(), Stage.MUTATION.executor(), Stage.READ.executor(), agent);
+ }
+
+ public AccordExecutorSimple(int executorId, String name, AccordCacheMetrics metrics, ExecutorPlus loadExecutor, ExecutorPlus saveExecutor, ExecutorPlus rangeLoadExecutor, Agent agent)
+ {
+ this(executorId, name, metrics, wrap(loadExecutor), wrap(saveExecutor), wrap(rangeLoadExecutor), agent);
+ }
+
+ public AccordExecutorSimple(int executorId, String name, AccordCacheMetrics metrics, ExecutorFunction loadExecutor, ExecutorFunction saveExecutor, ExecutorFunction rangeLoadExecutor, Agent agent)
+ {
+ this(new ReentrantLock(), executorId, name, metrics, loadExecutor, saveExecutor, rangeLoadExecutor, agent);
+ }
+
+ private AccordExecutorSimple(ReentrantLock lock, int executorId, String name, AccordCacheMetrics metrics, ExecutorFunction loadExecutor, ExecutorFunction saveExecutor, ExecutorFunction rangeLoadExecutor, Agent agent)
+ {
+ super(lock, executorId, metrics, constantFactory(loadExecutor), constantFactory(saveExecutor), constantFactory(rangeLoadExecutor), agent);
+ this.lock = lock;
+ this.executor = executorFactory().sequential(name);
+ }
+
+ public AccordExecutorSimple(int executorId, Mode mode, int threads, IntFunction<String> name, AccordCacheMetrics metrics, ExecutorFunctionFactory loadExecutor, ExecutorFunctionFactory saveExecutor, ExecutorFunctionFactory rangeLoadExecutor, Agent agent)
+ {
+ this(new ReentrantLock(), executorId, mode, threads, name, metrics, loadExecutor, saveExecutor, rangeLoadExecutor, agent);
+ }
+
+ public AccordExecutorSimple(ReentrantLock lock, int executorId, Mode mode, int threads, IntFunction<String> name, AccordCacheMetrics metrics, ExecutorFunctionFactory loadExecutor, ExecutorFunctionFactory saveExecutor, ExecutorFunctionFactory rangeLoadExecutor, Agent agent)
+ {
+ super(lock, executorId, metrics, loadExecutor, saveExecutor, rangeLoadExecutor, agent);
+ Invariants.checkArgument(threads == 1);
+ this.lock = lock;
+ this.executor = executorFactory().sequential(name.apply(0));
+
+ }
+
+ @Override
+ public boolean hasTasks()
+ {
+ return tasks + executor.getActiveTaskCount() + executor.getPendingTaskCount() > 0;
+ }
+
+ protected void run()
+ {
+ lock.lock();
+ try
+ {
+ running = 1;
+ while (true)
+ {
+ Task task = pollWaitingToRunExclusive();
+ if (task == null)
+ return;
+
+ --tasks;
+ try { task.preRunExclusive(); task.run(); }
+ catch (Throwable t) { task.fail(t); }
+ finally { task.cleanupExclusive(); }
+ }
+ }
+ catch (Throwable t)
+ {
+ throw t;
+ }
+ finally
+ {
+ running = 0;
+ if (hasWaitingToRun())
+ executor.execute(this::run);
+ lock.unlock();
+ }
+ }
+
+ @Override
+ <P1s, P1a, P2, P3, P4> void submit(QuintConsumer<AccordExecutor, P1s, P2, P3, P4> sync, QuadFunction<P1a, P2, P3, P4, Object> async, P1s p1s, P1a p1a, P2 p2, P3 p3, P4 p4)
+ {
+ lock.lock();
+ try
+ {
+ sync.accept(this, p1s, p2, p3, p4);
+ }
+ finally
+ {
+ if (hasWaitingToRun())
+ executor.execute(this::run);
+
+ lock.unlock();
+ }
+ }
+
+ @Override
+ boolean isOwningThread()
+ {
+ return lock.isHeldByCurrentThread();
+ }
+
+ @Override
+ public boolean isTerminated()
+ {
+ return executor.isTerminated();
+ }
+
+ @Override
+ public void shutdown()
+ {
+ executor.shutdown();
+ }
+
+ @Override
+ public Object shutdownNow()
+ {
+ return executor.shutdownNow();
+ }
+
+ @Override
+ public boolean awaitTermination(long timeout, TimeUnit units) throws InterruptedException
+ {
+ return executor.awaitTermination(timeout, units);
+ }
+
+}
diff --git a/src/java/org/apache/cassandra/service/accord/AccordExecutorSyncSubmit.java b/src/java/org/apache/cassandra/service/accord/AccordExecutorSyncSubmit.java
new file mode 100644
index 0000000000..2ae7d36ad5
--- /dev/null
+++ b/src/java/org/apache/cassandra/service/accord/AccordExecutorSyncSubmit.java
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.service.accord;
+
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.Condition;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.IntFunction;
+
+import accord.api.Agent;
+import accord.utils.QuadFunction;
+import accord.utils.QuintConsumer;
+import org.apache.cassandra.concurrent.ExecutorPlus;
+import org.apache.cassandra.metrics.AccordCacheMetrics;
+
+class AccordExecutorSyncSubmit extends AccordExecutorAbstractLockLoop
+{
+ private final AccordExecutorInfiniteLoops loops;
+ private final ReentrantLock lock;
+ private final Condition hasWork;
+
+ public AccordExecutorSyncSubmit(int executorId, Mode mode, String name, AccordCacheMetrics metrics, ExecutorPlus loadExecutor, ExecutorPlus saveExecutor, ExecutorPlus rangeLoadExecutor, Agent agent)
+ {
+ this(executorId, mode, 1, constant(name), metrics, loadExecutor, saveExecutor, rangeLoadExecutor, agent);
+ }
+
+ public AccordExecutorSyncSubmit(int executorId, Mode mode, int threads, IntFunction<String> name, AccordCacheMetrics metrics, ExecutorPlus loadExecutor, ExecutorPlus saveExecutor, ExecutorPlus rangeLoadExecutor, Agent agent)
+ {
+ this(executorId, mode, threads, name, metrics, constantFactory(loadExecutor), constantFactory(saveExecutor), constantFactory(rangeLoadExecutor), agent);
+ }
+
+ public AccordExecutorSyncSubmit(int executorId, Mode mode, int threads, IntFunction<String> name, AccordCacheMetrics metrics, ExecutorFunctionFactory loadExecutor, ExecutorFunctionFactory saveExecutor, ExecutorFunctionFactory rangeLoadExecutor, Agent agent)
+ {
+ this(new ReentrantLock(), executorId, mode, threads, name, metrics, loadExecutor, saveExecutor, rangeLoadExecutor, agent);
+ }
+
+ private AccordExecutorSyncSubmit(ReentrantLock lock, int executorId, Mode mode, int threads, IntFunction<String> name, AccordCacheMetrics metrics, ExecutorFunctionFactory loadExecutor, ExecutorFunctionFactory saveExecutor, ExecutorFunctionFactory rangeLoadExecutor, Agent agent)
+ {
+ super(lock, executorId, metrics, loadExecutor, saveExecutor, rangeLoadExecutor, agent);
+ this.lock = lock;
+ this.hasWork = lock.newCondition();
+ this.loops = new AccordExecutorInfiniteLoops(mode, threads, name, this::task);
+ }
+
+ @Override
+ void awaitExclusive() throws InterruptedException
+ {
+ hasWork.await();
+ }
+
+ @Override
+ boolean isInLoop()
+ {
+ return loops.isInLoop();
+ }
+
+ @Override
+ boolean isOwningThread()
+ {
+ return lock.isHeldByCurrentThread();
+ }
+
+ @Override
+ void notifyWorkExclusive()
+ {
+ hasWork.signal();
+ }
+
+ <P1s, P1a, P2, P3, P4> void submitExternal(QuintConsumer<AccordExecutor, P1s, P2, P3, P4> sync, QuadFunction<P1a, P2, P3, P4, Object> async, P1s p1s, P1a p1a, P2 p2, P3 p3, P4 p4)
+ {
+ lock.lock();
+ try
+ {
+ submitExternalExclusive(sync, async, p1s, p1a, p2, p3, p4);
+ }
+ finally
+ {
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void shutdown()
+ {
+ loops.shutdown();
+ }
+
+ @Override
+ public Object shutdownNow()
+ {
+ return loops.shutdownNow();
+ }
+
+ @Override
+ public boolean isTerminated()
+ {
+ return loops.isTerminated();
+ }
+
+ @Override
+ public boolean awaitTermination(long timeout, TimeUnit unit) throws InterruptedException
+ {
+ return loops.awaitTermination(timeout, unit);
+ }
+}
diff --git a/src/java/org/apache/cassandra/service/accord/AccordFetchCoordinator.java b/src/java/org/apache/cassandra/service/accord/AccordFetchCoordinator.java
index 61cb7a4e28..8eb5eeb2ad 100644
--- a/src/java/org/apache/cassandra/service/accord/AccordFetchCoordinator.java
+++ b/src/java/org/apache/cassandra/service/accord/AccordFetchCoordinator.java
@@ -203,7 +203,7 @@ public class AccordFetchCoordinator extends AbstractFetchCoordinator implements
future.addCallback((state, fail) -> {
if (fail == null) success(from, Ranges.of(range));
else fail(from, Ranges.of(range), fail);
- }, ((AccordCommandStore) commandStore()).executor());
+ }, ((AccordCommandStore) commandStore()).taskExecutor());
}
}
diff --git a/src/java/org/apache/cassandra/service/accord/AccordJournal.java b/src/java/org/apache/cassandra/service/accord/AccordJournal.java
index aa79a492a9..3b6bd4c916 100644
--- a/src/java/org/apache/cassandra/service/accord/AccordJournal.java
+++ b/src/java/org/apache/cassandra/service/accord/AccordJournal.java
@@ -62,6 +62,7 @@ import org.apache.cassandra.journal.ValueSerializer;
import org.apache.cassandra.net.MessagingService;
import org.apache.cassandra.service.accord.AccordJournalValueSerializers.IdentityAccumulator;
import org.apache.cassandra.service.accord.JournalKey.JournalKeySupport;
+import org.apache.cassandra.service.accord.api.AccordAgent;
import org.apache.cassandra.utils.ExecutorUtils;
import static accord.primitives.SaveStatus.ErasedOrVestigial;
@@ -86,14 +87,16 @@ public class AccordJournal implements IJournal, Shutdownable
private final Journal<JournalKey, Object> journal;
private final AccordJournalTable<JournalKey, Object> journalTable;
private final Params params;
+ private final AccordAgent agent;
Node node;
enum Status { INITIALIZED, STARTING, REPLAY, STARTED, TERMINATING, TERMINATED }
private volatile Status status = Status.INITIALIZED;
@VisibleForTesting
- public AccordJournal(Params params)
+ public AccordJournal(Params params, AccordAgent agent)
{
+ this.agent = agent;
File directory = new File(DatabaseDescriptor.getAccordJournalDirectory());
this.journal = new Journal<>("AccordJournal", directory, params, JournalKey.SUPPORT,
// In Accord, we are using streaming serialization, i.e. Reader/Writer interfaces instead of materializing objects
@@ -180,7 +183,7 @@ public class AccordJournal implements IJournal, Shutdownable
public Command loadCommand(int commandStoreId, TxnId txnId, RedundantBefore redundantBefore, DurableBefore durableBefore)
{
SavedCommand.Builder builder = loadDiffs(commandStoreId, txnId);
- Cleanup cleanup = builder.shouldCleanup(redundantBefore, durableBefore);
+ Cleanup cleanup = builder.shouldCleanup(agent, redundantBefore, durableBefore);
switch (cleanup)
{
case EXPUNGE_PARTIAL:
@@ -195,7 +198,10 @@ public class AccordJournal implements IJournal, Shutdownable
public SavedCommand.MinimalCommand loadMinimal(int commandStoreId, TxnId txnId, SavedCommand.Load load, RedundantBefore redundantBefore, DurableBefore durableBefore)
{
SavedCommand.Builder builder = loadDiffs(commandStoreId, txnId, load);
- Cleanup cleanup = builder.shouldCleanup(redundantBefore, durableBefore);
+ if (!builder.nextCalled)
+ return null;
+
+ Cleanup cleanup = builder.shouldCleanup(node.agent(), redundantBefore, durableBefore);
switch (cleanup)
{
case EXPUNGE_PARTIAL:
@@ -203,6 +209,7 @@ public class AccordJournal implements IJournal, Shutdownable
case ERASE:
return null;
}
+ Invariants.checkState(builder.saveStatus != null, "No saveSatus loaded, but next was called and cleanup was not: %s", builder);
return builder.asMinimal();
}
@@ -235,7 +242,7 @@ public class AccordJournal implements IJournal, Shutdownable
}
@Override
- public void appendCommand(int store, SavedCommand.DiffWriter value, Runnable onFlush)
+ public void appendCommand(int store, SavedCommand.Writer value, Runnable onFlush)
{
if (value == null || status == Status.REPLAY)
{
@@ -248,7 +255,7 @@ public class AccordJournal implements IJournal, Shutdownable
JournalKey key = new JournalKey(value.key(), JournalKey.Type.COMMAND_DIFF, store);
RecordPointer pointer = journal.asyncWrite(key, value, SENTINEL_HOSTS);
if (onFlush != null)
- journal.onFlush(pointer, onFlush);
+ journal.onDurable(pointer, onFlush);
}
@Override
@@ -266,7 +273,7 @@ public class AccordJournal implements IJournal, Shutdownable
JournalKey key = new JournalKey(TxnId.NONE, JournalKey.Type.DURABLE_BEFORE, 0);
RecordPointer pointer = appendInternal(key, addDurableBefore);
// TODO (required): what happens on failure?
- journal.onFlush(pointer, () -> result.setSuccess(null));
+ journal.onDurable(pointer, () -> result.setSuccess(null));
return result;
}
@@ -297,7 +304,7 @@ public class AccordJournal implements IJournal, Shutdownable
return;
if (pointer != null)
- journal.onFlush(pointer, onFlush);
+ journal.onDurable(pointer, onFlush);
else
onFlush.run();
}
@@ -459,7 +466,7 @@ public class AccordJournal implements IJournal, Shutdownable
}
});
- Cleanup cleanup = builder.shouldCleanup(compactionInfo.redundantBefores.get(key.commandStoreId), compactionInfo.durableBefores.get(key.commandStoreId));
+ Cleanup cleanup = builder.shouldCleanup(node.agent(), compactionInfo.redundantBefores.get(key.commandStoreId), compactionInfo.durableBefores.get(key.commandStoreId));
switch (cleanup)
{
case ERASE:
diff --git a/src/java/org/apache/cassandra/service/accord/AccordJournalValueSerializers.java b/src/java/org/apache/cassandra/service/accord/AccordJournalValueSerializers.java
index c6a2a46bf4..acda123245 100644
--- a/src/java/org/apache/cassandra/service/accord/AccordJournalValueSerializers.java
+++ b/src/java/org/apache/cassandra/service/accord/AccordJournalValueSerializers.java
@@ -55,7 +55,7 @@ public class AccordJournalValueSerializers
}
public static class CommandDiffSerializer
- implements FlyweightSerializer<SavedCommand.DiffWriter, SavedCommand.Builder>
+ implements FlyweightSerializer<SavedCommand.Writer, SavedCommand.Builder>
{
@Override
public SavedCommand.Builder mergerFor(JournalKey journalKey)
@@ -64,7 +64,7 @@ public class AccordJournalValueSerializers
}
@Override
- public void serialize(JournalKey key, SavedCommand.DiffWriter writer, DataOutputPlus out, int userVersion)
+ public void serialize(JournalKey key, SavedCommand.Writer writer, DataOutputPlus out, int userVersion)
{
try
{
diff --git a/src/java/org/apache/cassandra/service/accord/AccordKeyspace.java b/src/java/org/apache/cassandra/service/accord/AccordKeyspace.java
index b3b04a5172..0869e34846 100644
--- a/src/java/org/apache/cassandra/service/accord/AccordKeyspace.java
+++ b/src/java/org/apache/cassandra/service/accord/AccordKeyspace.java
@@ -29,6 +29,7 @@ import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
... 11116 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org