You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by be...@apache.org on 2024/09/14 19:06:08 UTC
(cassandra-accord) branch trunk updated: Redesign progress mechanisms to be memory efficient, use fewer messages and to resolve dependency chains promptly.
This is an automated email from the ASF dual-hosted git repository.
benedict pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra-accord.git
The following commit(s) were added to refs/heads/trunk by this push:
new fb3efe9b Redesign progress mechanisms to be memory efficient, use fewer messages and to resolve dependency chains promptly.
fb3efe9b is described below
commit fb3efe9b8a87f0a182545791a2b0563690d52d00
Author: Benedict Elliott Smith <be...@apache.org>
AuthorDate: Tue Aug 13 21:13:00 2024 +0100
Redesign progress mechanisms to be memory efficient, use fewer messages and to resolve dependency chains promptly.
The SimpleProgressLog had a number of problems:
1. It polled for progress with no attempt to determine whether progress could realistically be made, so:
- as the number of pending transactions grew, the proportion of useful work dropped (as many would be unable to make progress without earlier transactions completing)
- each transaction in the chain could recover only on average 1/2 poll interval behind the last transaction to complete
2. It requested full transaction state from every replica on each attempt
3. It maintained a lot of in-memory state
4. Polling happened en-masse, allowing for little per-transaction control
We also separately maintained fairly expensive per-command listener state that negatively affected our command loading and caching.
The new DefaultProgressLog makes use of several new features: LocalListeners, RemoteListeners, Timers and Await messages.
- LocalListeners provide a memory-efficient collection for managing each CommandStore’s transaction listeners, with dedicated record keeping for inter-transaction relationships.
- RemoteListeners provide a mechanism for request/response pairs that may be separated by longer than the normal Cassandra message timeout, and require minimal state on sender and recipient. This permits replicas to cheaply update their local state machine as soon as distributed information becomes available.
The DefaultProgressLog tracks each transaction with separate timers to handle per-transaction scheduling, backoff etc, and a succinct state machine. To reduce overhead correspondence is preferentially limited to a handful of replicas, and limited to the home shard where appropriate.
patch by Benedict; reviewed by Ariel Weisberg for CASSANDRA-19870
---
accord-core/src/main/java/accord/api/Agent.java | 43 ++
.../java/accord/{config => api}/LocalConfig.java | 7 +-
.../src/main/java/accord/api/LocalListeners.java | 63 ++
.../src/main/java/accord/api/ProgressLog.java | 233 +++----
.../src/main/java/accord/api/RemoteListeners.java | 71 ++
.../RequestTimeouts.java} | 22 +-
.../src/main/java/accord/api/Scheduler.java | 16 +-
.../java/accord/coordinate/AsynchronousAwait.java | 151 +++++
.../src/main/java/accord/coordinate/Barrier.java | 37 +-
.../main/java/accord/coordinate/CheckShards.java | 3 -
...CollectDeps.java => CollectCalculatedDeps.java} | 19 +-
.../coordinate/CoordinateGloballyDurable.java | 2 +-
.../accord/coordinate/CoordinateShardDurable.java | 4 +-
.../src/main/java/accord/coordinate/Exhausted.java | 18 +-
.../java/accord/coordinate/FailureAccumulator.java | 7 +-
.../java/accord/coordinate/FetchCoordinator.java | 18 +-
.../src/main/java/accord/coordinate/FetchData.java | 114 +++-
.../java/accord/coordinate/InformHomeOfTxn.java | 92 ---
.../main/java/accord/coordinate/Invalidate.java | 16 +-
.../main/java/accord/coordinate/MaybeRecover.java | 3 +-
.../src/main/java/accord/coordinate/Propose.java | 4 +-
.../java/accord/coordinate/RangeUnavailable.java | 56 --
.../java/accord/coordinate/ReadCoordinator.java | 18 +-
.../src/main/java/accord/coordinate/Recover.java | 67 +-
.../java/accord/coordinate/RecoverWithRoute.java | 13 +-
.../accord/coordinate/RecoverWithSomeRoute.java | 2 +-
.../java/accord/coordinate/SynchronousAwait.java | 100 +++
.../java/accord/coordinate/TopologyMismatch.java | 2 +-
.../{AppliedTracker.java => AllTracker.java} | 24 +-
.../accord/coordinate/tracking/AwaitTracker.java | 137 ++++
.../java/accord/impl/AbstractFetchCoordinator.java | 3 +-
.../impl/CoordinateDurabilityScheduling.java | 2 +-
.../java/accord/impl/DefaultLocalListeners.java | 457 +++++++++++++
.../java/accord/impl/DefaultRemoteListeners.java | 501 ++++++++++++++
.../java/accord/impl/DefaultRequestTimeouts.java | 146 +++++
.../main/java/accord/impl/ErasedSafeCommand.java | 23 +-
.../java/accord/impl/InMemoryCommandStore.java | 89 +--
.../java/accord/impl/InMemoryCommandStores.java | 17 +-
.../main/java/accord/impl/InMemorySafeCommand.java | 19 -
.../main/java/accord/impl/SimpleProgressLog.java | 729 ---------------------
.../java/accord/impl/progresslog/BaseTxnState.java | 172 +++++
.../progresslog/Callback.java} | 14 +-
.../accord/impl/progresslog/CallbackInvoker.java | 104 +++
.../accord/impl/progresslog/CoordinatePhase.java | 68 ++
.../impl/progresslog/DefaultProgressLog.java | 536 +++++++++++++++
.../impl/progresslog/DefaultProgressLogs.java | 76 +++
.../java/accord/impl/progresslog/HomeState.java | 219 +++++++
.../accord/impl/progresslog/PackedKeyTracker.java | 162 +++++
.../impl/progresslog/PackedLogLinearInteger.java | 74 +++
.../java/accord/impl/progresslog/Progress.java | 57 ++
.../progresslog/SavedProgressToken.java} | 22 +-
.../java/accord/impl/progresslog/TxnState.java | 157 +++++
.../progresslog/TxnStateKind.java} | 14 +-
.../java/accord/impl/progresslog/WaitingState.java | 665 +++++++++++++++++++
.../src/main/java/accord/local/Bootstrap.java | 16 +-
.../src/main/java/accord/local/Cleanup.java | 12 +-
.../src/main/java/accord/local/Command.java | 320 +++++----
.../src/main/java/accord/local/CommandStore.java | 27 +-
.../src/main/java/accord/local/CommandStores.java | 46 +-
.../src/main/java/accord/local/Commands.java | 240 ++++---
.../main/java/accord/local/CommonAttributes.java | 69 +-
.../src/main/java/accord/local/Listeners.java | 95 ---
accord-core/src/main/java/accord/local/Node.java | 103 ++-
.../main/java/accord/local/NodeTimeService.java | 16 +-
.../main/java/accord/local/RedundantBefore.java | 88 ++-
.../src/main/java/accord/local/SafeCommand.java | 39 +-
.../main/java/accord/local/SafeCommandStore.java | 70 +-
.../src/main/java/accord/local/SaveStatus.java | 162 +++--
accord-core/src/main/java/accord/local/Status.java | 67 +-
.../main/java/accord/local/cfk/CommandsForKey.java | 473 +++++++++----
.../accord/local/cfk/CommandsForKeyUpdate.java | 4 +-
.../src/main/java/accord/local/cfk/NotifySink.java | 64 +-
.../main/java/accord/local/cfk/PostProcess.java | 124 +++-
.../src/main/java/accord/local/cfk/Pruning.java | 146 ++++-
.../java/accord/local/cfk/SafeCommandsForKey.java | 2 +-
.../src/main/java/accord/local/cfk/Updating.java | 97 +--
.../src/main/java/accord/local/cfk/Utils.java | 2 +-
.../java/accord/messages/AbstractEpochRequest.java | 8 +-
.../src/main/java/accord/messages/Accept.java | 20 +-
.../src/main/java/accord/messages/Apply.java | 29 +-
.../accord/messages/ApplyThenWaitUntilApplied.java | 4 +-
.../src/main/java/accord/messages/Await.java | 311 +++++++++
.../java/accord/messages/BeginInvalidation.java | 35 +-
.../main/java/accord/messages/BeginRecovery.java | 8 +-
.../messages/{GetDeps.java => CalculateDeps.java} | 39 +-
.../src/main/java/accord/messages/CheckStatus.java | 18 +-
.../src/main/java/accord/messages/Commit.java | 20 +-
.../java/accord/messages/GetEphemeralReadDeps.java | 14 +-
.../main/java/accord/messages/GetMaxConflict.java | 14 +-
.../main/java/accord/messages/InformDurable.java | 21 +-
.../java/accord/messages/InformHomeDurable.java | 80 ---
.../main/java/accord/messages/InformOfTxnId.java | 88 ---
.../main/java/accord/messages/LocalRequest.java | 43 --
.../src/main/java/accord/messages/MessageType.java | 19 +-
.../src/main/java/accord/messages/PreAccept.java | 76 ++-
.../src/main/java/accord/messages/Propagate.java | 329 +++++-----
.../src/main/java/accord/messages/ReadData.java | 146 +++--
.../java/accord/messages/ReadEphemeralTxnData.java | 2 +-
.../src/main/java/accord/messages/Request.java | 1 -
.../src/main/java/accord/messages/TxnRequest.java | 107 +--
.../main/java/accord/messages/WaitOnCommit.java | 217 ------
.../java/accord/messages/WaitUntilApplied.java | 11 +-
.../java/accord/primitives/AbstractRanges.java | 10 +-
.../accord/primitives/AbstractUnseekableKeys.java | 10 +-
.../src/main/java/accord/primitives/Deps.java | 18 +
.../main/java/accord/primitives/FullKeyRoute.java | 10 +
.../java/accord/primitives/FullRangeRoute.java | 10 +
.../src/main/java/accord/primitives/FullRoute.java | 2 +-
.../src/main/java/accord/primitives/KeyDeps.java | 1 +
.../src/main/java/accord/primitives/KeyRoute.java | 30 +-
.../src/main/java/accord/primitives/Keys.java | 14 +-
.../java/accord/primitives/PartialKeyRoute.java | 53 +-
.../java/accord/primitives/PartialRangeRoute.java | 62 +-
.../main/java/accord/primitives/PartialRoute.java | 2 +-
.../main/java/accord/primitives/Participants.java | 14 +-
.../main/java/accord/primitives/ProgressToken.java | 37 +-
.../src/main/java/accord/primitives/RangeDeps.java | 2 +-
.../main/java/accord/primitives/RangeRoute.java | 54 +-
.../src/main/java/accord/primitives/Ranges.java | 9 +
.../src/main/java/accord/primitives/Routables.java | 14 +
.../src/main/java/accord/primitives/Route.java | 17 +-
.../main/java/accord/primitives/RoutingKeys.java | 9 +
.../src/main/java/accord/primitives/Seekables.java | 4 +-
.../src/main/java/accord/primitives/Txn.java | 2 +-
.../src/main/java/accord/primitives/TxnId.java | 9 +-
.../main/java/accord/primitives/Unseekables.java | 12 +-
.../src/main/java/accord/topology/Shard.java | 5 +-
.../main/java/accord/topology/TopologyManager.java | 24 +-
.../java/accord/utils/IntrusivePriorityHeap.java | 300 +++++++++
.../src/main/java/accord/utils/Invariants.java | 46 +-
.../src/main/java/accord/utils/LogGroupTimers.java | 607 +++++++++++++++++
.../main/java/accord/utils/RelationMultiMap.java | 1 -
.../src/main/java/accord/utils/SortedArrays.java | 81 ++-
.../java/accord/utils/ThreadPoolScheduler.java | 6 +
.../java/accord/utils/btree/AbstractBTreeMap.java | 8 +-
.../src/main/java/accord/utils/btree/BTree.java | 30 +-
.../main/java/accord/utils/btree/BTreeBiMap.java | 102 ---
.../src/main/java/accord/utils/btree/BTreeMap.java | 8 +-
.../java/accord/utils/btree/BTreeMultimap.java | 213 ------
.../main/java/accord/utils/btree/BTreeRemoval.java | 50 ++
.../java/accord/utils/btree/UpdateFunction.java | 39 +-
.../src/main/java/accord/utils/random/Picker.java | 7 +-
accord-core/src/test/java/accord/Utils.java | 20 +-
.../src/test/java/accord/burn/BurnTest.java | 50 +-
.../accord/burn/BurnTestConfigurationService.java | 6 -
.../src/test/java/accord/burn/TopologyUpdates.java | 2 +-
.../coordinate/CoordinateTransactionTest.java | 44 +-
.../test/java/accord/impl/LocalListenersTest.java | 528 +++++++++++++++
.../test/java/accord/impl/RemoteListenersTest.java | 423 ++++++++++++
.../src/test/java/accord/impl/TestAgent.java | 28 +
.../src/test/java/accord/impl/basic/Cluster.java | 111 +++-
.../accord/impl/basic/DelayedCommandStores.java | 15 +-
.../src/test/java/accord/impl/basic/Journal.java | 44 +-
.../src/test/java/accord/impl/basic/NodeSink.java | 1 -
.../java/accord/impl/basic/RandomDelayQueue.java | 4 +-
.../impl/basic/RecurringPendingRunnable.java | 13 +-
.../src/test/java/accord/impl/list/ListAgent.java | 45 +-
.../accord/impl/list/ListFetchCoordinator.java | 2 +-
.../test/java/accord/impl/list/ListRequest.java | 6 -
.../src/test/java/accord/impl/list/ListStore.java | 46 +-
.../test/java/accord/impl/mock/MockCluster.java | 17 +-
.../progresslog/PackedLogLinearIntegerTest.java | 70 ++
.../java/accord/local/BootstrapLocalTxnTest.java | 9 +-
.../test/java/accord/local/CheckedCommands.java | 34 +-
.../src/test/java/accord/local/CommandsTest.java | 5 -
.../java/accord/local/ImmutableCommandTest.java | 38 +-
.../src/test/java/accord/local/WaitingOnTest.java | 2 +-
.../java/accord/local/cfk/CommandsForKeyTest.java | 95 +--
.../test/java/accord/messages/PreAcceptTest.java | 5 +-
.../test/java/accord/messages/ReadDataTest.java | 19 +-
.../java/accord/primitives/AbstractRangesTest.java | 8 +
.../java/accord/topology/TopologyRandomizer.java | 6 +-
.../src/test/java/accord/utils/AccordGens.java | 1 -
.../test/java/accord/utils/ExtendedAssertions.java | 2 +-
.../test/java/accord/utils/LogGroupTimersTest.java | 302 +++++++++
.../src/test/java/accord/utils/MessageTask.java | 6 -
.../src/test/java/accord/utils/RangesTest.java | 14 +-
.../src/main/java/accord/maelstrom/Cluster.java | 27 +-
.../main/java/accord/maelstrom/MaelstromAgent.java | 30 +
.../java/accord/maelstrom/MaelstromRequest.java | 6 -
.../src/main/java/accord/maelstrom/Main.java | 16 +-
.../src/main/java/accord/maelstrom/Packet.java | 5 +-
182 files changed, 9585 insertions(+), 4082 deletions(-)
diff --git a/accord-core/src/main/java/accord/api/Agent.java b/accord-core/src/main/java/accord/api/Agent.java
index 93fe0164..116c8e4a 100644
--- a/accord-core/src/main/java/accord/api/Agent.java
+++ b/accord-core/src/main/java/accord/api/Agent.java
@@ -20,16 +20,24 @@ package accord.api;
import javax.annotation.Nonnull;
+import java.util.concurrent.TimeUnit;
+
+import accord.api.ProgressLog.BlockedUntil;
import accord.local.Command;
import accord.local.Node;
+import accord.local.SafeCommandStore;
+import accord.messages.ReplyContext;
import accord.primitives.Ranges;
import accord.primitives.Seekables;
import accord.primitives.Timestamp;
import accord.primitives.Txn;
import accord.primitives.TxnId;
+import accord.topology.Topologies;
/**
* Facility for augmenting node behaviour at specific points
+ *
+ * TODO (expected): rationalise LocalConfig and Agent
*/
public interface Agent extends UncaughtExceptionListener
{
@@ -100,4 +108,39 @@ public interface Agent extends UncaughtExceptionListener
{
return EventsListener.NOOP;
}
+
+ /**
+ * For each shard, select a small number of replicas that should be preferred for listening to progress updates
+ * from {@code from}. This should be 1-2 nodes that will be contacted preferentially for progress to minimise
+ * the number of messages we exchange. These nodes should be picked in a fashion so that there is a chain
+ * connecting all replicas of a shard together, e.g. in a ring picking the replicas directly behind you in the ring.
+ */
+ default Topologies selectPreferred(Node.Id from, Topologies to) { return to; }
+
+ long replyTimeout(ReplyContext replyContext, TimeUnit units);
+
+ /**
+ * This method permits implementations to configure the time at which a local home shard will attempt
+ * to coordinate a transaction to completion.
+ *
+ * This should aim to prevent two home replicas from attempting to initiate coordination at the same time.
+ */
+ long attemptCoordinationDelay(Node node, SafeCommandStore safeStore, TxnId txnId, TimeUnit units, int retryCount);
+
+ /**
+ * This method permits implementations to configure a delay for waiting to attempt to progress the local
+ * state machine for a transaction by querying its remote peers.
+ *
+ * This method should only attempt to minimise wasted work that would anyway be achieved by the transaction's
+ * coordinator, while ensuring prompt when the coordinator considers the transaction to be durable.
+ */
+ long seekProgressDelay(Node node, SafeCommandStore safeStore, TxnId txnId, int retryCount, BlockedUntil blockedUntil, TimeUnit units);
+
+ /**
+ * When a peer is queries for a local state, asynchronous callbacks may be registered.
+ * Any asynchronous reply is not guaranteed to be delivered, and only one attempt is made.
+ * This method configures a retry timeout on the node querying its peer to renew any callback registrations
+ * and re-query the local state.
+ */
+ long retryAwaitTimeout(Node node, SafeCommandStore safeStore, TxnId txnId, int retryCount, BlockedUntil retrying, TimeUnit units);
}
diff --git a/accord-core/src/main/java/accord/config/LocalConfig.java b/accord-core/src/main/java/accord/api/LocalConfig.java
similarity index 91%
rename from accord-core/src/main/java/accord/config/LocalConfig.java
rename to accord-core/src/main/java/accord/api/LocalConfig.java
index 7dfa5894..e8466cdc 100644
--- a/accord-core/src/main/java/accord/config/LocalConfig.java
+++ b/accord-core/src/main/java/accord/api/LocalConfig.java
@@ -16,7 +16,7 @@
* limitations under the License.
*/
-package accord.config;
+package accord.api;
import java.time.Duration;
@@ -24,11 +24,6 @@ public interface LocalConfig
{
LocalConfig DEFAULT = new LocalConfig() {};
- default Duration getProgressLogScheduleDelay()
- {
- return Duration.ofSeconds(1);
- }
-
// How long before we start notifying waiters on an epoch of timeout,
default Duration epochFetchInitialTimeout()
{
diff --git a/accord-core/src/main/java/accord/api/LocalListeners.java b/accord-core/src/main/java/accord/api/LocalListeners.java
new file mode 100644
index 00000000..861d5553
--- /dev/null
+++ b/accord-core/src/main/java/accord/api/LocalListeners.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.api;
+
+import accord.local.Command;
+import accord.local.CommandStore;
+import accord.local.SafeCommand;
+import accord.local.SafeCommandStore;
+import accord.local.SaveStatus;
+import accord.primitives.TxnId;
+
+public interface LocalListeners
+{
+ interface Factory
+ {
+ LocalListeners create(CommandStore store);
+ }
+
+ // to be used sparingly - much less efficient
+ interface ComplexListener
+ {
+ // return true if still listening, false if can be removed
+ boolean notify(SafeCommandStore safeStore, SafeCommand safeCommand);
+ }
+
+ interface Registered
+ {
+ // may be invoked from any thread
+ void cancel();
+ }
+
+ /**
+ * Cheap way to notify a transaction when a different transaction reaches a SaveStatus >= a target SaveStatus
+ */
+ void register(TxnId txnId, SaveStatus await, TxnId waiting);
+
+ /**
+ * Less efficient way to listen to a transaction that enables you to supply an arbitrary
+ * Java object as a listener instead of a transaction.
+ */
+ Registered register(TxnId txnId, ComplexListener listener);
+
+ /**
+ * Should forward notifications to the node's RemoteListeners object
+ */
+ void notify(SafeCommandStore safeStore, SafeCommand safeCommand, Command prev);
+}
diff --git a/accord-core/src/main/java/accord/api/ProgressLog.java b/accord-core/src/main/java/accord/api/ProgressLog.java
index c1fdb518..3a82ccd3 100644
--- a/accord-core/src/main/java/accord/api/ProgressLog.java
+++ b/accord-core/src/main/java/accord/api/ProgressLog.java
@@ -20,39 +20,47 @@ package accord.api;
import javax.annotation.Nullable;
-import accord.coordinate.InformHomeOfTxn;
import accord.local.Command;
import accord.local.CommandStore;
+import accord.local.Node;
import accord.local.SafeCommand;
-import accord.local.SaveStatus.LocalExecution;
+import accord.local.SafeCommandStore;
+import accord.local.SaveStatus;
+import accord.local.Status.Durability;
import accord.primitives.Participants;
import accord.primitives.Route;
+import accord.primitives.Timestamp;
import accord.primitives.TxnId;
+import accord.utils.SortedArrays;
+
+import static accord.api.ProgressLog.BlockedUntil.Query.HOME;
+import static accord.api.ProgressLog.BlockedUntil.Query.SHARD;
+import static accord.local.Status.Durability.Majority;
+import static accord.local.Status.Durability.NotDurable;
+import static accord.utils.SortedArrays.Search.FAST;
/**
- * This interface is responsible for managing incomplete transactions *and retrying them*.
- * Each stage is fenced by two methods, one entry and one exit. The entry method notifies the implementation
- * that it should soon be notified of the exit method, and if it is not that it should invoke some
- * pre-specified recovery mechanism.
- *
- * This is a per-CommandStore structure, with transactions primarily being managed by their home shard,
- * except during PreAccept as a transaction may not yet have been durably recorded by the home shard.
+ * This interface is responsible for managing incomplete transactions and ensuring they complete.
+ * This includes both ensuring a transaction is coordinated (normally handled by the home shard only)
+ * and ensuring the local command store executes the transaction.
*
* The basic logical flow for ensuring a transaction is committed and applied at all replicas is as follows:
*
- * - First, ensure a quorum of the home shard is aware of the transaction by invoking {@link InformHomeOfTxn}.
- * Entry by {@link #preaccepted} Non-home shards may now forget this transaction for replay purposes.
+ * - Members of the home shard will be informed of a transaction to monitor. This happens as soon as the
+ * home shard receives any Route. If this is not followed closely by {@link SaveStatus#Stable},
+ * {@link accord.coordinate.MaybeRecover} should be invoked to ensure the transaction is decided and executed.
+ *
+ * TODO (now): finish rewriting description and port to DefaultProgressLog
*
- * - Non-home shards may also be informed of transactions that are blocking the progress of other transactions.
- * If the {@code waitingOn} transaction that is blocking progress is uncommitted it is required that the progress
- * log invoke {@link accord.coordinate.FetchData#fetch} for the transaction if no {@link #stable} is witnessed.
+ * - Non-home shards that have not witnessed an Accept phase or later should inform the home shard of the transaction.
+ * This can be done at any time. The default implementation does this only when the transaction is blocking the
+ * progress of some other transaction.
*
- * - Members of the home shard will be informed of a transaction to monitor by the invocation of {@link #preaccepted} or
- * {@link #accepted}. If this is not followed closely by {@link #stable}, {@link accord.coordinate.MaybeRecover} should
- * be invoked.
+ * - If a {@code blockedBy} transaction is uncommitted it is required that the progress log invoke
+ * {@link accord.coordinate.FetchData#fetch} for the transaction if no {@link #stable} is witnessed.
*
* - Members of the home shard will later be informed that the transaction is {@link #readyToExecute}.
- * If this is not followed closely by {@link #executed}, {@link accord.coordinate.MaybeRecover} should be invoked.
+ * If this is not followed closely by {@link #preapplied}, {@link accord.coordinate.MaybeRecover} should be invoked.
*
* - Finally, it is up to each shard to independently coordinate disseminating the write to every replica.
*/
@@ -63,99 +71,89 @@ public interface ProgressLog
ProgressLog create(CommandStore store);
}
- enum ProgressShard
+ enum BlockedUntil
{
- /* We do not have enough information to say whether the shard is a progress shard or not */
- Unsure,
-
/**
- * This shard is not a progress shard
+ * Wait for the transaction to decide its executeAt (or else decide to be invalidated).
+ *
+ * This also waits for a FullRoute to be known.
*/
- No,
-
- /* Adhoc Local Progress Shard, i.e. where the local node is not a replica for the coordination epoch */
- Adhoc,
+ HasDecidedExecuteAt(HOME, HOME, SaveStatus.PreCommitted, NotDurable),
- /* Designated Local Progress Shard (selected from keys replicated locally at coordination epoch) */
- Local,
+ /**
+ * Wait for the transaction to be Committed.
+ *
+ * Note that we only set Committed during coordination, we do not propagate Committed directly between replicas,
+ * so for local progress it only makes sense to request HasStableDeps.
+ *
+ * This BlockedUntil is useful for remote listeners performing recovery that are waiting for transactions in
+ * the Accept phase that need to reach Committed to advance the recovery machine.
+ */
+ HasCommittedDeps(SHARD, SHARD, SaveStatus.Committed, NotDurable),
- /* Designated Home (Global Progress) Shard (if local node is a replica of home key on coordination epoch) */
- Home,
+ /**
+ * Wait for the transaction's dependencies to stabilise. This provides enough information
+ * to locally execute a transaction (if all the dependencies have applied).
+ */
+ HasStableDeps(SHARD, SHARD, SaveStatus.Stable, NotDurable),
- /** A special variant of Home that is not managed by the progress log */
- UnmanagedHome;
+ /**
+ * Wait for all shards to be ReadyToExecute so that a recovery coordinator may make progress
+ */
+ CanCoordinateExecution(SHARD, SHARD, SaveStatus.ReadyToExecute, NotDurable),
- public boolean isHome() { return this.compareTo(Home) >= 0; }
- public boolean isProgress() { return this.compareTo(Local) >= 0 && this.compareTo(Home) <= 0; }
+ /**
+ * Wait for the transaction to have enough information to apply.
+ * It does not need to be ready to apply yet.
+ */
+ CanApply(HOME, SHARD, SaveStatus.PreApplied, Majority);
+
+ public enum Query { HOME, SHARD }
+
+ private static final BlockedUntil[] lookup = values();
+ public final Query waitsOn, fetchFrom;
+ public final SaveStatus minSaveStatus;
+ // have remote listeners wait for the Durability before responding
+ // this permits us to wait for only the home shard for CanApply
+ public final Durability remoteDurability;
+
+ BlockedUntil(Query waitsOn, Query fetchFrom, SaveStatus minSaveStatus, Durability remoteDurability)
+ {
+ this.waitsOn = waitsOn;
+ this.fetchFrom = fetchFrom;
+ this.minSaveStatus = minSaveStatus;
+ this.remoteDurability = remoteDurability;
+ }
+
+ public long fetchEpoch(TxnId txnId, Timestamp executeAt)
+ {
+ if (this != CanApply || executeAt == null || executeAt.equals(Timestamp.NONE))
+ return txnId.epoch();
+
+ return executeAt.epoch();
+ }
+
+ public static BlockedUntil forOrdinal(int ordinal)
+ {
+ if (ordinal < 0 || ordinal > lookup.length)
+ throw new IndexOutOfBoundsException(ordinal);
+ return lookup[ordinal];
+ }
+
+ public static BlockedUntil forSaveStatus(SaveStatus saveStatus)
+ {
+ int i = SortedArrays.binarySearch(lookup, 0, lookup.length, saveStatus, (s, w) -> s.compareTo(w.minSaveStatus), FAST);
+ if (i < 0) i = Math.max(0, -2 - i);
+ return lookup[i];
+ }
}
/**
- * Has not been pre-accepted, but has been witnessed by ourselves (only partially) or another node that has informed us
- *
- * A home shard should monitor this transaction for global progress.
- * A non-home shard should not receive this message.
- */
- void unwitnessed(TxnId txnId, ProgressShard shard);
-
- /**
- * Has been pre-accepted.
- *
- * A home shard should monitor this transaction for global progress.
- * A non-home shard should begin monitoring this transaction only to ensure it reaches the Accept phase, or is
- * witnessed by a majority of the home shard.
- */
- void preaccepted(Command command, ProgressShard shard);
-
- /**
- * Has been accepted
- *
- * A home shard should monitor this transaction for global progress.
- * A non-home shard can safely ignore this transaction, as it has been witnessed by a majority of the home shard.
- */
- void accepted(Command command, ProgressShard shard);
-
- /**
- * The transaction's executeAt is known. This may be enough to satisfy some waiting transactions.
- */
- void precommitted(Command command);
-
- /**
- * Has stable dependencies, so should execute soon.
- *
- * A home shard should monitor this transaction for global progress.
- * A non-home shard can safely ignore this transaction, as it has been witnessed by a majority of the home shard.
- */
- void stable(Command command, ProgressShard shard);
-
- /**
- * The transaction is waiting to make progress, as all local dependencies have applied.
- *
- * A home shard should monitor this transaction for global progress.
- * A non-home shard can safely ignore this transaction, as it has been witnessed by a majority of the home shard.
- */
- void readyToExecute(Command command);
-
- /**
- * The transaction's outcome has been durably recorded (but not necessarily applied) locally.
- * It will be applied once all local dependencies have been.
- *
- * Invoked on both home and non-home command stores, and is required to trigger per-shard processes
- * that ensure the transaction's outcome is durably persisted on all replicas of the shard.
- *
- * May also permit aborting a pending waitingOn-triggered event.
+ * Record an updated local status for the transaction, to clear any waiting state it satisfies.
*/
- void executed(Command command, ProgressShard shard);
+ void update(SafeCommandStore safeStore, TxnId txnId, Command before, Command after);
- /**
- * The transaction's outcome has been durably recorded (but not necessarily applied) at a quorum of all shards.
- *
- * If this replica has not witnessed the outcome of the transaction, it should poll a majority of each shard
- * for its outcome.
- *
- * Otherwise, this transaction no longer needs to be monitored, but implementations may wish to ensure that
- * the result is propagated to every live replica.
- */
- void durable(Command command);
+ void remoteCallback(SafeCommandStore safeStore, SafeCommand safeCommand, SaveStatus remoteStatus, int callbackId, Node.Id from);
/**
* The parameter is a command that some other command's execution is most proximally blocked by.
@@ -172,43 +170,26 @@ public interface ProgressLog
* to either complete or invalidate it, so that we may make progress.
* <p>
* In all other scenarios, the implementation is free to choose its course of action.
- *
- * Either blockedOnRoute or blockedOnParticipants should be non-null.
- *
- * @param blockedBy is the transaction id that is blocking progress
- * @param blockedUntil either Committed or Executed; the state we are waiting for
- * @param blockedOnRoute the route (if any) we are blocked on execution for
- * @param blockedOnParticipants the participating keys on which we are blocked for execution
- */
- void waiting(SafeCommand blockedBy, LocalExecution blockedUntil, @Nullable Route<?> blockedOnRoute, @Nullable Participants<?> blockedOnParticipants);
-
- /**
- * The parameter is a command that some other command's execution is most proximally blocked by.
- * This may be invoked by either the home or non-home command store.
- * <p>
- * If invoked by the non-home command store for a {@code blockedBy} transaction that has not yet been committed, this
- * must eventually trigger contact with the home shard of this {@code blockedBy} transaction in order to check on the
- * transaction's progress (unless the transaction is committed first). This is to avoid unnecessary additional messages
- * being exchanged in the common case, where a transaction may be committed successfully to members of its home shard,
- * but not to all non-home shards. In such a case the transaction may be a false-dependency of another transaction that
- * needs to perform a read, and all nodes which may do so are waiting for the commit record to arrive.
* <p>
- * If a quorum of the home shard does not know of the transaction, then we can ask the home shard to perform recovery
- * to either complete or invalidate it, so that we may make progress.
- * <p>
- * In all other scenarios, the implementation is free to choose its course of action.
- *
* Either blockedOnRoute or blockedOnParticipants should be non-null.
*
+ * @param blockedUntil what we are waiting for
* @param blockedBy is the transaction id that is blocking progress
- * @param blockedUntil either Committed or Executed; the state we are waiting for
* @param blockedOnRoute the route (if any) we are blocked on execution for
* @param blockedOnParticipants the participating keys on which we are blocked for execution
*/
- void waiting(TxnId blockedBy, LocalExecution blockedUntil, @Nullable Route<?> blockedOnRoute, @Nullable Participants<?> blockedOnParticipants);
+ void waiting(BlockedUntil blockedUntil, SafeCommandStore safeStore, SafeCommand blockedBy, @Nullable Route<?> blockedOnRoute, @Nullable Participants<?> blockedOnParticipants);
/**
* We have finished processing this transaction; ensure its state is cleared
*/
void clear(TxnId txnId);
+
+ class NoOpProgressLog implements ProgressLog
+ {
+ @Override public void update(SafeCommandStore safeStore, TxnId txnId, Command before, Command after) {}
+ @Override public void remoteCallback(SafeCommandStore safeStore, SafeCommand safeCommand, SaveStatus remoteStatus, int callbackId, Node.Id from) {}
+ @Override public void waiting(BlockedUntil blockedUntil, SafeCommandStore safeStore, SafeCommand blockedBy, Route<?> blockedOnRoute, Participants<?> blockedOnParticipants) {}
+ @Override public void clear(TxnId txnId) {}
+ }
}
diff --git a/accord-core/src/main/java/accord/api/RemoteListeners.java b/accord-core/src/main/java/accord/api/RemoteListeners.java
new file mode 100644
index 00000000..06fdad13
--- /dev/null
+++ b/accord-core/src/main/java/accord/api/RemoteListeners.java
@@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.api;
+
+import accord.local.Command;
+import accord.local.Node;
+import accord.local.SafeCommand;
+import accord.local.SafeCommandStore;
+import accord.local.SaveStatus;
+import accord.local.Status.Durability;
+import accord.primitives.TxnId;
+
+/**
+ * A collection of remote listeners awaiting some set of local responses from command stores.
+ * Registrations here are permanent (don't time out) and one shot; if the reply is lost the remote listener will need to be re-registered.
+ * Remote listeners may periodically poll to check the listeners are still registered.
+ */
+public interface RemoteListeners
+{
+ interface Factory
+ {
+ RemoteListeners create(Node node);
+ }
+
+ /**
+ * An in-progress registration that collects command stores it is waiting on.
+ * This may be cancelled until done() is invoked, at which point the registration is permanent.
+ */
+ interface Registration
+ {
+ // thread safe
+ void add(SafeCommandStore safeStore, SafeCommand safeCommand);
+
+ // thread unsafe
+ int done();
+ }
+
+ /**
+ * CallbackId must be a non-negative integer
+ */
+ Registration register(TxnId txnId, SaveStatus awaitSaveStatus, Durability awaitDurability, Node.Id listener, int callbackId);
+ void notify(SafeCommandStore safeStore, SafeCommand safeCommand, Command prev);
+
+ class NoOpRemoteListeners implements RemoteListeners
+ {
+ static class NoOpRegistration implements Registration
+ {
+ @Override public void add(SafeCommandStore safeStore, SafeCommand safeCommand) {}
+ @Override public int done() { return 0; }
+ }
+
+ @Override public Registration register(TxnId txnId, SaveStatus awaitSaveStatus, Durability awaitDurability, Node.Id listener, int callbackId) { return new NoOpRegistration();}
+ @Override public void notify(SafeCommandStore safeStore, SafeCommand safeCommand, Command prev) {}
+ }
+}
diff --git a/accord-core/src/main/java/accord/config/MutableLocalConfig.java b/accord-core/src/main/java/accord/api/RequestTimeouts.java
similarity index 60%
rename from accord-core/src/main/java/accord/config/MutableLocalConfig.java
rename to accord-core/src/main/java/accord/api/RequestTimeouts.java
index f3c42782..e1af5efe 100644
--- a/accord-core/src/main/java/accord/config/MutableLocalConfig.java
+++ b/accord-core/src/main/java/accord/api/RequestTimeouts.java
@@ -16,25 +16,21 @@
* limitations under the License.
*/
-package accord.config;
+package accord.api;
-import java.time.Duration;
+import java.util.concurrent.TimeUnit;
-import com.google.common.annotations.VisibleForTesting;
-
-@VisibleForTesting
-public class MutableLocalConfig implements LocalConfig
+public interface RequestTimeouts
{
- private volatile Duration progressLogScheduleDelay = LocalConfig.super.getProgressLogScheduleDelay();
-
- public void setProgressLogScheduleDelay(Duration progressLogScheduleDelay)
+ interface RegisteredTimeout
{
- this.progressLogScheduleDelay = progressLogScheduleDelay;
+ void cancel();
}
- @Override
- public Duration getProgressLogScheduleDelay()
+ interface Timeout
{
- return progressLogScheduleDelay;
+ void timeout();
}
+
+ RegisteredTimeout register(Timeout timeout, long delay, TimeUnit units);
}
diff --git a/accord-core/src/main/java/accord/api/Scheduler.java b/accord-core/src/main/java/accord/api/Scheduler.java
index 34ab7f0c..77a12c3d 100644
--- a/accord-core/src/main/java/accord/api/Scheduler.java
+++ b/accord-core/src/main/java/accord/api/Scheduler.java
@@ -25,7 +25,20 @@ import java.util.concurrent.TimeUnit;
*/
public interface Scheduler
{
- Scheduled CANCELLED = () -> {};
+ Scheduled CANCELLED = new Scheduled()
+ {
+ @Override
+ public void cancel()
+ {
+ }
+
+ @Override
+ public boolean isDone()
+ {
+ return true;
+ }
+ };
+
Scheduler NEVER_RUN_SCHEDULED = new Scheduler()
{
@Override
@@ -50,6 +63,7 @@ public interface Scheduler
interface Scheduled
{
void cancel();
+ boolean isDone();
}
default Scheduled recurring(Runnable run) { return recurring(run, 1L, TimeUnit.SECONDS); }
diff --git a/accord-core/src/main/java/accord/coordinate/AsynchronousAwait.java b/accord-core/src/main/java/accord/coordinate/AsynchronousAwait.java
new file mode 100644
index 00000000..4d40a4cd
--- /dev/null
+++ b/accord-core/src/main/java/accord/coordinate/AsynchronousAwait.java
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.coordinate;
+
+import java.util.function.BiConsumer;
+import javax.annotation.Nullable;
+
+import accord.api.ProgressLog.BlockedUntil;
+import accord.coordinate.tracking.AwaitTracker;
+import accord.coordinate.tracking.RequestStatus;
+import accord.local.Commands;
+import accord.local.Node;
+import accord.local.Node.Id;
+import accord.messages.Await;
+import accord.messages.Await.AwaitOk;
+import accord.messages.Callback;
+import accord.primitives.Route;
+import accord.primitives.TxnId;
+import accord.primitives.Unseekables;
+import accord.topology.Topologies;
+
+import static accord.coordinate.tracking.RequestStatus.Success;
+
+/**
+ * Perform a distributed wait operation: each commandStore that is contacted either reports that the relevant key(s)
+ * meet the required wait condition, or else registers a callback that will be notified once the wait condition is met
+ * on the replica for those keys. The response informs the initiating operation which keys met the wait condition
+ * before replying, and which are awaiting a distributed response.
+ *
+ * Note that remote listeners may have been registered regardless, as replicas may not be in agreement for each key.
+ * Any listener must be resilient to redundant callbacks.
+ *
+ * Asynchronous awaits will not time out if the wait is longer than message/request timeouts.
+ */
+public class AsynchronousAwait implements Callback<AwaitOk>
+{
+ // TODO (desired, efficiency): this should collect the executeAt of any commit, and terminate as soon as one is found
+ // that is earlier than TxnId for the Txn we are recovering; if all commits we wait for
+ // are given earlier timestamps we can retry without restarting.
+
+ public static class SynchronousResult
+ {
+ public final Unseekables<?> ready;
+ public final @Nullable Unseekables<?> notReady;
+
+ public SynchronousResult(Unseekables<?> ready, @Nullable Unseekables<?> notReady)
+ {
+ this.ready = ready;
+ this.notReady = notReady;
+ }
+ }
+
+ final TxnId txnId;
+ final Unseekables<?> contact;
+ final AwaitTracker tracker;
+ final int asynchronousCallbackId;
+ final BiConsumer<SynchronousResult, Throwable> synchronousCallback;
+ private boolean isDone;
+ private Throwable failure;
+
+ public AsynchronousAwait(TxnId txnId, Unseekables<?> contact, AwaitTracker tracker, int asynchronousCallbackId, BiConsumer<SynchronousResult, Throwable> synchronousCallback)
+ {
+ this.txnId = txnId;
+ this.contact = contact;
+ this.tracker = tracker;
+ this.asynchronousCallbackId = asynchronousCallbackId;
+ this.synchronousCallback = synchronousCallback;
+ }
+
+ /**
+ * we require a Route to contact so we can be sure a home shard recipient invokes {@link Commands#updateRouteOrParticipants},
+ * notifying the progress log of a Route to determine it is the home shard.
+ */
+ public static AsynchronousAwait awaitAny(Node node, Topologies topologies, TxnId txnId, Route<?> contact, BlockedUntil awaiting, int asynchronousCallbackId, BiConsumer<SynchronousResult, Throwable> synchronousCallback)
+ {
+ AwaitTracker tracker = new AwaitTracker(topologies);
+ AsynchronousAwait result = new AsynchronousAwait(txnId, contact, tracker, asynchronousCallbackId, synchronousCallback);
+ result.start(node, topologies, contact, awaiting);
+ return result;
+ }
+
+ private void start(Node node, Topologies topologies, Route<?> route, BlockedUntil blockedUntil)
+ {
+ node.send(topologies.nodes(), to -> new Await(to, topologies, txnId, route, blockedUntil, asynchronousCallbackId), this);
+ }
+
+ @Override
+ public synchronized void onSuccess(Id from, AwaitOk reply)
+ {
+ if (isDone) return;
+
+ if (tracker.recordSuccess(from, reply == AwaitOk.Ready) == Success)
+ onSuccess();
+ }
+
+ @Override
+ public synchronized void onFailure(Id from, Throwable failure)
+ {
+ if (isDone) return;
+
+ this.failure = FailureAccumulator.append(this.failure, failure);
+ RequestStatus status = tracker.recordFailure(from);
+ switch (status)
+ {
+ default: throw new AssertionError("Unhandled RequestStatus: " + status);
+ case NoChange: break;
+ case Success:
+ onSuccess();
+ break;
+ case Failed:
+ isDone = true;
+ synchronousCallback.accept(null, this.failure);
+ }
+ }
+
+ private void onSuccess()
+ {
+ Unseekables<?> ready = tracker.ready(contact);
+ Unseekables<?> notReady = tracker.notReady(contact);
+ if (notReady.isEmpty())
+ notReady = null;
+ isDone = true;
+ synchronousCallback.accept(new SynchronousResult(ready, notReady), null);
+ }
+
+ @Override
+ public synchronized void onCallbackFailure(Id from, Throwable failure)
+ {
+ if (isDone) return;
+
+ isDone = true;
+ synchronousCallback.accept(null, failure);
+ }
+}
+
diff --git a/accord-core/src/main/java/accord/coordinate/Barrier.java b/accord-core/src/main/java/accord/coordinate/Barrier.java
index fbc9bcbd..bd21ea4a 100644
--- a/accord-core/src/main/java/accord/coordinate/Barrier.java
+++ b/accord-core/src/main/java/accord/coordinate/Barrier.java
@@ -26,17 +26,18 @@ import org.slf4j.LoggerFactory;
import accord.api.BarrierType;
import accord.api.Key;
+import accord.api.LocalListeners;
import accord.api.RoutingKey;
import accord.local.Command;
import accord.local.KeyHistory;
import accord.local.Node;
-import accord.local.PreLoadContext;
import accord.local.SafeCommand;
import accord.local.SafeCommandStore;
import accord.local.SafeCommandStore.TestDep;
import accord.local.SafeCommandStore.TestStartedAt;
import accord.local.Status;
import accord.primitives.Routable.Domain;
+import accord.primitives.RoutableKey;
import accord.primitives.Seekables;
import accord.primitives.SyncPoint;
import accord.primitives.Timestamp;
@@ -180,8 +181,7 @@ public class Barrier<S extends Seekables<?, ?>> extends AsyncResults.AbstractRes
TxnId txnId = syncPoint.syncId;
long epoch = txnId.epoch();
RoutingKey homeKey = syncPoint.homeKey;
- node.commandStores().ifLocal(contextFor(txnId), homeKey, epoch, epoch,
- safeStore -> safeStore.get(txnId, homeKey).addAndInvokeListener(safeStore, new BarrierCommandListener()))
+ node.commandStores().ifLocal(contextFor(txnId), homeKey, epoch, epoch, safeStore -> register(safeStore, txnId, homeKey))
.begin(node.agent());
}
else
@@ -191,10 +191,10 @@ public class Barrier<S extends Seekables<?, ?>> extends AsyncResults.AbstractRes
});
}
- private class BarrierCommandListener implements Command.TransientListener
+ private class BarrierCommandListener implements LocalListeners.ComplexListener
{
@Override
- public void onChange(SafeCommandStore safeStore, SafeCommand safeCommand)
+ public boolean notify(SafeCommandStore safeStore, SafeCommand safeCommand)
{
Command command = safeCommand.current();
if (command.is(Status.Applied))
@@ -202,17 +202,15 @@ public class Barrier<S extends Seekables<?, ?>> extends AsyncResults.AbstractRes
// In all the cases where we add a listener (listening to existing command, async completion of CoordinateSyncPoint)
// we want to notify the agent
doBarrierSuccess(command.txnId());
+ return false;
}
else if (command.hasBeen(Status.Truncated))
+ {
// Surface the invalidation/truncation and the barrier can be retried by the caller
Barrier.this.tryFailure(new Timeout(command.txnId(), command.homeKey()));
-
- }
-
- @Override
- public PreLoadContext listenerPreLoadContext(TxnId caller)
- {
- return PreLoadContext.contextFor(caller);
+ return false;
+ }
+ return true;
}
}
@@ -230,6 +228,12 @@ public class Barrier<S extends Seekables<?, ?>> extends AsyncResults.AbstractRes
return check;
}
+ private void register(SafeCommandStore safeStoreWithTxn, TxnId txnId, RoutableKey key)
+ {
+ BarrierCommandListener listener = new BarrierCommandListener();
+ safeStoreWithTxn.registerAndInvoke(txnId, key.toUnseekable(), listener);
+ }
+
// Hold result of looking for a transaction to act as a barrier for an Epoch
static class BarrierTxn
{
@@ -283,10 +287,11 @@ public class Barrier<S extends Seekables<?, ?>> extends AsyncResults.AbstractRes
// It's not applied so add a listener to find out when it is applied
if (found != null)
{
- safeStore.commandStore().execute(
- contextFor(found.txnId),
- safeStoreWithTxn -> safeStoreWithTxn.get(found.txnId, found.key.toUnseekable()).addAndInvokeListener(safeStore, new BarrierCommandListener())
- ).begin(node.agent());
+ //noinspection SillyAssignment,ConstantConditions
+ safeStore = safeStore; // prevent use in lambda
+ safeStore.commandStore()
+ .execute(contextFor(found.txnId), safeStoreWithTxn -> register(safeStoreWithTxn, found.txnId, found.key))
+ .begin(node.agent());
}
return found;
}
diff --git a/accord-core/src/main/java/accord/coordinate/CheckShards.java b/accord-core/src/main/java/accord/coordinate/CheckShards.java
index 84ab1b6f..a778325e 100644
--- a/accord-core/src/main/java/accord/coordinate/CheckShards.java
+++ b/accord-core/src/main/java/accord/coordinate/CheckShards.java
@@ -90,9 +90,6 @@ public abstract class CheckShards<U extends Unseekables<?>> extends ReadCoordina
@Override
protected Action process(Id from, CheckStatusReply reply)
{
- if (debug != null)
- debug.put(from, reply);
-
if (reply.isOk())
{
CheckStatusOk ok = (CheckStatusOk) reply;
diff --git a/accord-core/src/main/java/accord/coordinate/CollectDeps.java b/accord-core/src/main/java/accord/coordinate/CollectCalculatedDeps.java
similarity index 76%
rename from accord-core/src/main/java/accord/coordinate/CollectDeps.java
rename to accord-core/src/main/java/accord/coordinate/CollectCalculatedDeps.java
index 31e2308b..e715001a 100644
--- a/accord-core/src/main/java/accord/coordinate/CollectDeps.java
+++ b/accord-core/src/main/java/accord/coordinate/CollectCalculatedDeps.java
@@ -28,28 +28,27 @@ import accord.local.CommandStore;
import accord.local.Node;
import accord.local.Node.Id;
import accord.messages.Callback;
-import accord.messages.GetDeps;
-import accord.messages.GetDeps.GetDepsOk;
+import accord.messages.CalculateDeps;
+import accord.messages.CalculateDeps.CalculateDepsOk;
import accord.primitives.*;
import accord.topology.Topologies;
import static accord.coordinate.tracking.RequestStatus.Failed;
import static accord.coordinate.tracking.RequestStatus.Success;
-public class CollectDeps implements Callback<GetDepsOk>
+public class CollectCalculatedDeps implements Callback<CalculateDepsOk>
{
final Node node;
final TxnId txnId;
final RoutingKey homeKey;
-
final Timestamp executeAt;
- private final List<GetDepsOk> oks;
+ private final List<CalculateDepsOk> oks;
private final QuorumTracker tracker;
private final BiConsumer<Deps, Throwable> callback;
private boolean isDone;
- CollectDeps(Node node, Topologies topologies, TxnId txnId, RoutingKey homeKey, Timestamp executeAt, BiConsumer<Deps, Throwable> callback)
+ CollectCalculatedDeps(Node node, Topologies topologies, TxnId txnId, RoutingKey homeKey, Timestamp executeAt, BiConsumer<Deps, Throwable> callback)
{
this.node = node;
this.txnId = txnId;
@@ -60,19 +59,19 @@ public class CollectDeps implements Callback<GetDepsOk>
this.tracker = new QuorumTracker(topologies);
}
- public static void withDeps(Node node, TxnId txnId, FullRoute<?> fullRoute, Unseekables<?> sendTo, Seekables<?, ?> keysOrRanges, Timestamp executeAt, BiConsumer<Deps, Throwable> callback)
+ public static void withCalculatedDeps(Node node, TxnId txnId, FullRoute<?> fullRoute, Unseekables<?> sendTo, Seekables<?, ?> keysOrRanges, Timestamp executeAt, BiConsumer<Deps, Throwable> callback)
{
Topologies topologies = node.topology().withUnsyncedEpochs(sendTo, txnId, executeAt);
- CollectDeps collect = new CollectDeps(node, topologies, txnId, fullRoute.homeKey(), executeAt, callback);
+ CollectCalculatedDeps collect = new CollectCalculatedDeps(node, topologies, txnId, fullRoute.homeKey(), executeAt, callback);
CommandStore store = CommandStore.maybeCurrent();
if (store == null)
store = node.commandStores().select(fullRoute);
- node.send(collect.tracker.nodes(), to -> new GetDeps(to, topologies, fullRoute, txnId, keysOrRanges, executeAt),
+ node.send(collect.tracker.nodes(), to -> new CalculateDeps(to, topologies, fullRoute, txnId, keysOrRanges, executeAt),
store, collect);
}
@Override
- public void onSuccess(Id from, GetDepsOk ok)
+ public void onSuccess(Id from, CalculateDepsOk ok)
{
if (isDone)
return;
diff --git a/accord-core/src/main/java/accord/coordinate/CoordinateGloballyDurable.java b/accord-core/src/main/java/accord/coordinate/CoordinateGloballyDurable.java
index 94528bb8..0cc51c57 100644
--- a/accord-core/src/main/java/accord/coordinate/CoordinateGloballyDurable.java
+++ b/accord-core/src/main/java/accord/coordinate/CoordinateGloballyDurable.java
@@ -80,7 +80,7 @@ public class CoordinateGloballyDurable extends SettableResult<Void> implements C
public void onFailure(Node.Id from, Throwable failure)
{
if (tracker.recordFailure(from) == RequestStatus.Failed)
- tryFailure(new Exhausted(null, null));
+ tryFailure(new Exhausted(null, null, null));
}
@Override
diff --git a/accord-core/src/main/java/accord/coordinate/CoordinateShardDurable.java b/accord-core/src/main/java/accord/coordinate/CoordinateShardDurable.java
index 73ba7f2c..a055f1f6 100644
--- a/accord-core/src/main/java/accord/coordinate/CoordinateShardDurable.java
+++ b/accord-core/src/main/java/accord/coordinate/CoordinateShardDurable.java
@@ -18,7 +18,7 @@
package accord.coordinate;
-import accord.coordinate.tracking.AppliedTracker;
+import accord.coordinate.tracking.AllTracker;
import accord.local.Node;
import accord.messages.Callback;
import accord.messages.ReadData.ReadReply;
@@ -32,7 +32,7 @@ public class CoordinateShardDurable extends ExecuteSyncPoint<Ranges> implements
{
private CoordinateShardDurable(Node node, SyncPoint<Ranges> exclusiveSyncPoint)
{
- super(node, new AppliedTracker(node.topology().forEpoch(exclusiveSyncPoint.keysOrRanges, exclusiveSyncPoint.sourceEpoch())), exclusiveSyncPoint);
+ super(node, new AllTracker(node.topology().forEpoch(exclusiveSyncPoint.keysOrRanges, exclusiveSyncPoint.sourceEpoch())), exclusiveSyncPoint);
}
public static AsyncResult<SyncPoint<Ranges>> coordinate(Node node, SyncPoint<Ranges> exclusiveSyncPoint)
diff --git a/accord-core/src/main/java/accord/coordinate/Exhausted.java b/accord-core/src/main/java/accord/coordinate/Exhausted.java
index 6733ce38..b116748b 100644
--- a/accord-core/src/main/java/accord/coordinate/Exhausted.java
+++ b/accord-core/src/main/java/accord/coordinate/Exhausted.java
@@ -21,6 +21,7 @@ package accord.coordinate;
import javax.annotation.Nullable;
import accord.api.RoutingKey;
+import accord.primitives.Ranges;
import accord.primitives.TxnId;
import static accord.utils.Invariants.checkState;
@@ -30,25 +31,32 @@ import static accord.utils.Invariants.checkState;
*/
public class Exhausted extends CoordinationFailed
{
- public Exhausted(TxnId txnId, @Nullable RoutingKey homeKey)
+ final Ranges unavailable;
+ private String message;
+
+ public Exhausted(TxnId txnId, @Nullable RoutingKey homeKey, Ranges unavailable)
{
super(txnId, homeKey);
+ this.unavailable = unavailable;
}
- public Exhausted(TxnId txnId, @Nullable RoutingKey homeKey, String message)
+ public String getMessage()
{
- super(txnId, homeKey, message);
+ if (message == null)
+ message = unavailable == null ? "No more nodes to try" : "No more nodes to try for: " + unavailable;
+ return message;
}
- protected Exhausted(TxnId txnId, @Nullable RoutingKey homeKey, Exhausted cause)
+ Exhausted(TxnId txnId, @Nullable RoutingKey homeKey, Ranges unavailable, Exhausted cause)
{
super(txnId, homeKey, cause);
+ this.unavailable = unavailable;
}
@Override
public Exhausted wrap()
{
checkState(this.getClass() == Exhausted.class);
- return new Exhausted(txnId(), homeKey(), this);
+ return new Exhausted(txnId(), homeKey(), unavailable, this);
}
}
diff --git a/accord-core/src/main/java/accord/coordinate/FailureAccumulator.java b/accord-core/src/main/java/accord/coordinate/FailureAccumulator.java
index c5b537da..c2112d2e 100644
--- a/accord-core/src/main/java/accord/coordinate/FailureAccumulator.java
+++ b/accord-core/src/main/java/accord/coordinate/FailureAccumulator.java
@@ -22,6 +22,7 @@ import java.util.function.Predicate;
import javax.annotation.Nullable;
import accord.api.RoutingKey;
+import accord.primitives.Ranges;
import accord.primitives.TxnId;
public class FailureAccumulator
@@ -59,11 +60,11 @@ public class FailureAccumulator
return createFailure(current, txnId, homeKey, null);
}
- public static CoordinationFailed createFailure(@Nullable Throwable current, TxnId txnId, @Nullable RoutingKey homeKey, @Nullable String exhaustedMsg)
+ public static CoordinationFailed createFailure(@Nullable Throwable current, TxnId txnId, @Nullable RoutingKey homeKey, @Nullable Ranges unavailable)
{
- if (current == null) return new Exhausted(txnId, homeKey, exhaustedMsg);
+ if (current == null) return new Exhausted(txnId, homeKey, unavailable);
if (isTimeout(current)) return new Timeout(txnId, homeKey);
- Exhausted e = new Exhausted(txnId, homeKey, exhaustedMsg);
+ Exhausted e = new Exhausted(txnId, homeKey, unavailable);
e.initCause(current);
return e;
}
diff --git a/accord-core/src/main/java/accord/coordinate/FetchCoordinator.java b/accord-core/src/main/java/accord/coordinate/FetchCoordinator.java
index b33192aa..266321f2 100644
--- a/accord-core/src/main/java/accord/coordinate/FetchCoordinator.java
+++ b/accord-core/src/main/java/accord/coordinate/FetchCoordinator.java
@@ -84,21 +84,21 @@ public abstract class FetchCoordinator
contacted = contacted.with(ranges);
inflight = inflight.with(ranges);
waitingOn = waitingOn.with(ranges);
- uncontacted = uncontacted.subtract(ranges);
+ uncontacted = uncontacted.without(ranges);
}
void slow(Ranges ranges)
{
checkArgument(waitingOn.containsAll(ranges));
- waitingOn = waitingOn.subtract(ranges);
+ waitingOn = waitingOn.without(ranges);
}
void successful(Ranges ranges)
{
checkArgument(inflight.containsAll(ranges));
successful = successful.with(ranges);
- inflight = inflight.subtract(ranges);
- waitingOn = waitingOn.subtract(ranges);
+ inflight = inflight.without(ranges);
+ waitingOn = waitingOn.without(ranges);
}
Ranges waitingOn(Ranges ranges)
@@ -110,8 +110,8 @@ public abstract class FetchCoordinator
{
checkArgument(inflight.containsAll(ranges));
unsuccessful = unsuccessful.with(ranges);
- inflight = inflight.subtract(ranges);
- waitingOn = waitingOn.subtract(ranges);
+ inflight = inflight.without(ranges);
+ waitingOn = waitingOn.without(ranges);
}
void fail()
@@ -204,13 +204,13 @@ public abstract class FetchCoordinator
// some portion of the range is completely unavailable
isDone = true;
- failures = FailureAccumulator.createFailure(failures, syncPoint.syncId, null, "No more nodes to contact for " + needed);
+ failures = FailureAccumulator.createFailure(failures, syncPoint.syncId, null, needed);
onDone(success, failures);
}
protected void exhausted(Ranges exhausted)
{
- fetchRanges.fail(exhausted, new Exhausted(syncPoint.syncId, null, "No more nodes to contact for " + exhausted));
+ fetchRanges.fail(exhausted, new Exhausted(syncPoint.syncId, null, exhausted));
}
// this method can be completely overridden by an implementing class, which may simply call contact()
@@ -227,7 +227,7 @@ public abstract class FetchCoordinator
++inflight;
state.contact(contact);
contact(state.id, contact);
- needed = needed.subtract(contact);
+ needed = needed.without(contact);
if (needed.isEmpty())
break;
}
diff --git a/accord-core/src/main/java/accord/coordinate/FetchData.java b/accord-core/src/main/java/accord/coordinate/FetchData.java
index f71fb9cd..387a2d05 100644
--- a/accord-core/src/main/java/accord/coordinate/FetchData.java
+++ b/accord-core/src/main/java/accord/coordinate/FetchData.java
@@ -28,7 +28,6 @@ import accord.messages.CheckStatus.CheckStatusOkFull;
import accord.messages.Propagate;
import accord.primitives.EpochSupplier;
import accord.primitives.FullRoute;
-import accord.primitives.PartialRoute;
import accord.primitives.Ranges;
import accord.primitives.Route;
import accord.primitives.Timestamp;
@@ -36,6 +35,8 @@ import accord.primitives.TxnId;
import accord.primitives.Unseekables;
import accord.utils.Invariants;
+import javax.annotation.Nonnull;
+
import static accord.coordinate.Infer.InvalidateAndCallback.locallyInvalidateAndCallback;
import static accord.primitives.Route.castToRoute;
import static accord.primitives.Route.isFullRoute;
@@ -48,25 +49,32 @@ import static accord.primitives.Route.isRoute;
*/
public class FetchData extends CheckShards<Route<?>>
{
- public static void fetch(Known fetch, Node node, TxnId txnId, Unseekables<?> someUnseekables, @Nullable EpochSupplier forLocalEpoch, @Nullable Timestamp executeAt, BiConsumer<? super Known, Throwable> callback)
+ public static class FetchResult
+ {
+ public final @Nonnull Known achieved;
+ public final Unseekables<?> achievedTarget;
+ public final @Nullable Unseekables<?> didNotAchieveTarget;
+
+ public FetchResult(Known achieved, Unseekables<?> achievedTarget, @Nullable Unseekables<?> didNotAchieveTarget)
+ {
+ this.achieved = Invariants.nonNull(achieved);
+ this.achievedTarget = achievedTarget;
+ this.didNotAchieveTarget = didNotAchieveTarget == null || didNotAchieveTarget.isEmpty() ? null : didNotAchieveTarget;
+ }
+ }
+
+ public static void fetch(Known fetch, Node node, TxnId txnId, Unseekables<?> someUnseekables, @Nullable EpochSupplier forLocalEpoch, @Nullable Timestamp executeAt, BiConsumer<? super FetchResult, Throwable> callback)
{
if (someUnseekables.kind().isRoute()) fetch(fetch, node, txnId, castToRoute(someUnseekables), forLocalEpoch, executeAt, callback);
else fetchViaSomeRoute(fetch, node, txnId, someUnseekables, forLocalEpoch, executeAt, callback);
}
- public static void fetch(Known fetch, Node node, TxnId txnId, Route<?> route, @Nullable EpochSupplier forLocalEpoch, @Nullable Timestamp executeAt, BiConsumer<? super Known, Throwable> callback)
+ public static void fetch(Known fetch, Node node, TxnId txnId, Route<?> route, @Nullable EpochSupplier forLocalEpoch, @Nullable Timestamp executeAt, BiConsumer<? super FetchResult, Throwable> callback)
{
long srcEpoch = fetch.fetchEpoch(txnId, executeAt);
if (!node.topology().hasEpoch(srcEpoch))
{
- node.withEpoch(srcEpoch, (ignore, withEpochFailure) -> {
- if (withEpochFailure != null)
- {
- callback.accept(null, CoordinationFailed.wrap(withEpochFailure));
- return;
- }
- fetch(fetch, node, txnId, route, forLocalEpoch, executeAt, callback);
- });
+ node.withEpoch(srcEpoch, callback, () -> fetch(fetch, node, txnId, route, forLocalEpoch, executeAt, callback));
return;
}
@@ -79,12 +87,30 @@ public class FetchData extends CheckShards<Route<?>>
}
else
{
- fetchInternal(ranges, fetch, node, txnId, route.slice(ranges), executeAt, toEpoch, callback);
+ Route<?> slicedRoute = route.slice(ranges);
+ fetchInternal(ranges, fetch, node, txnId, slicedRoute, slicedRoute, executeAt, toEpoch, callback);
}
}
+ /**
+ * Do not make an attempt to discern what keys need to be contacted; fetch from only the specific remote keys that were requested.
+ */
+ public static void fetchSpecific(Known fetch, Node node, TxnId txnId, Route<?> query, Unseekables<?> propagateTo, @Nullable EpochSupplier forLocalEpoch, @Nullable Timestamp executeAt, BiConsumer<? super FetchResult, Throwable> callback)
+ {
+ long srcEpoch = fetch.fetchEpoch(txnId, executeAt);
+ if (!node.topology().hasEpoch(srcEpoch))
+ {
+ node.withEpoch(srcEpoch, callback, () -> fetchSpecific(fetch, node, txnId, query, propagateTo, forLocalEpoch, executeAt, callback));
+ return;
+ }
+
+ Invariants.checkArgument(node.topology().hasEpoch(srcEpoch), "Unknown epoch %d, latest known is %d", srcEpoch, node.epoch());
+ long toEpoch = Math.max(srcEpoch, forLocalEpoch == null ? 0 : forLocalEpoch.epoch());
+ fetchData(fetch, node, txnId, query, propagateTo, srcEpoch, toEpoch, callback);
+ }
+
@SuppressWarnings({"unchecked", "rawtypes"})
- private static void fetchViaSomeRoute(Known fetch, Node node, TxnId txnId, Unseekables<?> someUnseekables, @Nullable EpochSupplier forLocalEpoch, @Nullable Timestamp executeAt, BiConsumer<? super Known, Throwable> callback)
+ private static void fetchViaSomeRoute(Known fetch, Node node, TxnId txnId, Unseekables<?> someUnseekables, @Nullable EpochSupplier forLocalEpoch, @Nullable Timestamp executeAt, BiConsumer<? super FetchResult, Throwable> callback)
{
FindSomeRoute.findSomeRoute(node, txnId, someUnseekables, (foundRoute, fail) -> {
if (fail != null) callback.accept(null, fail);
@@ -112,7 +138,7 @@ public class FetchData extends CheckShards<Route<?>>
});
}
- private static void reportRouteNotFound(Node node, TxnId txnId, @Nullable Timestamp executeAt, @Nullable EpochSupplier forLocalEpoch, Unseekables<?> someUnseekables, Known found, BiConsumer<? super Known, Throwable> callback)
+ private static void reportRouteNotFound(Node node, TxnId txnId, @Nullable Timestamp executeAt, @Nullable EpochSupplier forLocalEpoch, Unseekables<?> fetchKeys, Known found, BiConsumer<? super FetchResult, Throwable> callback)
{
Invariants.checkState(executeAt == null);
switch (found.outcome)
@@ -120,15 +146,27 @@ public class FetchData extends CheckShards<Route<?>>
default: throw new AssertionError("Unknown outcome: " + found.outcome);
case Invalidated:
if (forLocalEpoch == null) forLocalEpoch = txnId;
- locallyInvalidateAndCallback(node, txnId, forLocalEpoch, someUnseekables, found, callback);
+ locallyInvalidateAndCallback(node, txnId, forLocalEpoch, fetchKeys, new FetchResult(found, fetchKeys, null), callback);
break;
case Unknown:
if (found.canProposeInvalidation())
{
- Invalidate.invalidate(node, txnId, someUnseekables, (outcome, throwable) -> {
- Known known = throwable != null ? null : outcome.asProgressToken().status == Status.Invalidated ? Known.Invalidated : Known.Nothing;
- callback.accept(known, throwable);
+ Invalidate.invalidate(node, txnId, fetchKeys, (outcome, throwable) -> {
+ FetchResult result = null;
+ if (throwable == null)
+ {
+ Known achieved = Known.Invalidated;
+ Unseekables<?> achievedTarget = fetchKeys, didNotAchieveTarget = null;
+ if (outcome.asProgressToken().status != Status.Invalidated)
+ {
+ achievedTarget = fetchKeys.slice(0, 0);
+ didNotAchieveTarget = fetchKeys;
+ achieved = Known.Nothing;
+ }
+ result = new FetchResult(achieved, achievedTarget, didNotAchieveTarget);
+ }
+ callback.accept(result, throwable);
});
break;
}
@@ -136,11 +174,11 @@ public class FetchData extends CheckShards<Route<?>>
case WasApply:
case Apply:
// TODO (expected): we may be stale
- callback.accept(found, null);
+ callback.accept(new FetchResult(found, fetchKeys.slice(0, 0), fetchKeys), null);
}
}
- private static void fetchWithIncompleteRoute(Known fetch, Node node, TxnId txnId, Route<?> someRoute, @Nullable EpochSupplier forLocalEpoch, @Nullable Timestamp executeAt, BiConsumer<? super Known, Throwable> callback)
+ private static void fetchWithIncompleteRoute(Known fetch, Node node, TxnId txnId, Route<?> someRoute, @Nullable EpochSupplier forLocalEpoch, @Nullable Timestamp executeAt, BiConsumer<? super FetchResult, Throwable> callback)
{
long srcEpoch = fetch.fetchEpoch(txnId, executeAt);
Invariants.checkArgument(node.topology().hasEpoch(srcEpoch), "Unknown epoch %d, latest known is %d", srcEpoch, node.epoch());
@@ -151,7 +189,7 @@ public class FetchData extends CheckShards<Route<?>>
});
}
- public static void fetch(Known fetch, Node node, TxnId txnId, FullRoute<?> route, @Nullable EpochSupplier forLocalEpoch, @Nullable Timestamp executeAt, BiConsumer<? super Known, Throwable> callback)
+ public static void fetch(Known fetch, Node node, TxnId txnId, FullRoute<?> route, @Nullable EpochSupplier forLocalEpoch, @Nullable Timestamp executeAt, BiConsumer<? super FetchResult, Throwable> callback)
{
node.withEpoch(executeAt, (ignore, withEpochFailure) -> {
if (withEpochFailure != null)
@@ -161,22 +199,24 @@ public class FetchData extends CheckShards<Route<?>>
}
long toEpoch = Math.max(fetch.fetchEpoch(txnId, executeAt), forLocalEpoch == null ? 0 : forLocalEpoch.epoch());
Ranges ranges = node.topology().localRangesForEpochs(txnId.epoch(), toEpoch);
- fetchInternal(ranges, fetch, node, txnId, route.slice(ranges), executeAt, toEpoch, callback);
+
+ Route<?> slicedRoute = route.slice(ranges);
+ fetchInternal(ranges, fetch, node, txnId, slicedRoute, slicedRoute, executeAt, toEpoch, callback);
});
}
- private static void fetchInternal(Ranges ranges, Known target, Node node, TxnId txnId, PartialRoute<?> route, @Nullable Timestamp executeAt, long forLocalEpoch, BiConsumer<? super Known, Throwable> callback)
+ private static Object fetchInternal(Ranges ranges, Known target, Node node, TxnId txnId, Route<?> route, Unseekables<?> propagateTo, @Nullable Timestamp executeAt, long forLocalEpoch, BiConsumer<? super FetchResult, Throwable> callback)
{
long srcEpoch = target.fetchEpoch(txnId, executeAt);
Invariants.checkArgument(node.topology().hasEpoch(srcEpoch), "Unknown epoch %d, latest known is %d", srcEpoch, node.epoch());
- PartialRoute<?> fetch = route.slice(ranges);
- fetchData(target, node, txnId, fetch, srcEpoch, forLocalEpoch, (sufficientFor, fail) -> {
+ Route<?> fetch = route.slice(ranges);
+ return fetchData(target, node, txnId, fetch, propagateTo, srcEpoch, forLocalEpoch, (result, fail) -> {
if (fail != null) callback.accept(null, fail);
- else callback.accept(sufficientFor, null);
+ else callback.accept(result, null);
});
}
- final BiConsumer<Known, Throwable> callback;
+ final BiConsumer<? super FetchResult, Throwable> callback;
/**
* The epoch until which we want to persist any response for locally
*/
@@ -186,29 +226,33 @@ public class FetchData extends CheckShards<Route<?>>
// (i.e. if preaccept/accept contact a later epoch than execution is decided for)
final long toEpoch;
- private FetchData(Node node, Known target, TxnId txnId, Route<?> route, long sourceEpoch, long toEpoch, BiConsumer<Known, Throwable> callback)
+ final Unseekables<?> propagateTo;
+
+ private FetchData(Node node, Known target, TxnId txnId, Route<?> route, Unseekables<?> propagateTo, long sourceEpoch, long toEpoch, BiConsumer<? super FetchResult, Throwable> callback)
{
- this(node, target, txnId, route, route.withHomeKey(), sourceEpoch, toEpoch, callback);
+ this(node, target, txnId, route, route.withHomeKey(), propagateTo, sourceEpoch, toEpoch, callback);
}
- private FetchData(Node node, Known target, TxnId txnId, Route<?> route, Route<?> routeWithHomeKey, long sourceEpoch, long toEpoch, BiConsumer<Known, Throwable> callback)
+ private FetchData(Node node, Known target, TxnId txnId, Route<?> route, Route<?> routeWithHomeKey, Unseekables<?> propagateTo, long sourceEpoch, long toEpoch, BiConsumer<? super FetchResult, Throwable> callback)
{
// TODO (desired, efficiency): restore behaviour of only collecting info if e.g. Committed or Executed
super(node, txnId, routeWithHomeKey, sourceEpoch, CheckStatus.IncludeInfo.All);
+ this.propagateTo = propagateTo;
Invariants.checkArgument(routeWithHomeKey.contains(route.homeKey()), "route %s does not contain %s", routeWithHomeKey, route.homeKey());
this.target = target;
this.toEpoch = toEpoch;
this.callback = callback;
}
- private static FetchData fetchData(Known sufficientStatus, Node node, TxnId txnId, Route<?> route, long sourceEpoch, long toEpoch, BiConsumer<Known, Throwable> callback)
+ private static FetchData fetchData(Known sufficientStatus, Node node, TxnId txnId, Route<?> route, Unseekables<?> propagateTo, long sourceEpoch, long toEpoch, BiConsumer<? super FetchResult, Throwable> callback)
{
- FetchData fetch = new FetchData(node, sufficientStatus, txnId, route, sourceEpoch, toEpoch, callback);
+ Invariants.checkState(!propagateTo.isEmpty());
+ FetchData fetch = new FetchData(node, sufficientStatus, txnId, route, propagateTo, sourceEpoch, toEpoch, callback);
fetch.start();
return fetch;
}
- protected Route<?> route()
+ protected Route<?> query()
{
return route;
}
@@ -217,7 +261,7 @@ public class FetchData extends CheckShards<Route<?>>
protected boolean isSufficient(Node.Id from, CheckStatus.CheckStatusOk ok)
{
Ranges rangesForNode = topologies().computeRangesForNode(from);
- PartialRoute<?> scope = this.route.slice(rangesForNode);
+ Route<?> scope = this.route.slice(rangesForNode);
return isSufficient(scope, ok);
}
@@ -249,7 +293,7 @@ public class FetchData extends CheckShards<Route<?>>
}
// TODO (expected): should we automatically trigger a new fetch if we find executeAt but did not request enough information? would be more rob ust
- Propagate.propagate(node, txnId, sourceEpoch, toEpoch, success.withQuorum, route(), target, (CheckStatusOkFull) merged, callback);
+ Propagate.propagate(node, txnId, sourceEpoch, toEpoch, success.withQuorum, query(), propagateTo, target, (CheckStatusOkFull) merged, callback);
}
}
}
diff --git a/accord-core/src/main/java/accord/coordinate/InformHomeOfTxn.java b/accord-core/src/main/java/accord/coordinate/InformHomeOfTxn.java
deleted file mode 100644
index 68767958..00000000
--- a/accord-core/src/main/java/accord/coordinate/InformHomeOfTxn.java
+++ /dev/null
@@ -1,92 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package accord.coordinate;
-
-import accord.coordinate.tracking.QuorumTracker.QuorumShardTracker;
-import accord.local.Node;
-import accord.local.Node.Id;
-import accord.messages.Callback;
-import accord.messages.InformOfTxnId;
-import accord.messages.SimpleReply;
-import accord.primitives.Route;
-import accord.topology.Shard;
-import accord.primitives.TxnId;
-import accord.utils.async.AsyncChain;
-import accord.utils.async.AsyncResults;
-
-import static accord.coordinate.tracking.AbstractTracker.ShardOutcomes.Fail;
-import static accord.coordinate.tracking.AbstractTracker.ShardOutcomes.Success;
-
-public class InformHomeOfTxn extends AsyncResults.SettableResult<Void> implements Callback<SimpleReply>
-{
- final TxnId txnId;
- final Route<?> someRoute;
- final QuorumShardTracker tracker;
- Throwable failure;
-
- InformHomeOfTxn(TxnId txnId, Route<?> someRoute, Shard homeShard)
- {
- this.txnId = txnId;
- this.someRoute = someRoute;
- this.tracker = new QuorumShardTracker(homeShard);
- }
-
- public static AsyncChain<Void> inform(Node node, TxnId txnId, Route<?> someRoute)
- {
- return node.withEpoch(txnId.epoch(), () -> {
- Shard homeShard = node.topology().forEpoch(someRoute.homeKey(), txnId.epoch());
- InformHomeOfTxn inform = new InformHomeOfTxn(txnId, someRoute, homeShard);
- node.send(homeShard.nodes, new InformOfTxnId(txnId, someRoute), inform);
- return inform;
- });
- }
-
- @Override
- public void onSuccess(Id from, SimpleReply reply)
- {
- switch (reply)
- {
- default:
- case Ok:
- if (tracker.onSuccess(null) == Success)
- trySuccess(null);
- break;
-
- case Nack:
- // TODO (required, consider): stale topology should be impossible right now
- onFailure(from, new StaleTopology());
- }
- }
-
- @Override
- public void onFailure(Id from, Throwable failure)
- {
- if (this.failure == null) this.failure = failure;
- else this.failure.addSuppressed(failure);
-
- if (tracker.onFailure(null) == Fail)
- tryFailure(this.failure);
- }
-
- @Override
- public void onCallbackFailure(Id from, Throwable failure)
- {
- tryFailure(failure);
- }
-}
diff --git a/accord-core/src/main/java/accord/coordinate/Invalidate.java b/accord-core/src/main/java/accord/coordinate/Invalidate.java
index b4cefb4b..f1af5287 100644
--- a/accord-core/src/main/java/accord/coordinate/Invalidate.java
+++ b/accord-core/src/main/java/accord/coordinate/Invalidate.java
@@ -44,7 +44,7 @@ import static accord.local.PreLoadContext.contextFor;
import static accord.local.Status.Accepted;
import static accord.local.Status.PreAccepted;
import static accord.primitives.ProgressToken.INVALIDATED;
-import static accord.primitives.ProgressToken.TRUNCATED;
+import static accord.primitives.ProgressToken.TRUNCATED_DURABLE_OR_INVALIDATED;
import static accord.utils.Invariants.illegalState;
public class Invalidate implements Callback<InvalidateReply>
@@ -154,13 +154,13 @@ public class Invalidate implements Callback<InvalidateReply>
// first look to see if it has already been decided/invalidated
// check each shard independently - if we find any that can be invalidated, do so
InvalidateReply max = InvalidateReply.max(replies);
- InvalidateReply maxNotTruncated = max.status != Status.Truncated ? max : InvalidateReply.maxNotTruncated(replies);
+ InvalidateReply maxNotTruncated = max.maxKnowledgeStatus.is(Status.Truncated) ? max : InvalidateReply.maxNotTruncated(replies);
if (maxNotTruncated != null)
{
- switch (maxNotTruncated.status)
+ switch (maxNotTruncated.maxKnowledgeStatus.status)
{
- default: throw new AssertionError("Unhandled status: " + maxNotTruncated.status);
+ default: throw new AssertionError("Unhandled status: " + maxNotTruncated.maxKnowledgeStatus.status);
case Truncated: throw illegalState();
case AcceptedInvalidate:
@@ -177,7 +177,7 @@ public class Invalidate implements Callback<InvalidateReply>
case Stable:
case Committed:
case PreCommitted:
- Invariants.checkState(maxNotTruncated.status == PreAccepted || !invalidateWith.contains(someRoute.homeKey()) || fullRoute != null);
+ Invariants.checkState(maxNotTruncated.maxKnowledgeStatus.status == PreAccepted || !invalidateWith.contains(someRoute.homeKey()) || fullRoute != null);
case Accepted:
// TODO (desired, efficiency): if we see Committed or above, go straight to Execute if we have assembled enough information
@@ -194,7 +194,7 @@ public class Invalidate implements Callback<InvalidateReply>
// Note that there's lots of scope for variations in behaviour here, but lots of care is needed.
- Status witnessedByInvalidation = maxNotTruncated.status;
+ Status witnessedByInvalidation = maxNotTruncated.maxKnowledgeStatus.status;
if (!witnessedByInvalidation.hasBeen(Accepted))
{
Invariants.checkState(tracker.all(InvalidationShardTracker::isPromised));
@@ -218,12 +218,12 @@ public class Invalidate implements Callback<InvalidateReply>
for (Shard shard : topology.shards())
{
InvalidateReply maxReply = InvalidateReply.max(replies, shard, topology.nodes());
- allShardsTruncated &= maxReply.status == Status.Truncated;
+ allShardsTruncated &= maxReply.maxStatus.is(Status.Truncated);
}
if (allShardsTruncated)
{
isDone = true;
- callback.accept(TRUNCATED, null);
+ callback.accept(TRUNCATED_DURABLE_OR_INVALIDATED, null);
return;
}
}
diff --git a/accord-core/src/main/java/accord/coordinate/MaybeRecover.java b/accord-core/src/main/java/accord/coordinate/MaybeRecover.java
index 10aa0994..fea1aec3 100644
--- a/accord-core/src/main/java/accord/coordinate/MaybeRecover.java
+++ b/accord-core/src/main/java/accord/coordinate/MaybeRecover.java
@@ -63,7 +63,8 @@ public class MaybeRecover extends CheckShards<Route<?>>
public boolean hasMadeProgress(CheckStatusOk ok)
{
- return ok != null && (ok.isCoordinating // TODO (required, liveness): make this coordinatingSince so can be pre-empted by others if stuck
+ // TODO (required, liveness): if Ballot.hlc is stale enough then preempt; also do not query isCoordinating, query directly the node that owns the ballot (or TxnId if Ballot is ZERO)
+ return ok != null && (ok.isCoordinating
|| ok.toProgressToken().compareTo(prevProgress) > 0);
}
diff --git a/accord-core/src/main/java/accord/coordinate/Propose.java b/accord-core/src/main/java/accord/coordinate/Propose.java
index e0cbaf03..74cf7c38 100644
--- a/accord-core/src/main/java/accord/coordinate/Propose.java
+++ b/accord-core/src/main/java/accord/coordinate/Propose.java
@@ -78,8 +78,8 @@ abstract class Propose<R> implements Callback<AcceptReply>
this.callback = callback;
this.acceptOks = new ArrayList<>();
this.acceptTracker = new QuorumTracker(topologies);
- Invariants.checkState(txnId.kind().isSyncPoint() || deps.keyDeps.txnIdCount() == 0 || deps.keyDeps.txnId(deps.keyDeps.txnIdCount() - 1).compareTo(executeAt) < 0,
- "Attempted to propose a standard transaction with an earlier executeAt than a conflicting transaction it witnessed: %s vs executeAt: %s", deps, executeAt);
+ Invariants.checkState(txnId.kind().isSyncPoint() || deps.maxTxnId(txnId).compareTo(executeAt) <= 0,
+ "Attempted to propose %s with an earlier executeAt than a conflicting transaction it witnessed: %s vs executeAt: %s", txnId, deps, executeAt);
}
void start()
diff --git a/accord-core/src/main/java/accord/coordinate/RangeUnavailable.java b/accord-core/src/main/java/accord/coordinate/RangeUnavailable.java
deleted file mode 100644
index 55089b0f..00000000
--- a/accord-core/src/main/java/accord/coordinate/RangeUnavailable.java
+++ /dev/null
@@ -1,56 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package accord.coordinate;
-
-import javax.annotation.Nullable;
-
-import accord.api.RoutingKey;
-import accord.primitives.Ranges;
-import accord.primitives.TxnId;
-
-import static accord.utils.Invariants.checkState;
-
-public class RangeUnavailable extends Exhausted
-{
- public final Ranges unavailable;
-
- public RangeUnavailable(Ranges unavailable, TxnId txnId)
- {
- super(txnId, null, buildMessage(unavailable));
- this.unavailable = unavailable;
- }
-
- private RangeUnavailable(Ranges unavailable, TxnId txnId, @Nullable RoutingKey homeKey, RangeUnavailable cause)
- {
- super(txnId, homeKey, cause);
- this.unavailable = unavailable;
- }
-
- @Override
- public RangeUnavailable wrap()
- {
- checkState(this.getClass() == RangeUnavailable.class);
- return new RangeUnavailable(unavailable, txnId(), homeKey(), this);
- }
-
- private static String buildMessage(Ranges unavailable)
- {
- return "The following ranges are unavailable to read: " + unavailable;
- }
-}
diff --git a/accord-core/src/main/java/accord/coordinate/ReadCoordinator.java b/accord-core/src/main/java/accord/coordinate/ReadCoordinator.java
index 1d944be7..1e2e99da 100644
--- a/accord-core/src/main/java/accord/coordinate/ReadCoordinator.java
+++ b/accord-core/src/main/java/accord/coordinate/ReadCoordinator.java
@@ -40,6 +40,7 @@ import static accord.messages.CheckStatus.WithQuorum.HasQuorum;
import static accord.messages.CheckStatus.WithQuorum.NoQuorum;
import static accord.utils.Invariants.debug;
+// TODO (expected): configure the number of initial requests we send
public abstract class ReadCoordinator<Reply extends accord.messages.Reply> extends ReadTracker implements Callback<Reply>
{
public enum Action
@@ -110,6 +111,8 @@ public abstract class ReadCoordinator<Reply extends accord.messages.Reply> exten
protected abstract Action process(Id from, Reply reply);
protected abstract void onDone(Success success, Throwable failure);
protected abstract void contact(Id to);
+
+ // TODO (desired): this isn't very clean way of integrating these responses
protected Ranges unavailable(Reply reply) { throw new UnsupportedOperationException(); }
@Override
@@ -216,15 +219,20 @@ public abstract class ReadCoordinator<Reply extends accord.messages.Reply> exten
if (failure == null)
{
Ranges unavailable = Ranges.EMPTY;
- Ranges exhausted = Ranges.EMPTY;
for (ReadShardTracker tracker : trackers)
{
- if (tracker.hasSucceeded())
+ if (tracker == null || tracker.hasSucceeded())
continue;
- if (tracker.unavailable() != null) unavailable = unavailable.with(tracker.unavailable());
- else exhausted = exhausted.with(Ranges.of(tracker.shard.range));
+
+ if (tracker.unavailable() != null)
+ {
+ if (tracker.unavailable() != null)
+ unavailable = unavailable.with(tracker.unavailable());
+ }
+ else unavailable = unavailable.with(Ranges.of(tracker.shard.range));
}
- failure = new Exhausted(txnId, null, (unavailable.isEmpty() ? "" : "unavailable: " + unavailable + (exhausted.isEmpty() ? "" : "; ")) + (exhausted.isEmpty() ? "" : "no response: " + exhausted));
+
+ failure = new Exhausted(txnId, null, unavailable);
}
finishOnFailure();
}
diff --git a/accord-core/src/main/java/accord/coordinate/Recover.java b/accord-core/src/main/java/accord/coordinate/Recover.java
index b4fde287..132baf4b 100644
--- a/accord-core/src/main/java/accord/coordinate/Recover.java
+++ b/accord-core/src/main/java/accord/coordinate/Recover.java
@@ -27,9 +27,9 @@ import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.BiConsumer;
import java.util.stream.Stream;
+import accord.api.ProgressLog.BlockedUntil;
import accord.api.Result;
import accord.coordinate.CoordinationAdapter.Invoke;
-import accord.coordinate.tracking.QuorumTracker;
import accord.coordinate.tracking.RecoveryTracker;
import accord.local.Node;
import accord.local.Node.Id;
@@ -39,8 +39,6 @@ import accord.messages.BeginRecovery.RecoverOk;
import accord.messages.BeginRecovery.RecoverReply;
import accord.messages.Callback;
import accord.messages.Commit;
-import accord.messages.WaitOnCommit;
-import accord.messages.WaitOnCommit.WaitOnCommitOk;
import accord.primitives.Ballot;
import accord.primitives.Deps;
import accord.primitives.FullRoute;
@@ -69,54 +67,16 @@ import static accord.coordinate.tracking.RequestStatus.Failed;
import static accord.coordinate.tracking.RequestStatus.Success;
import static accord.messages.BeginRecovery.RecoverOk.maxAccepted;
import static accord.messages.BeginRecovery.RecoverOk.maxAcceptedNotTruncated;
-import static accord.primitives.ProgressToken.TRUNCATED;
+import static accord.primitives.ProgressToken.TRUNCATED_DURABLE_OR_INVALIDATED;
import static accord.utils.Invariants.debug;
import static accord.utils.Invariants.illegalState;
// TODO (low priority, cleanup): rename to Recover (verb); rename Recover message to not clash
// TODO (expected): do not recover transactions that are known to be Stable and waiting to execute.
// TODO (expected): separate out recovery of sync points from standard transactions
+// TODO (required): do not wait for a quorum if we find enough information to execute
public class Recover implements Callback<RecoverReply>, BiConsumer<Result, Throwable>
{
- class AwaitCommit extends AsyncResults.SettableResult<Timestamp> implements Callback<WaitOnCommitOk>
- {
- // TODO (desired, efficiency): this should collect the executeAt of any commit, and terminate as soon as one is found
- // that is earlier than TxnId for the Txn we are recovering; if all commits we wait for
- // are given earlier timestamps we can retry without restarting.
- final QuorumTracker tracker;
-
- AwaitCommit(Node node, TxnId txnId, Participants<?> participants)
- {
- Topology topology = node.topology().globalForEpoch(txnId.epoch()).forSelection(participants);
- this.tracker = new QuorumTracker(new Topologies.Single(node.topology().sorter(), topology));
- node.send(topology.nodes(), to -> new WaitOnCommit(to, topology, txnId, participants), this);
- }
-
- @Override
- public synchronized void onSuccess(Id from, WaitOnCommitOk reply)
- {
- if (isDone()) return;
-
- if (tracker.recordSuccess(from) == Success)
- trySuccess(null);
- }
-
- @Override
- public synchronized void onFailure(Id from, Throwable failure)
- {
- if (isDone()) return;
-
- if (tracker.recordFailure(from) == Failed)
- tryFailure(new Timeout(txnId, route.homeKey()));
- }
-
- @Override
- public void onCallbackFailure(Id from, Throwable failure)
- {
- tryFailure(failure);
- }
- }
-
AsyncResult<Object> awaitCommits(Node node, Deps waitOn)
{
AtomicInteger remaining = new AtomicInteger(waitOn.txnIdCount());
@@ -124,10 +84,12 @@ public class Recover implements Callback<RecoverReply>, BiConsumer<Result, Throw
for (int i = 0 ; i < waitOn.txnIdCount() ; ++i)
{
TxnId txnId = waitOn.txnId(i);
- new AwaitCommit(node, txnId, waitOn.participants(txnId)).addCallback((success, failure) -> {
+ Participants<?> participants = waitOn.participants(txnId);
+ Topology topology = node.topology().globalForEpoch(txnId.epoch()).forSelection(participants);
+ SynchronousAwait.awaitQuorum(node, topology, txnId, route.homeKey(), BlockedUntil.HasCommittedDeps, participants).addCallback((success, failure) -> {
if (result.isDone())
return;
- if (success != null && remaining.decrementAndGet() == 0)
+ if (failure == null && remaining.decrementAndGet() == 0)
result.setSuccess(success);
else
result.tryFailure(failure);
@@ -333,7 +295,7 @@ public class Recover implements Callback<RecoverReply>, BiConsumer<Result, Throw
}
}
- if (acceptOrCommit != acceptOrCommitNotTruncated)
+ if (acceptOrCommit != null)
{
boolean allShardsTruncated = true;
for (Shard shard : topology.shards())
@@ -343,8 +305,15 @@ public class Recover implements Callback<RecoverReply>, BiConsumer<Result, Throw
}
if (allShardsTruncated)
{
+ // TODO (required, correctness): this is not a safe inference in the case of an ErasedOrInvalidOrVestigial response.
+ // We need to tighten up the inference and spread of truncation/invalid outcomes.
+ // In this case, at minimum this can lead to liveness violations as the home shard stops coordinating
+ // a command that it hasn't invalidated, but nor is it possible to recover. This happens because
+ // when the home shard shares all of its replicas with another shard that has autonomously invalidated
+ // the transaction, so that all received InvalidateReply show truncation (when in fact this is only partial).
+ // We could paper over this, but better to revisit and provide stronger invariants we can rely on.
isDone = true;
- callback.accept(TRUNCATED, null);
+ callback.accept(TRUNCATED_DURABLE_OR_INVALIDATED, null);
return;
}
}
@@ -388,14 +357,14 @@ public class Recover implements Callback<RecoverReply>, BiConsumer<Result, Throw
withDeps.accept(null, CoordinationFailed.wrap(withEpochFailure));
return;
}
- Seekables<?, ?> missing = txn.keys().subtract(merged.sufficientFor);
+ Seekables<?, ?> missing = txn.keys().without(merged.sufficientFor);
if (missing.isEmpty())
{
withDeps.accept(merged.deps, null);
}
else
{
- CollectDeps.withDeps(node, txnId, route, missing.toParticipants(), missing, executeAt, (extraDeps, fail) -> {
+ CollectCalculatedDeps.withCalculatedDeps(node, txnId, route, missing.toParticipants(), missing, executeAt, (extraDeps, fail) -> {
if (fail != null)
{
isDone = true;
diff --git a/accord-core/src/main/java/accord/coordinate/RecoverWithRoute.java b/accord-core/src/main/java/accord/coordinate/RecoverWithRoute.java
index e33615e4..8fcafe4b 100644
--- a/accord-core/src/main/java/accord/coordinate/RecoverWithRoute.java
+++ b/accord-core/src/main/java/accord/coordinate/RecoverWithRoute.java
@@ -33,7 +33,6 @@ import accord.messages.Propagate;
import accord.primitives.Ballot;
import accord.primitives.Deps;
import accord.primitives.FullRoute;
-import accord.primitives.PartialRoute;
import accord.primitives.Participants;
import accord.primitives.Ranges;
import accord.primitives.Route;
@@ -50,7 +49,7 @@ import static accord.local.Status.KnownExecuteAt.ExecuteAtKnown;
import static accord.local.Status.Outcome.Apply;
import static accord.primitives.ProgressToken.APPLIED;
import static accord.primitives.ProgressToken.INVALIDATED;
-import static accord.primitives.ProgressToken.TRUNCATED;
+import static accord.primitives.ProgressToken.TRUNCATED_DURABLE_OR_INVALIDATED;
import static accord.primitives.Route.castToFullRoute;
import static accord.utils.Invariants.illegalState;
@@ -110,7 +109,7 @@ public class RecoverWithRoute extends CheckShards<FullRoute<?>>
protected boolean isSufficient(Id from, CheckStatusOk ok)
{
Ranges rangesForNode = topologies().forEpoch(txnId.epoch()).rangesForNode(from);
- PartialRoute<?> route = this.route.slice(rangesForNode);
+ Route<?> route = this.route.slice(rangesForNode);
return isSufficient(route, ok);
}
@@ -178,7 +177,7 @@ public class RecoverWithRoute extends CheckShards<FullRoute<?>>
{
// we might have only part of the full transaction, and a shard may have truncated;
// in this case we want to skip straight to apply, but only for the shards that haven't truncated
- Participants<?> sendTo = route.participants().subtract(full.truncatedResponse());
+ Participants<?> sendTo = route.participants().without(full.truncatedResponse());
if (!sendTo.isEmpty())
{
known = full.knownFor(sendTo);
@@ -203,7 +202,7 @@ public class RecoverWithRoute extends CheckShards<FullRoute<?>>
propagate = full;
}
- Propagate.propagate(node, txnId, sourceEpoch, success.withQuorum, route, null, propagate, (s, f) -> callback.accept(f == null ? propagate.toProgressToken() : null, f));
+ Propagate.propagate(node, txnId, sourceEpoch, success.withQuorum, route, route, null, propagate, (s, f) -> callback.accept(f == null ? propagate.toProgressToken() : null, f));
break;
}
@@ -231,11 +230,11 @@ public class RecoverWithRoute extends CheckShards<FullRoute<?>>
if (witnessedByInvalidation != null && witnessedByInvalidation.hasBeen(Status.PreCommitted))
throw illegalState("We previously invalidated, finding a status that should be recoverable");
- Propagate.propagate(node, txnId, sourceEpoch, success.withQuorum, route, null, full, (s, f) -> callback.accept(f == null ? INVALIDATED : null, f));
+ Propagate.propagate(node, txnId, sourceEpoch, success.withQuorum, route, route, null, full, (s, f) -> callback.accept(f == null ? INVALIDATED : null, f));
break;
case Erased:
- Propagate.propagate(node, txnId, sourceEpoch, success.withQuorum, route, null, full, (s, f) -> callback.accept(f == null ? TRUNCATED : null, f));
+ Propagate.propagate(node, txnId, sourceEpoch, success.withQuorum, route, route, null, full, (s, f) -> callback.accept(f == null ? TRUNCATED_DURABLE_OR_INVALIDATED : null, f));
break;
}
}
diff --git a/accord-core/src/main/java/accord/coordinate/RecoverWithSomeRoute.java b/accord-core/src/main/java/accord/coordinate/RecoverWithSomeRoute.java
index c2a2ccba..2e98333b 100644
--- a/accord-core/src/main/java/accord/coordinate/RecoverWithSomeRoute.java
+++ b/accord-core/src/main/java/accord/coordinate/RecoverWithSomeRoute.java
@@ -92,7 +92,7 @@ public class RecoverWithSomeRoute extends CheckShards<Route<?>> implements BiCon
case Quorum:
if (merged != null && merged.isTruncatedResponse())
{
- callback.accept(ProgressToken.TRUNCATED, null);
+ callback.accept(ProgressToken.TRUNCATED_DURABLE_OR_INVALIDATED, null);
}
else
{
diff --git a/accord-core/src/main/java/accord/coordinate/SynchronousAwait.java b/accord-core/src/main/java/accord/coordinate/SynchronousAwait.java
new file mode 100644
index 00000000..6f76ff8b
--- /dev/null
+++ b/accord-core/src/main/java/accord/coordinate/SynchronousAwait.java
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.coordinate;
+
+import javax.annotation.Nullable;
+
+import accord.api.ProgressLog.BlockedUntil;
+import accord.api.RoutingKey;
+import accord.coordinate.tracking.QuorumTracker;
+import accord.coordinate.tracking.ResponseTracker;
+import accord.local.Node;
+import accord.local.Node.Id;
+import accord.messages.Await;
+import accord.messages.Await.AwaitOk;
+import accord.messages.Callback;
+import accord.primitives.Participants;
+import accord.primitives.TxnId;
+import accord.topology.Topologies;
+import accord.topology.Topology;
+import accord.utils.async.AsyncResults;
+
+import static accord.coordinate.tracking.RequestStatus.Failed;
+import static accord.coordinate.tracking.RequestStatus.Success;
+
+/**
+ * Synchronously await some set of replicas reaching a given wait condition.
+ * This may or may not be a condition we expect to reach promptly, but we will wait only until the timeout passes
+ * at which point we will report failure.
+ */
+public class SynchronousAwait extends AsyncResults.SettableResult<Void> implements Callback<AwaitOk>
+{
+ // TODO (desired, efficiency): this should collect the executeAt of any commit, and terminate as soon as one is found
+ // that is earlier than TxnId for the Txn we are recovering; if all commits we wait for
+ // are given earlier timestamps we can retry without restarting.
+
+ final TxnId txnId;
+ final @Nullable RoutingKey homeKey;
+ final ResponseTracker tracker;
+
+ public SynchronousAwait(TxnId txnId, @Nullable RoutingKey homeKey, ResponseTracker tracker)
+ {
+ this.txnId = txnId;
+ this.homeKey = homeKey;
+ this.tracker = tracker;
+ }
+
+ public static SynchronousAwait awaitQuorum(Node node, Topology topology, TxnId txnId, @Nullable RoutingKey homeKey, BlockedUntil awaiting, Participants<?> participants)
+ {
+ QuorumTracker tracker = new QuorumTracker(new Topologies.Single(node.topology().sorter(), topology));
+ SynchronousAwait result = new SynchronousAwait(txnId, homeKey, tracker);
+ result.start(node, topology, participants, awaiting);
+ return result;
+ }
+
+ private void start(Node node, Topology topology, Participants<?> participants, BlockedUntil blockedUntil)
+ {
+ node.send(topology.nodes(), to -> new Await(to, topology, txnId, participants, blockedUntil), this);
+ }
+
+ @Override
+ public synchronized void onSuccess(Id from, AwaitOk reply)
+ {
+ if (isDone()) return;
+
+ if (tracker.recordSuccess(from) == Success)
+ trySuccess(null);
+ }
+
+ @Override
+ public synchronized void onFailure(Id from, Throwable failure)
+ {
+ if (isDone()) return;
+
+ if (tracker.recordFailure(from) == Failed)
+ tryFailure(new Timeout(txnId, homeKey));
+ }
+
+ @Override
+ public void onCallbackFailure(Id from, Throwable failure)
+ {
+ tryFailure(failure);
+ }
+}
+
diff --git a/accord-core/src/main/java/accord/coordinate/TopologyMismatch.java b/accord-core/src/main/java/accord/coordinate/TopologyMismatch.java
index 33b48f07..1e4748ba 100644
--- a/accord-core/src/main/java/accord/coordinate/TopologyMismatch.java
+++ b/accord-core/src/main/java/accord/coordinate/TopologyMismatch.java
@@ -76,7 +76,7 @@ public class TopologyMismatch extends CoordinationFailed
private static String buildMessage(Topology t, Unseekables<?> select)
{
- return String.format("Attempted to access %s that are no longer valid globally (%d -> %s)", select.subtract(t.ranges()), t.epoch(), t.ranges());
+ return String.format("Attempted to access %s that are no longer valid globally (%d -> %s)", select.without(t.ranges()), t.epoch(), t.ranges());
}
@Nullable
diff --git a/accord-core/src/main/java/accord/coordinate/tracking/AppliedTracker.java b/accord-core/src/main/java/accord/coordinate/tracking/AllTracker.java
similarity index 73%
rename from accord-core/src/main/java/accord/coordinate/tracking/AppliedTracker.java
rename to accord-core/src/main/java/accord/coordinate/tracking/AllTracker.java
index 155e4f8f..86d7aa5e 100644
--- a/accord-core/src/main/java/accord/coordinate/tracking/AppliedTracker.java
+++ b/accord-core/src/main/java/accord/coordinate/tracking/AllTracker.java
@@ -26,13 +26,13 @@ import static accord.coordinate.tracking.AbstractTracker.ShardOutcomes.Fail;
import static accord.coordinate.tracking.AbstractTracker.ShardOutcomes.NoChange;
import static accord.coordinate.tracking.AbstractTracker.ShardOutcomes.Success;
-public class AppliedTracker extends AbstractSimpleTracker<AppliedTracker.AppliedShardTracker> implements ResponseTracker
+public class AllTracker extends AbstractSimpleTracker<AllTracker.AllShardTracker> implements ResponseTracker
{
- public static class AppliedShardTracker extends ShardTracker
+ public static class AllShardTracker extends ShardTracker
{
protected int waitingOn;
- public AppliedShardTracker(Shard shard)
+ public AllShardTracker(Shard shard)
{
super(shard);
waitingOn = shard.rf();
@@ -52,7 +52,7 @@ public class AppliedTracker extends AbstractSimpleTracker<AppliedTracker.Applied
return Fail;
}
- public boolean hasReachedQuorum()
+ public boolean hasSucceeded()
{
return waitingOn == 0;
}
@@ -68,34 +68,34 @@ public class AppliedTracker extends AbstractSimpleTracker<AppliedTracker.Applied
}
}
- public AppliedTracker(Topologies topologies)
+ public AllTracker(Topologies topologies)
{
- super(topologies, AppliedShardTracker[]::new, shard -> new AppliedShardTracker(shard));
+ super(topologies, AllShardTracker[]::new, shard -> new AllShardTracker(shard));
}
public RequestStatus recordSuccess(Node.Id node)
{
- return recordResponse(this, node, AppliedShardTracker::onSuccess, null);
+ return recordResponse(this, node, AllShardTracker::onSuccess, null);
}
// return true iff hasFailed()
public RequestStatus recordFailure(Node.Id node)
{
- return recordResponse(this, node, AppliedShardTracker::onFailure, null);
+ return recordResponse(this, node, AllShardTracker::onFailure, null);
}
public boolean hasFailed()
{
- return any(AppliedShardTracker::hasFailed);
+ return any(AllShardTracker::hasFailed);
}
public boolean hasInFlight()
{
- return any(AppliedShardTracker::hasInFlight);
+ return any(AllShardTracker::hasInFlight);
}
- public boolean hasReachedQuorum()
+ public boolean hasSucceeded()
{
- return all(AppliedShardTracker::hasReachedQuorum);
+ return all(AllShardTracker::hasSucceeded);
}
}
diff --git a/accord-core/src/main/java/accord/coordinate/tracking/AwaitTracker.java b/accord-core/src/main/java/accord/coordinate/tracking/AwaitTracker.java
new file mode 100644
index 00000000..240a3946
--- /dev/null
+++ b/accord-core/src/main/java/accord/coordinate/tracking/AwaitTracker.java
@@ -0,0 +1,137 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.coordinate.tracking;
+
+import java.util.function.Predicate;
+
+import accord.local.Node;
+import accord.primitives.Range;
+import accord.primitives.Ranges;
+import accord.primitives.Unseekables;
+import accord.topology.Shard;
+import accord.topology.Topologies;
+import accord.utils.Invariants;
+
+import static accord.coordinate.tracking.AbstractTracker.ShardOutcomes.Fail;
+import static accord.coordinate.tracking.AbstractTracker.ShardOutcomes.NoChange;
+import static accord.coordinate.tracking.AbstractTracker.ShardOutcomes.Success;
+import static accord.utils.ArrayBuffers.cachedRanges;
+
+public class AwaitTracker extends AbstractTracker<AwaitTracker.AwaitShardTracker>
+{
+ public static class AwaitShardTracker extends ShardTracker
+ {
+ protected int inFlight;
+ protected int notReady;
+ protected int ready;
+
+ public AwaitShardTracker(Shard shard)
+ {
+ super(shard);
+ inFlight = shard.rf();
+ }
+
+ public ShardOutcomes onSuccess(Boolean isReady)
+ {
+ if (isReady)
+ {
+ --inFlight;
+ return ready++ == 0 ? Success : NoChange;
+ }
+ else
+ {
+ ++notReady;
+ return --inFlight == 0 && ready == 0 ? Success : NoChange;
+ }
+ }
+
+ // return true iff hasFailed()
+ public ShardOutcomes onFailure(Object ignore)
+ {
+ if (--inFlight > 0)
+ return NoChange;
+ return hasFailed() ? Fail : Success;
+ }
+
+ public boolean hasSucceeded()
+ {
+ return notReady > 0;
+ }
+
+ boolean hasInFlight()
+ {
+ return inFlight > 0;
+ }
+
+ boolean hasFailed()
+ {
+ return inFlight == 0 && notReady == 0;
+ }
+ }
+
+ public AwaitTracker(Topologies topologies)
+ {
+ super(topologies, AwaitShardTracker[]::new, AwaitShardTracker::new);
+ Invariants.checkArgument(topologies.size() == 1, "AwaitTracker does not currently support multiple epochs; logic to compute ready keys assumes a single epoch");
+ }
+
+ public RequestStatus recordSuccess(Node.Id node, boolean isReady)
+ {
+ return recordResponse(this, node, AwaitShardTracker::onSuccess, isReady);
+ }
+
+ // return true iff hasFailed()
+ public RequestStatus recordFailure(Node.Id node)
+ {
+ return recordResponse(this, node, AwaitShardTracker::onFailure, null);
+ }
+
+ public boolean hasFailed()
+ {
+ return any(AwaitShardTracker::hasFailed);
+ }
+
+ public Unseekables<?> ready(Unseekables<?> select)
+ {
+ return select(select, t -> t.ready > 0);
+ }
+
+ public Unseekables<?> notReady(Unseekables<?> select)
+ {
+ return select(select, t -> t.ready == 0);
+ }
+
+ private Unseekables<?> select(Unseekables<?> select, Predicate<AwaitShardTracker> include)
+ {
+ // this logic assumes a single epoch
+ Range[] rangeBuffer = cachedRanges().get(trackers.length);
+ int rangeCount = 0;
+ for (int i = 0 ; i < trackers.length ; ++i)
+ {
+ if (trackers[i] == null) continue;
+ if (!include.test(trackers[i])) continue;
+ rangeBuffer[rangeCount++] = trackers[i].shard.range;
+ }
+
+ if (rangeCount == 0)
+ return select.slice(0, 0);
+ Ranges slice = Ranges.ofSortedAndDeoverlapped(cachedRanges().completeAndDiscard(rangeBuffer, rangeCount));
+ return select.slice(slice);
+ }
+}
diff --git a/accord-core/src/main/java/accord/impl/AbstractFetchCoordinator.java b/accord-core/src/main/java/accord/impl/AbstractFetchCoordinator.java
index 6f16b6c1..e79c9a79 100644
--- a/accord-core/src/main/java/accord/impl/AbstractFetchCoordinator.java
+++ b/accord-core/src/main/java/accord/impl/AbstractFetchCoordinator.java
@@ -168,6 +168,7 @@ public abstract class AbstractFetchCoordinator extends FetchCoordinator
}
FetchResponse ok = (FetchResponse) reply;
+ // TODO (required): implement support for notReady and retryInFutureEpoch, or else have option of disabling this behaviour on recipient
Ranges received;
if (ok.unavailable != null)
{
@@ -177,7 +178,7 @@ public abstract class AbstractFetchCoordinator extends FetchCoordinator
inflight.remove(key).cancel();
return;
}
- received = ranges.subtract(ok.unavailable);
+ received = ranges.without(ok.unavailable);
}
else
{
diff --git a/accord-core/src/main/java/accord/impl/CoordinateDurabilityScheduling.java b/accord-core/src/main/java/accord/impl/CoordinateDurabilityScheduling.java
index f50b6f0a..08cbbceb 100644
--- a/accord-core/src/main/java/accord/impl/CoordinateDurabilityScheduling.java
+++ b/accord-core/src/main/java/accord/impl/CoordinateDurabilityScheduling.java
@@ -314,7 +314,7 @@ public class CoordinateDurabilityScheduling
localIndexes.clear();
for (Shard shard : latestLocal.shards())
- localIndexes.put(shard, shard.sortedNodes.indexOf(node.id()));
+ localIndexes.put(shard, shard.nodes.find(node.id()));
}
/**
diff --git a/accord-core/src/main/java/accord/impl/DefaultLocalListeners.java b/accord-core/src/main/java/accord/impl/DefaultLocalListeners.java
new file mode 100644
index 00000000..d311475b
--- /dev/null
+++ b/accord-core/src/main/java/accord/impl/DefaultLocalListeners.java
@@ -0,0 +1,457 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.impl;
+
+import java.util.Arrays;
+import java.util.EnumMap;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.BiFunction;
+
+import accord.api.LocalListeners;
+import accord.api.RemoteListeners;
+import accord.local.Command;
+import accord.local.CommandStore;
+import accord.local.Commands;
+import accord.local.Node;
+import accord.local.PreLoadContext;
+import accord.local.SafeCommand;
+import accord.local.SafeCommandStore;
+import accord.local.SaveStatus;
+import accord.primitives.TxnId;
+import accord.utils.AsymmetricComparator;
+import accord.utils.Invariants;
+import accord.utils.btree.BTree;
+import accord.utils.btree.BTreeRemoval;
+
+// TODO (required): evict to disk
+public class DefaultLocalListeners implements LocalListeners
+{
+ public static class Factory implements LocalListeners.Factory
+ {
+ final RemoteListeners remoteListeners;
+ final NotifySink notifySink;
+
+ public Factory(Node node)
+ {
+ this(node, DefaultNotifySink.INSTANCE);
+ }
+
+ public Factory(Node node, NotifySink notifySink)
+ {
+ this.remoteListeners = node.remoteListeners();
+ this.notifySink = notifySink;
+ }
+
+ @Override
+ public LocalListeners create(CommandStore store)
+ {
+ return new DefaultLocalListeners(remoteListeners, notifySink);
+ }
+ }
+
+ public interface NotifySink
+ {
+ void notify(SafeCommandStore safeStore, SafeCommand safeCommand, TxnId listener);
+ boolean notify(SafeCommandStore safeStore, SafeCommand safeCommand, ComplexListener listener);
+ }
+
+ public static class DefaultNotifySink implements NotifySink
+ {
+ public static final DefaultNotifySink INSTANCE = new DefaultNotifySink();
+
+ @Override
+ public void notify(SafeCommandStore safeStore, SafeCommand safeCommand, TxnId listenerId)
+ {
+ SafeCommand listener = safeStore.ifLoadedAndInitialised(listenerId);
+ if (listener != null) Commands.listenerUpdate(safeStore, listener, safeCommand);
+ else
+ {
+ //noinspection SillyAssignment,ConstantConditions
+ safeStore = safeStore; // prevent use in lambda
+ TxnId updatedId = safeCommand.txnId();
+ PreLoadContext context = PreLoadContext.contextFor(listenerId, updatedId);
+ safeStore.commandStore()
+ .execute(context, safeStore0 -> notify(safeStore0, listenerId, updatedId))
+ .begin(safeStore.agent());
+ }
+ }
+
+ private static void notify(SafeCommandStore safeStore, TxnId listenerId, TxnId updatedId)
+ {
+ Commands.listenerUpdate(safeStore, safeStore.unsafeGet(listenerId), safeStore.unsafeGet(updatedId));
+ }
+
+ @Override
+ public boolean notify(SafeCommandStore safeStore, SafeCommand safeCommand, ComplexListener listener)
+ {
+ return listener.notify(safeStore, safeCommand);
+ }
+ }
+
+ /*
+ * A list that allows duplicates and sorts and removes duplicates on notify and when the list would have to resize
+ */
+ static class TxnListeners extends TxnId
+ {
+ final SaveStatus await;
+ TxnId[] listeners = NO_TXNIDS;
+ int count;
+
+ TxnListeners(TxnId txnId, SaveStatus await)
+ {
+ super(txnId);
+ this.await = await;
+ }
+
+ public int compareListeners(TxnListeners that)
+ {
+ int c = this.compareTo(that);
+ if (c == 0) c = this.await.compareTo(that.await);
+ return c;
+ }
+
+ public static int compareBefore(TxnId txnId, TxnListeners that)
+ {
+ int c = txnId.compareTo(that);
+ if (c == 0) c = -1;
+ return c;
+ }
+
+ public static int compare(TxnId txnId, SaveStatus await, TxnListeners that, int ifEqual)
+ {
+ int c = txnId.compareTo(that);
+ if (c == 0) c = await.compareTo(that.await);
+ if (c == 0) c = ifEqual;
+ return c;
+ }
+
+ void notify(NotifySink notifySink, SafeCommandStore safeStore, SafeCommand safeCommand)
+ {
+ trim();
+ for (int i = 0 ; i < count ; ++i)
+ {
+ TxnId listenerId = listeners[i];
+ notifySink.notify(safeStore, safeCommand, listenerId);
+ }
+ }
+
+ /*
+ * Removes duplicates
+ */
+ private int trim()
+ {
+ Arrays.sort(listeners, 0, count);
+ int removedCount = 0;
+ for (int i = 1 ; i < count ; ++i)
+ {
+ if (listeners[i - 1].compareTo(listeners[i]) == 0) ++removedCount;
+ else if (removedCount > 0) listeners[i - removedCount] = listeners[i];
+ }
+
+ if (removedCount != 0)
+ {
+ int prevCount = count;
+ count -= removedCount;
+ Arrays.fill(listeners, count, prevCount, null);
+ }
+ return removedCount;
+ }
+
+ void add(TxnId listener)
+ {
+ if (count == listeners.length)
+ {
+ if (count == 0)
+ {
+ listeners = new TxnId[4];
+ }
+ else
+ {
+ int removedCount = trim();
+ if (removedCount < listeners.length / 2)
+ {
+ TxnId[] newListeners = new TxnId[count * 2];
+ System.arraycopy(listeners, 0, newListeners, 0, count);
+ listeners = newListeners;
+ }
+ }
+ }
+
+ listeners[count++] = listener;
+ }
+ }
+
+ class RegisteredComplexListener implements Registered, BiFunction<TxnId, RegisteredComplexListeners, RegisteredComplexListeners>
+ {
+ final TxnId txnId;
+ final ComplexListener listener;
+ int index;
+
+ RegisteredComplexListener(TxnId txnId, ComplexListener listener)
+ {
+ this.listener = listener;
+ this.txnId = txnId;
+ }
+
+ @Override
+ public void cancel()
+ {
+ if (index < 0)
+ return;
+
+ complexListeners.compute(txnId, this);
+ }
+
+ @Override
+ public RegisteredComplexListeners apply(TxnId txnId, RegisteredComplexListeners listeners)
+ {
+ if (listeners == null)
+ return null;
+ return listeners.remove(this);
+ }
+ }
+
+ /**
+ * a very simple list that we can leave null entries in to make removals and reentry easier.
+ * - removal is easier because we can store an index to the array entry to null it out instantly
+ * - reentry is easier because the notifying thread doesn't need to worry about stuff moving around
+ *
+ * Methods assume mutual exclusion is guaranteed by the caller, but DOES permit reentry.
+ */
+ static class RegisteredComplexListeners
+ {
+ static final RegisteredComplexListener[] NO_LISTENERS = new RegisteredComplexListener[0];
+ RegisteredComplexListener[] listeners = NO_LISTENERS;
+ int count, length;
+ boolean notifying;
+
+ /**
+ * Append to the end of the list; if we aren't reentering from notify then if the next position
+ * in the list is unavailable and the list is half empty we first compact the list to remove null entries
+ */
+ RegisteredComplexListeners remove(RegisteredComplexListener remove)
+ {
+ int index = remove.index;
+ if (index < 0)
+ return this; // already removed
+
+ Invariants.checkState(listeners[index] == remove);
+ listeners[index] = null;
+ remove.index = -1;
+ // we don't decrement length even if count==length so as to simplify reentry
+ --count;
+ return count > 0 || notifying ? this : null;
+ }
+
+ /**
+ * Append to the end of the list; if we aren't reentering from notify then if the next position
+ * in the list is unavailable and the list is half empty we first compact the list to remove null entries;
+ * otherwise we resize the array leaving the entries in their original position
+ */
+ void add(RegisteredComplexListener add)
+ {
+ if (listeners.length == length)
+ {
+ RegisteredComplexListener[] oldListeners = listeners;
+ if (length >= count / 2 || notifying)
+ listeners = new RegisteredComplexListener[Math.max(2, length * 2)];
+
+ if (count == length || notifying)
+ {
+ // copy to same positions
+ System.arraycopy(oldListeners, 0, listeners, 0, length);
+ }
+ else
+ {
+ // copy and compact
+ int c = 0;
+ for (int i = 0 ; i < length ; ++i)
+ {
+ if (oldListeners[i] == null) continue;
+ Invariants.checkState(oldListeners[i].index == i);
+ listeners[c] = oldListeners[i];
+ listeners[c].index = c;
+ c++;
+ }
+ if (listeners == oldListeners)
+ Arrays.fill(listeners, c, length, null);
+ Invariants.checkState(c == count);
+ length = count;
+ }
+ }
+ listeners[length] = add;
+ add.index = length;
+ length++;
+ count++;
+ }
+
+ /**
+ * Notify any listeners, permitting those listeners to reenter and register/cancel listeners against this TxnId.
+ * We do this by ensuring the position of listeners doesn't change while notifying, and visiting only those
+ * listeners that were present when we started. We compact the listener collection as we go, though given
+ * reentry there is no guarantee the list at exit is compacted.
+ */
+ RegisteredComplexListeners notify(SafeCommandStore safeStore, SafeCommand safeCommand, NotifySink notifySink)
+ {
+ int count = 0;
+ int length = this.length;
+
+ notifying = true;
+ for (int i = 0 ; i < length ; ++i)
+ {
+ RegisteredComplexListener next = listeners[i];
+ if (next == null) continue;
+ Invariants.checkState(next.index == i);
+ if (!notifySink.notify(safeStore, safeCommand, listeners[i].listener))
+ {
+ if (next.index >= 0)
+ --this.count;
+ next.index = -1;
+ }
+ else if (next.index >= 0) // can be cancelled by notify, without notify return false
+ {
+ if (i != count)
+ {
+ listeners[count] = next;
+ next.index = count;
+ }
+ ++count;
+ }
+ }
+ notifying = false;
+
+ if (length != this.length)
+ {
+ // we have had some concurrent insertions (concurrent removals do not alter length)
+ // we also have some empty slots, so compact the new entries
+ for (int i = length ; i < this.length ; ++i)
+ {
+ RegisteredComplexListener next = listeners[i];
+ if (next == null)
+ continue;
+
+ Invariants.checkState(next.index == i);
+ listeners[count] = next;
+ next.index = count;
+ count++;
+ }
+ Invariants.checkState(this.count <= count); // we could have already removed some items from the compacted section
+ length = this.length;
+ }
+
+ Arrays.fill(listeners, count, length, null);
+ this.length = count;
+
+ return count == 0 ? null : this;
+ }
+ }
+
+ private static final EnumMap<SaveStatus, AsymmetricComparator<TxnId, TxnListeners>> compareExact, compareAfter;
+ static
+ {
+ compareAfter = new EnumMap<>(SaveStatus.class);
+ compareExact = new EnumMap<>(SaveStatus.class);
+ for (SaveStatus saveStatus : SaveStatus.values())
+ {
+ compareAfter.put(saveStatus, (id, listeners) -> TxnListeners.compare(id, saveStatus, listeners, 1));
+ compareExact.put(saveStatus, (id, listeners) -> TxnListeners.compare(id, saveStatus, listeners, 0));
+ }
+ }
+
+ private final RemoteListeners remoteListeners;
+ private final NotifySink notifySink;
+
+ private final ConcurrentHashMap<TxnId, RegisteredComplexListeners> complexListeners = new ConcurrentHashMap<>();
+ private Object[] txnListeners = BTree.empty();
+
+ public DefaultLocalListeners(RemoteListeners remoteListeners, NotifySink notifySink)
+ {
+ this.remoteListeners = remoteListeners;
+ this.notifySink = notifySink;
+ }
+
+ @Override
+ public void register(TxnId txnId, SaveStatus await, TxnId listener)
+ {
+ TxnListeners entry = BTree.find(txnListeners, compareExact.get(await), txnId);
+ if (entry == null)
+ txnListeners = BTree.update(txnListeners, BTree.singleton(entry = new TxnListeners(txnId, await)), TxnListeners::compareListeners);
+ entry.add(listener);
+ }
+
+ @Override
+ public Registered register(TxnId txnId, ComplexListener listener)
+ {
+ RegisteredComplexListener entry = new RegisteredComplexListener(txnId, listener);
+ complexListeners.compute(txnId, (id, cur) -> {
+ if (cur == null)
+ cur = new RegisteredComplexListeners();
+ cur.add(entry);
+ return cur;
+ });
+ return entry;
+ }
+
+ @Override
+ public void notify(SafeCommandStore safeStore, SafeCommand safeCommand, Command prev)
+ {
+ notifyTxnListeners(safeStore, safeCommand);
+ notifyComplexListeners(safeStore, safeCommand);
+ remoteListeners.notify(safeStore, safeCommand, prev);
+ }
+
+ private void notifyTxnListeners(SafeCommandStore safeStore, SafeCommand safeCommand)
+ {
+ Object[] txnListeners = this.txnListeners;
+ TxnId txnId = safeCommand.txnId();
+ SaveStatus saveStatus = safeCommand.current().saveStatus();
+ // TODO (desired): faster iteration, currently this is O(n.lg(n))
+ int start = -1 - BTree.findIndex(txnListeners, TxnListeners::compareBefore, txnId);
+ int end = -1 - BTree.findIndex(txnListeners, compareAfter.get(saveStatus), txnId);
+ while (start < end)
+ {
+ TxnListeners notify = BTree.findByIndex(txnListeners, start);
+ Invariants.checkState(txnId.equals(notify));
+ notify.notify(notifySink, safeStore, safeCommand);
+ if (this.txnListeners != txnListeners)
+ {
+ // listener registrations were changed by this listener's notify invocation, so reset our cursor
+ txnListeners = this.txnListeners;
+ start = BTree.findIndex(txnListeners, TxnListeners::compareListeners, notify);
+ end = -1 - BTree.findIndex(txnListeners, compareAfter.get(saveStatus), txnId);
+ // we only permit callers to insert into this collection, so we do not have to consider
+ // the case where the listener we are processing has been deleted under us
+ }
+
+ this.txnListeners = txnListeners = BTreeRemoval.remove(txnListeners, TxnListeners::compareListeners, notify);
+ --end;
+ }
+ this.txnListeners = txnListeners;
+ }
+
+ private void notifyComplexListeners(SafeCommandStore safeStore, SafeCommand safeCommand)
+ {
+ complexListeners.compute(safeCommand.txnId(), (id, cur) -> {
+ if (cur == null)
+ return null;
+ return cur.notify(safeStore, safeCommand, notifySink);
+ });
+ }
+
+}
diff --git a/accord-core/src/main/java/accord/impl/DefaultRemoteListeners.java b/accord-core/src/main/java/accord/impl/DefaultRemoteListeners.java
new file mode 100644
index 00000000..ba72455b
--- /dev/null
+++ b/accord-core/src/main/java/accord/impl/DefaultRemoteListeners.java
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.impl;
+
+import java.util.Arrays;
+import java.util.concurrent.ConcurrentHashMap;
+
+import com.google.common.annotations.VisibleForTesting;
+
+import accord.api.RemoteListeners;
+import accord.local.Command;
+import accord.local.Node;
+import accord.local.SafeCommand;
+import accord.local.SafeCommandStore;
+import accord.local.SaveStatus;
+import accord.local.Status.Durability;
+import accord.messages.Await.AsyncAwaitComplete;
+import accord.primitives.Route;
+import accord.primitives.TxnId;
+import accord.utils.Invariants;
+
+// TODO (required): evict to disk
+public class DefaultRemoteListeners implements RemoteListeners
+{
+ public interface NotifySink
+ {
+ void notify(TxnId txnId, SaveStatus saveStatus, Route<?> participants, long[] listeners, int listenerCount);
+ }
+
+ public static class DefaultNotifySink implements NotifySink
+ {
+ final Node node;
+ public DefaultNotifySink(Node node)
+ {
+ this.node = node;
+ }
+
+ @Override
+ public void notify(TxnId txnId, SaveStatus saveStatus, Route<?> route, long[] listeners, int listenerCount)
+ {
+ for (int i = 0 ; i < listenerCount ; ++i)
+ {
+ long listener = listeners[i];
+ // we could in theory reply to the same node multiple times here, but should not be common enough to optimise
+ Node.Id replyTo = new Node.Id(listenerNodeId(listener));
+ int callbackId = listenerCallbackId(listener);
+ node.send(replyTo, new AsyncAwaitComplete(txnId, route, saveStatus, callbackId));
+ }
+ }
+ }
+
+ static class StatusListeners
+ {
+ final int await;
+
+ /**
+ * waitingOn is a sorted list of commandStore ids that have not reached our wait condition.
+ * To save time on deletions, we only set the top bit to mark deleted until we have halved the number of entries,
+ * at which point we remove the tombstones.
+ * waitingOnSize tracks the top entry, waitingOnCount the number of live entries
+ */
+ int[] waitingOn;
+ int waitingOnSize, waitingOnCount;
+
+ /**
+ * This is a packed list of listeners, encoding the nodeId to respond to and the callbackId
+ */
+ long[] listeners;
+ int listenerCount;
+
+ StatusListeners(SaveStatus awaitSaveStatus, Durability awaitDurability, int[] waitingOn, int waitingOnCount, int listenerNodeId, int listenerCallbackId)
+ {
+ this.await = encodeAwait(awaitSaveStatus, awaitDurability);
+ this.waitingOn = waitingOn;
+ this.waitingOnCount = this.waitingOnSize = waitingOnCount;
+ this.listeners = new long[] { encodeListener(listenerNodeId, listenerCallbackId) };
+ this.listenerCount = 1;
+ }
+
+ StatusListeners merge(StatusListeners that)
+ {
+ mergeInts(this.waitingOn, this.waitingOnSize, that.waitingOn, that.waitingOnSize, StatusListeners::updateWaitingOn);
+ mergeLongs(this.listeners, this.listenerCount, that.listeners, that.listenerCount, StatusListeners::updateListeners);
+ return this;
+ }
+
+ interface IntMergeConsumer
+ {
+ void accept(StatusListeners listeners, int[] out, int count);
+ }
+
+ interface LongMergeConsumer
+ {
+ void accept(StatusListeners listeners, long[] out, int count);
+ }
+
+ private void updateWaitingOn(int[] newWaitingOn, int newWaitingOnCount)
+ {
+ this.waitingOn = newWaitingOn;
+ this.waitingOnCount = this.waitingOnSize = newWaitingOnCount;
+ Invariants.checkArgumentSorted(newWaitingOn, 0, newWaitingOnCount);
+ }
+
+ private void updateListeners(long[] newListeners, int newListenerCount)
+ {
+ this.listeners = newListeners;
+ this.listenerCount = newListenerCount;
+ Invariants.checkArgumentSorted(newListeners, 0, newListenerCount);
+ }
+
+ void mergeInts(int[] vis, int viSize, int[] vjs, int vjSize, IntMergeConsumer consumer)
+ {
+ int count = 0;
+ int i = 0, j = 0;
+ while (i < viSize && j < vjSize)
+ {
+ int vi = vis[i], vj = vjs[j];
+ if (Math.min(vi, vj) < 0)
+ {
+ i += vi >>> 31;
+ j += vj >>> 31;
+ continue;
+ }
+
+ int c = vi - vj;
+ if (c <= 0)
+ {
+ vis[count++] = vis[i++];
+ j += c == 0 ? 1 : 0;
+ }
+ else if (count < i)
+ {
+ vis[count++] = vjs[j++];
+ }
+ else
+ {
+ int[] newvis = new int[count + (viSize - i) + (vjSize - j)];
+ System.arraycopy(vis, 0, newvis, 0, count);
+ int remaining = (viSize - i);
+ System.arraycopy(vis, i, newvis, newvis.length - remaining, remaining);
+ vis = newvis;
+ i = vis.length - remaining;
+ viSize = newvis.length;
+ }
+ }
+
+ while (i < viSize)
+ {
+ if (vis[i] >= 0)
+ vis[count++] = vis[i];
+ i++;
+ }
+
+ if (vis.length < count + vjSize - j)
+ vis = Arrays.copyOf(vis, Math.max(viSize + viSize/2, count + vjSize - j));
+
+ while (j < vjSize)
+ {
+ if (vjs[j] >= 0)
+ vis[count++] = vjs[j];
+ ++j;
+ }
+
+ consumer.accept(this, vis, count);
+ }
+
+ void mergeLongs(long[] vis, int viSize, long[] vjs, int vjSize, LongMergeConsumer consumer)
+ {
+ int count = 0;
+ int i = 0, j = 0;
+ while (i < viSize && j < vjSize)
+ {
+ long vi = vis[i], vj = vjs[j];
+
+ Invariants.checkState(vi >= 0);
+ Invariants.checkState(vj >= 0);
+
+ long c = vi - vj;
+ if (c <= 0)
+ {
+ vis[count++] = vis[i++];
+ j += c == 0 ? 1 : 0;
+ }
+ else if (count < i)
+ {
+ vis[count++] = vjs[j++];
+ }
+ else
+ {
+ long[] newvis = new long[count + (viSize - i) + (vjSize - j)];
+ System.arraycopy(vis, 0, newvis, 0, count);
+ int remaining = (viSize - i);
+ System.arraycopy(vis, i, newvis, newvis.length - remaining, remaining);
+ vis = newvis;
+ i = vis.length - remaining;
+ viSize = newvis.length;
+ }
+ }
+
+ while (i < viSize)
+ {
+ if (vis[i] >= 0)
+ vis[count++] = vis[i];
+ i++;
+ }
+
+ if (vis.length < count + vjSize - j)
+ vis = Arrays.copyOf(vis, Math.max(viSize + viSize/2, count + vjSize - j));
+
+ while (j < vjSize)
+ {
+ if (vjs[j] >= 0)
+ vis[count++] = vjs[j];
+ ++j;
+ }
+
+ consumer.accept(this, vis, count);
+ }
+
+ void removeWaitingOn(int storeId)
+ {
+ int index = find(waitingOn, 0, waitingOnSize, storeId);
+ if (index >= 0 && waitingOn[index] >= 0)
+ {
+ // set tombstone flag and decrement count
+ waitingOn[index] |= Integer.MIN_VALUE;
+ --waitingOnCount;
+ if (waitingOnCount <= waitingOnSize / 2)
+ {
+ int count = 0;
+ for (int i = 0 ; i < waitingOnSize ; ++i)
+ {
+ if (waitingOn[i] >= 0)
+ waitingOn[count++] = waitingOn[i];
+ }
+ Invariants.checkState(count == waitingOnCount);
+ waitingOnSize = waitingOnCount;
+ }
+ }
+ }
+
+ // copied and simplified from SortedArrays
+ private static int find(int[] waitingOn, int from, int to, int find)
+ {
+ while (from < to)
+ {
+ int i = (from + to) >>> 1;
+ int c = Integer.compare(find, waitingOn[i] & Integer.MAX_VALUE);
+ if (c < 0) to = i;
+ else if (c > 0) from = i + 1;
+ else return i;
+ }
+ return -1 - from;
+ }
+
+ @Override
+ public String toString()
+ {
+ return awaitSaveStatus(await) + "+" + awaitDurability(await);
+ }
+ }
+
+ static class Listeners
+ {
+ /**
+ * listeners with their associated status they are waiting for;
+ * this is a sorted list populated [start..end)
+ */
+ StatusListeners[] statusListeners;
+ int start, end;
+
+ Listeners(StatusListeners statusListeners)
+ {
+ this.statusListeners = new StatusListeners[] { statusListeners };
+ this.end = 1;
+ }
+
+ Listeners merge(Listeners that)
+ {
+ int count = 0;
+ int i = this.start, j = that.start;
+ while (i < this.end && j < that.end)
+ {
+ int c = Integer.compare(this.statusListeners[i].await, that.statusListeners[j].await);
+ if (c == 0)
+ {
+ this.statusListeners[count++] = this.statusListeners[i++].merge(that.statusListeners[j++]);
+ }
+ else if (c < 0)
+ {
+ this.statusListeners[count++] = this.statusListeners[i++];
+ }
+ else if (count < i)
+ {
+ this.statusListeners[count++] = that.statusListeners[j++];
+ }
+ else
+ {
+ StatusListeners[] newStatusListeners = new StatusListeners[count + (this.end - i) + (that.end - j)];
+ System.arraycopy(statusListeners, 0, newStatusListeners, 0, count);
+ int remaining = (this.end - i);
+ System.arraycopy(statusListeners, i, newStatusListeners, newStatusListeners.length - remaining, remaining);
+ statusListeners = newStatusListeners;
+ i = statusListeners.length - remaining;
+ end = statusListeners.length;
+ }
+ }
+
+ while (i < this.end)
+ statusListeners[count++] = this.statusListeners[i++];
+
+ if (statusListeners.length < count + that.end - j)
+ statusListeners = Arrays.copyOf(statusListeners, Math.max(statusListeners.length + statusListeners.length/2, count + that.end - j));
+
+ while (j < that.end)
+ statusListeners[count++] = that.statusListeners[j++];
+
+ if (count < end)
+ Arrays.fill(statusListeners, count, end, null);
+
+ start = 0;
+ end = count;
+ return this;
+ }
+ }
+
+ class Register implements Registration
+ {
+ final TxnId txnId;
+ final SaveStatus awaitSaveStatus;
+ final Durability awaitDurability;
+ final Node.Id listeningNodeId;
+ final int callbackId;
+
+ int[] waitingOn = new int[4];
+ int count = 0;
+
+ Register(TxnId txnId, SaveStatus awaitSaveStatus, Durability awaitDurability, Node.Id listeningNodeId, int callbackId)
+ {
+ this.txnId = txnId;
+ this.awaitSaveStatus = awaitSaveStatus;
+ this.awaitDurability = awaitDurability;
+ this.listeningNodeId = listeningNodeId;
+ this.callbackId = callbackId;
+ }
+
+ @Override
+ public synchronized void add(SafeCommandStore safeStore, SafeCommand safeCommand)
+ {
+ Invariants.checkState(safeCommand.current().saveStatus().compareTo(awaitSaveStatus) < 0);
+ if (count == waitingOn.length)
+ {
+ trim();
+ if (count >= waitingOn.length/2)
+ {
+ int[] newWaitingOn = new int[count * 2];
+ System.arraycopy(waitingOn, 0, newWaitingOn, 0, count);
+ waitingOn = newWaitingOn;
+ }
+ }
+ waitingOn[count++] = safeStore.commandStore().id();
+ }
+
+ private void trim()
+ {
+ Arrays.sort(waitingOn, 0, count);
+ int removedCount = 0;
+ for (int i = 1 ; i < count ; ++i)
+ {
+ if (waitingOn[i - 1] == waitingOn[i]) ++removedCount;
+ else if (removedCount > 0) waitingOn[i - removedCount] = waitingOn[i];
+ }
+ count -= removedCount;
+ }
+
+ @Override
+ public int done()
+ {
+ if (count == 0)
+ return 0;
+
+ trim();
+ if (waitingOn.length > 4 && count < waitingOn.length / 2)
+ waitingOn = Arrays.copyOf(waitingOn, count);
+
+ StatusListeners listener = new StatusListeners(awaitSaveStatus, awaitDurability, waitingOn, count, listeningNodeId.id, callbackId);
+ listeners.merge(txnId, new Listeners(listener), Listeners::merge);
+ return count;
+ }
+ }
+
+ final NotifySink notifySink;
+ final ConcurrentHashMap<TxnId, Listeners> listeners = new ConcurrentHashMap<>();
+
+ public DefaultRemoteListeners(NotifySink notifySink)
+ {
+ this.notifySink = notifySink;
+ }
+
+ public DefaultRemoteListeners(Node node)
+ {
+ this.notifySink = new DefaultNotifySink(node);
+ }
+
+ @Override
+ public Registration register(TxnId txnId, SaveStatus awaitSaveStatus, Durability awaitDurability, Node.Id listener, int callbackId)
+ {
+ Invariants.checkArgument(callbackId >= 0);
+ return new Register(txnId, awaitSaveStatus, awaitDurability, listener, callbackId);
+ }
+
+ @Override
+ public void notify(SafeCommandStore safeStore, SafeCommand safeCommand, Command prev)
+ {
+ TxnId txnId = safeCommand.txnId();
+ Listeners entry = this.listeners.get(txnId);
+ if (entry == null)
+ return;
+
+ int storeId = safeStore.commandStore().id();
+ SaveStatus newStatus = safeCommand.current().saveStatus();
+ Durability newDurability = safeCommand.current().durability();
+
+ int start = entry.start, end = entry.end;
+ StatusListeners[] listeners = entry.statusListeners;
+ for (int i = start ; i < end ; ++i)
+ {
+ StatusListeners listener = listeners[i];
+ if (awaitSaveStatus(listener.await).compareTo(newStatus) > 0)
+ return;
+
+ if (awaitDurability(listener.await).compareTo(newDurability) > 0)
+ continue;
+
+ listener.removeWaitingOn(storeId);
+ if (listener.waitingOnCount == 0)
+ {
+ // if we get invalidated we don't save the route, so we take the combined route of before and after the new status
+ Route<?> route = Route.merge(safeCommand.current().route(), prev == null ? null : (Route)prev.route());
+ notifySink.notify(txnId, newStatus, route, listener.listeners, listener.listenerCount);
+ if (i != start)
+ System.arraycopy(listeners, start, listeners, start + 1, i - start);
+ listeners[start] = null;
+ start = ++entry.start;
+ }
+ }
+ }
+
+ private static int listenerNodeId(long encodedListener)
+ {
+ return (int) (encodedListener >>> 31);
+ }
+
+ private static int listenerCallbackId(long encodedListener)
+ {
+ return (int) (encodedListener & 0x3fffffff);
+ }
+
+ @VisibleForTesting
+ public static long encodeListener(int nodeId, int callbackId)
+ {
+ Invariants.checkArgument(callbackId >= 0);
+ return callbackId | ((long)nodeId << 31);
+ }
+
+ static
+ {
+ Invariants.checkState(Durability.maxOrdinal() < 16);
+ }
+
+ private static SaveStatus awaitSaveStatus(int encodedAwait)
+ {
+ return SaveStatus.forOrdinal(encodedAwait >>> 4);
+ }
+
+ private static Durability awaitDurability(int encodedAwait)
+ {
+ return Durability.forOrdinal(encodedAwait & 0xf);
+ }
+
+ private static int encodeAwait(SaveStatus awaitSaveStatus, Durability awaitDurability)
+ {
+ return (awaitSaveStatus.ordinal() << 4) | awaitDurability.ordinal();
+ }
+}
diff --git a/accord-core/src/main/java/accord/impl/DefaultRequestTimeouts.java b/accord-core/src/main/java/accord/impl/DefaultRequestTimeouts.java
new file mode 100644
index 00000000..0407d1e8
--- /dev/null
+++ b/accord-core/src/main/java/accord/impl/DefaultRequestTimeouts.java
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.impl;
+
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.function.Function;
+
+import accord.api.RequestTimeouts;
+import accord.local.Node;
+import accord.utils.LogGroupTimers;
+
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+import static org.agrona.BitUtil.findNextPositivePowerOfTwo;
+
+public class DefaultRequestTimeouts implements RequestTimeouts
+{
+ static class Stripe implements Runnable, Function<Stripe.Registered, Runnable>
+ {
+ class Registered extends LogGroupTimers.Timer implements RegisteredTimeout
+ {
+ final Timeout timeout;
+
+ Registered(Timeout timeout)
+ {
+ this.timeout = timeout;
+ }
+
+ @Override
+ public void cancel()
+ {
+ lock.lock();
+ try
+ {
+ timeouts.remove(this);
+ }
+ finally
+ {
+ lock.unlock();
+ }
+ }
+ }
+
+ final ReentrantLock lock = new ReentrantLock();
+ final Node node;
+ final LogGroupTimers<Registered> timeouts = new LogGroupTimers<>(TimeUnit.MILLISECONDS);
+ final LogGroupTimers<Registered>.Scheduling timeoutScheduling;
+
+ public Stripe(Node node)
+ {
+ this.node = node;
+ this.timeoutScheduling = timeouts.new Scheduling(node.scheduler(), this,
+ 100, 1L, SECONDS.toMillis(1));
+ }
+
+ public RegisteredTimeout tryRegister(Timeout timeout, long now, long deadline)
+ {
+ if (!lock.tryLock())
+ return null;
+
+ try
+ {
+ Registered registered = new Registered(timeout);
+ timeouts.add(deadline, registered);
+ timeoutScheduling.maybeReschedule(now, deadline);
+ return registered;
+ }
+ finally
+ {
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public void run()
+ {
+ lock.lock();
+ try
+ {
+ long now = node.elapsed(MILLISECONDS);
+ // TODO (expected): should we handle reentrancy? Or at least throw an exception?
+ timeouts.advance(now, this, (s, r) -> r.timeout.timeout());
+ }
+ finally
+ {
+ lock.unlock();
+ }
+ }
+
+ @Override
+ public Runnable apply(Registered registered)
+ {
+ return this;
+ }
+ }
+
+ final Node node;
+ final Stripe[] stripes;
+
+ public DefaultRequestTimeouts(Node node)
+ {
+ this(node, 8);
+ }
+
+ public DefaultRequestTimeouts(Node node, int stripeCount)
+ {
+ if (stripeCount > 1024)
+ throw new IllegalArgumentException("Far too many stripes requested");
+ this.stripes = new Stripe[findNextPositivePowerOfTwo(stripeCount)];
+ for (int i = 0 ; i < stripes.length; ++i)
+ stripes[i] = new Stripe(node);
+ this.node = node;
+ }
+
+ @Override
+ public RegisteredTimeout register(Timeout timeout, long delay, TimeUnit units)
+ {
+ long now = node.elapsed(MILLISECONDS);
+ long deadline = now + Math.max(1, MILLISECONDS.convert(delay, units));
+ int i = timeout.hashCode() & (stripes.length - 1);
+ while (true)
+ {
+ RegisteredTimeout result = stripes[i].tryRegister(timeout, now, deadline);
+ if (result != null)
+ return result;
+ i = (i + 1) & (stripes.length - 1);
+ }
+ }
+}
diff --git a/accord-core/src/main/java/accord/impl/ErasedSafeCommand.java b/accord-core/src/main/java/accord/impl/ErasedSafeCommand.java
index d2285f91..0da04ca7 100644
--- a/accord-core/src/main/java/accord/impl/ErasedSafeCommand.java
+++ b/accord-core/src/main/java/accord/impl/ErasedSafeCommand.java
@@ -19,14 +19,11 @@
package accord.impl;
import accord.local.Command;
-import accord.local.Command.TransientListener;
-import accord.local.Listeners;
import accord.local.SafeCommand;
import accord.local.SaveStatus;
import accord.primitives.TxnId;
import accord.utils.Invariants;
-import static accord.local.Listeners.Immutable.EMPTY;
import static accord.local.SaveStatus.Erased;
import static accord.local.SaveStatus.ErasedOrInvalidOrVestigial;
import static accord.local.Status.Durability.NotDurable;
@@ -40,7 +37,7 @@ public class ErasedSafeCommand extends SafeCommand
{
super(txnId);
Invariants.checkArgument(saveStatus == Erased || saveStatus == ErasedOrInvalidOrVestigial);
- this.erased = new Command.Truncated(txnId, saveStatus, saveStatus == ErasedOrInvalidOrVestigial ? NotDurable : UniversalOrInvalidated, null, null, EMPTY, null, null);
+ this.erased = new Command.Truncated(txnId, saveStatus, saveStatus == ErasedOrInvalidOrVestigial ? NotDurable : UniversalOrInvalidated, null, null, null, null);
}
@Override
@@ -61,24 +58,6 @@ public class ErasedSafeCommand extends SafeCommand
return false;
}
- @Override
- public void addListener(Command.TransientListener listener)
- {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public boolean removeListener(Command.TransientListener listener)
- {
- throw new UnsupportedOperationException();
- }
-
- @Override
- public Listeners<TransientListener> transientListeners()
- {
- return Listeners.EMPTY;
- }
-
@Override
protected void set(Command command)
{
diff --git a/accord-core/src/main/java/accord/impl/InMemoryCommandStore.java b/accord-core/src/main/java/accord/impl/InMemoryCommandStore.java
index cedc2161..bcc6d6f7 100644
--- a/accord-core/src/main/java/accord/impl/InMemoryCommandStore.java
+++ b/accord-core/src/main/java/accord/impl/InMemoryCommandStore.java
@@ -33,6 +33,7 @@ import java.util.concurrent.Callable;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
+import java.util.concurrent.TimeUnit;
import java.util.function.BiConsumer;
import java.util.function.BiFunction;
import java.util.function.Consumer;
@@ -42,6 +43,8 @@ import javax.annotation.Nullable;
import com.google.common.annotations.VisibleForTesting;
+import accord.api.LocalListeners;
+import accord.impl.progresslog.DefaultProgressLog;
import accord.local.*;
import org.slf4j.Logger;
@@ -53,7 +56,6 @@ import accord.api.Key;
import accord.api.ProgressLog;
import accord.local.CommandStores.RangesForEpoch;
import accord.local.cfk.CommandsForKey;
-import accord.local.cfk.SafeCommandsForKey;
import accord.primitives.AbstractKeys;
import accord.primitives.Deps;
import accord.primitives.PartialDeps;
@@ -64,6 +66,7 @@ import accord.primitives.Routable;
import accord.primitives.Routable.Domain;
import accord.primitives.RoutableKey;
import accord.primitives.Routables;
+import accord.primitives.Route;
import accord.primitives.Seekable;
import accord.primitives.Seekables;
import accord.primitives.Timestamp;
@@ -82,6 +85,7 @@ import static accord.local.SaveStatus.Erased;
import static accord.local.SaveStatus.ErasedOrInvalidOrVestigial;
import static accord.local.SaveStatus.ReadyToExecute;
import static accord.local.Status.Applied;
+import static accord.local.Status.PreCommitted;
import static accord.local.Status.Stable;
import static accord.local.Status.Truncated;
import static accord.local.Status.NotDefined;
@@ -106,9 +110,9 @@ public abstract class InMemoryCommandStore extends CommandStore
private InMemorySafeStore current;
- public InMemoryCommandStore(int id, NodeTimeService time, Agent agent, DataStore store, ProgressLog.Factory progressLogFactory, EpochUpdateHolder epochUpdateHolder)
+ public InMemoryCommandStore(int id, NodeTimeService time, Agent agent, DataStore store, ProgressLog.Factory progressLogFactory, LocalListeners.Factory listenersFactory, EpochUpdateHolder epochUpdateHolder)
{
- super(id, time, agent, store, progressLogFactory, epochUpdateHolder);
+ super(id, time, agent, store, progressLogFactory, listenersFactory, epochUpdateHolder);
}
protected boolean canExposeUnloaded()
@@ -164,10 +168,10 @@ public abstract class InMemoryCommandStore extends CommandStore
{
if (CHECK_DEPENDENCY_INVARIANTS)
{
- newGlobalCommand.addListener(new Command.TransientListener()
+ listeners.register(newGlobalCommand.txnId, new LocalListeners.ComplexListener()
{
@Override
- public void onChange(SafeCommandStore safeStore, SafeCommand safeCommand)
+ public boolean notify(SafeCommandStore safeStore, SafeCommand safeCommand)
{
Command cur = safeCommand.current();
if (cur.saveStatus() == ReadyToExecute || cur.saveStatus() == Applying) // TODO (desired): only run the check once
@@ -211,15 +215,10 @@ public abstract class InMemoryCommandStore extends CommandStore
Invariants.illegalState(cur.txnId() + " does not maintain dependency invariants with immediately preceding transaction " + prev.txnId() + "; intersecting participants: " + intersectingParticipants + "; dependency participants: " + depParticipants);
if (prev.txnId().isWrite())
- participants = participants.subtract(intersectingParticipants);
+ participants = participants.without(intersectingParticipants);
}
}
- }
-
- @Override
- public PreLoadContext listenerPreLoadContext(TxnId caller)
- {
- return PreLoadContext.contextFor(caller);
+ return !cur.hasBeen(Status.Applied);
}
});
}
@@ -346,11 +345,13 @@ public abstract class InMemoryCommandStore extends CommandStore
@Override
protected void updatedRedundantBefore(SafeCommandStore safeStore, TxnId syncId, Ranges ranges)
{
+ InMemorySafeStore inMemorySafeStore = (InMemorySafeStore) safeStore;
ranges.forEach(r -> {
commandsForKey.subMap(r.start(), r.startInclusive(), r.end(), r.endInclusive()).forEach((forKey, forValue) -> {
if (!forValue.isEmpty())
{
- SafeCommandsForKey safeCfk = forValue.createSafeReference();
+ InMemorySafeCommandsForKey safeCfk = forValue.createSafeReference();
+ inMemorySafeStore.commandsForKey.put(forKey, safeCfk);
safeCfk.refresh(safeStore);
}
});
@@ -370,7 +371,7 @@ public abstract class InMemoryCommandStore extends CommandStore
rangeCommands.entrySet().removeIf(tx -> {
if (tx.getKey().compareTo(syncId) >= 0)
return false;
- Ranges newRanges = tx.getValue().ranges.subtract(ranges);
+ Ranges newRanges = tx.getValue().ranges.without(ranges);
if (!newRanges.isEmpty())
{
tx.getValue().ranges = newRanges;
@@ -382,6 +383,29 @@ public abstract class InMemoryCommandStore extends CommandStore
return true;
}
});
+
+ // verify we're clearing the progress log
+ ((Node)time).scheduler().once(() -> {
+ DefaultProgressLog progressLog = (DefaultProgressLog) this.progressLog;
+ commands.headMap(syncId, false).forEach((id, cmd) -> {
+ Command command = cmd.value();
+ if (!command.hasBeen(PreCommitted)) return;
+ if (!command.txnId().kind().isGloballyVisible()) return;
+
+ Ranges allRanges = unsafeRangesForEpoch().allBetween(id.epoch(), command.executeAtOrTxnId().epoch());
+ boolean done = command.hasBeen(Truncated);
+ if (!done)
+ {
+ if (redundantBefore().status(cmd.txnId, command.executeAtOrTxnId(), command.route()) == RedundantStatus.PRE_BOOTSTRAP_OR_STALE)
+ return;
+
+ Route<?> route = cmd.value().route().slice(allRanges);
+ done = !route.isEmpty() && ranges.containsAll(route);
+ }
+
+ if (done) Invariants.checkState(progressLog.get(id) == null);
+ });
+ }, 5L, TimeUnit.SECONDS);
}
@@ -580,7 +604,6 @@ public abstract class InMemoryCommandStore extends CommandStore
public static class GlobalCommand extends GlobalState<Command>
{
private final TxnId txnId;
- private Listeners<Command.TransientListener> transientListeners = null;
public GlobalCommand(TxnId txnId)
{
@@ -592,33 +615,11 @@ public abstract class InMemoryCommandStore extends CommandStore
return new InMemorySafeCommand(txnId, this);
}
- public void addListener(Command.TransientListener listener)
- {
- if (transientListeners == null) transientListeners = new Listeners<>();
- transientListeners.add(listener);
- }
-
- public boolean removeListener(Command.TransientListener listener)
- {
- if (transientListeners == null || !transientListeners.remove(listener))
- return false;
-
- if (transientListeners.isEmpty())
- transientListeners = null;
-
- return true;
- }
-
@Override
public GlobalState<Command> value(Command value)
{
return super.value(value);
}
-
- public Listeners<Command.TransientListener> transientListeners()
- {
- return transientListeners == null ? Listeners.EMPTY : transientListeners;
- }
}
public static class GlobalCommandsForKey extends GlobalState<CommandsForKey>
@@ -756,7 +757,7 @@ public abstract class InMemoryCommandStore extends CommandStore
return;
Ranges slice = ranges().allBetween(txnId, updated.executeAtOrTxnId());
- slice = commandStore.redundantBefore().removeShardRedundant(txnId, updated.executeAt(), slice);
+ slice = commandStore.redundantBefore().removeShardRedundant(txnId, updated.executeAtOrTxnId(), slice);
commandStore.rangeCommands.computeIfAbsent(txnId, ignore -> new RangeCommand(commandStore.commands.get(txnId)))
.update(((Ranges)keysOrRanges).slice(slice, Minimal));
@@ -1054,9 +1055,9 @@ public abstract class InMemoryCommandStore extends CommandStore
Runnable active = null;
final Queue<Runnable> queue = new ConcurrentLinkedQueue<>();
- public Synchronized(int id, NodeTimeService time, Agent agent, DataStore store, ProgressLog.Factory progressLogFactory, EpochUpdateHolder epochUpdateHolder)
+ public Synchronized(int id, NodeTimeService time, Agent agent, DataStore store, ProgressLog.Factory progressLogFactory, LocalListeners.Factory listenersFactory, EpochUpdateHolder epochUpdateHolder)
{
- super(id, time, agent, store, progressLogFactory, epochUpdateHolder);
+ super(id, time, agent, store, progressLogFactory, listenersFactory, epochUpdateHolder);
}
private synchronized void maybeRun()
@@ -1146,9 +1147,9 @@ public abstract class InMemoryCommandStore extends CommandStore
private Thread thread; // when run in the executor this will be non-null, null implies not running in this store
private final ExecutorService executor;
- public SingleThread(int id, NodeTimeService time, Agent agent, DataStore store, ProgressLog.Factory progressLogFactory, EpochUpdateHolder epochUpdateHolder)
+ public SingleThread(int id, NodeTimeService time, Agent agent, DataStore store, ProgressLog.Factory progressLogFactory, LocalListeners.Factory listenersFactory, EpochUpdateHolder epochUpdateHolder)
{
- super(id, time, agent, store, progressLogFactory, epochUpdateHolder);
+ super(id, time, agent, store, progressLogFactory, listenersFactory, epochUpdateHolder);
this.executor = Executors.newSingleThreadExecutor(r -> {
Thread thread = new Thread(r);
thread.setName(CommandStore.class.getSimpleName() + '[' + time.id() + ']');
@@ -1230,9 +1231,9 @@ public abstract class InMemoryCommandStore extends CommandStore
}
}
- public Debug(int id, NodeTimeService time, Agent agent, DataStore store, ProgressLog.Factory progressLogFactory, EpochUpdateHolder epochUpdateHolder)
+ public Debug(int id, NodeTimeService time, Agent agent, DataStore store, ProgressLog.Factory progressLogFactory, LocalListeners.Factory listenersFactory, EpochUpdateHolder epochUpdateHolder)
{
- super(id, time, agent, store, progressLogFactory, epochUpdateHolder);
+ super(id, time, agent, store, progressLogFactory, listenersFactory, epochUpdateHolder);
}
@Override
diff --git a/accord-core/src/main/java/accord/impl/InMemoryCommandStores.java b/accord-core/src/main/java/accord/impl/InMemoryCommandStores.java
index a92565d3..fdb94d8a 100644
--- a/accord-core/src/main/java/accord/impl/InMemoryCommandStores.java
+++ b/accord-core/src/main/java/accord/impl/InMemoryCommandStores.java
@@ -18,6 +18,7 @@
package accord.impl;
+import accord.api.LocalListeners;
import accord.local.*;
import accord.api.Agent;
import accord.api.DataStore;
@@ -28,30 +29,30 @@ public class InMemoryCommandStores
{
public static class Synchronized extends CommandStores
{
- public Synchronized(NodeTimeService time, Agent agent, DataStore store, RandomSource random, ShardDistributor shardDistributor, ProgressLog.Factory progressLogFactory)
+ public Synchronized(NodeTimeService time, Agent agent, DataStore store, RandomSource random, ShardDistributor shardDistributor, ProgressLog.Factory progressLogFactory, LocalListeners.Factory listenersFactory)
{
- super(time, agent, store, random, shardDistributor, progressLogFactory, InMemoryCommandStore.Synchronized::new);
+ super(time, agent, store, random, shardDistributor, progressLogFactory, listenersFactory, InMemoryCommandStore.Synchronized::new);
}
}
public static class SingleThread extends CommandStores
{
- public SingleThread(NodeTimeService time, Agent agent, DataStore store, RandomSource random, ShardDistributor shardDistributor, ProgressLog.Factory progressLogFactory)
+ public SingleThread(NodeTimeService time, Agent agent, DataStore store, RandomSource random, ShardDistributor shardDistributor, ProgressLog.Factory progressLogFactory, LocalListeners.Factory listenersFactory)
{
- super(time, agent, store, random, shardDistributor, progressLogFactory, InMemoryCommandStore.SingleThread::new);
+ super(time, agent, store, random, shardDistributor, progressLogFactory, listenersFactory, InMemoryCommandStore.SingleThread::new);
}
- public SingleThread(NodeTimeService time, Agent agent, DataStore store, RandomSource random, ShardDistributor shardDistributor, ProgressLog.Factory progressLogFactory, CommandStore.Factory shardFactory)
+ public SingleThread(NodeTimeService time, Agent agent, DataStore store, RandomSource random, ShardDistributor shardDistributor, ProgressLog.Factory progressLogFactory, LocalListeners.Factory listenersFactory, CommandStore.Factory shardFactory)
{
- super(time, agent, store, random, shardDistributor, progressLogFactory, shardFactory);
+ super(time, agent, store, random, shardDistributor, progressLogFactory, listenersFactory, shardFactory);
}
}
public static class Debug extends InMemoryCommandStores.SingleThread
{
- public Debug(NodeTimeService time, Agent agent, DataStore store, RandomSource random, ShardDistributor shardDistributor, ProgressLog.Factory progressLogFactory)
+ public Debug(NodeTimeService time, Agent agent, DataStore store, RandomSource random, ShardDistributor shardDistributor, ProgressLog.Factory progressLogFactory, LocalListeners.Factory listenersFactory)
{
- super(time, agent, store, random, shardDistributor, progressLogFactory, InMemoryCommandStore.Debug::new);
+ super(time, agent, store, random, shardDistributor, progressLogFactory, listenersFactory, InMemoryCommandStore.Debug::new);
}
}
}
diff --git a/accord-core/src/main/java/accord/impl/InMemorySafeCommand.java b/accord-core/src/main/java/accord/impl/InMemorySafeCommand.java
index 946f255a..3fe3c7fb 100644
--- a/accord-core/src/main/java/accord/impl/InMemorySafeCommand.java
+++ b/accord-core/src/main/java/accord/impl/InMemorySafeCommand.java
@@ -25,7 +25,6 @@ import javax.annotation.Nullable;
import accord.impl.InMemoryCommandStore.GlobalCommand;
import accord.local.Command;
-import accord.local.Listeners;
import accord.local.SafeCommand;
import accord.primitives.TxnId;
@@ -82,24 +81,6 @@ public class InMemorySafeCommand extends SafeCommand implements SafeState<Comman
global.value(update);
}
- @Override
- public void addListener(Command.TransientListener listener)
- {
- global.addListener(listener);
- }
-
- @Override
- public boolean removeListener(Command.TransientListener listener)
- {
- return global.removeListener(listener);
- }
-
- @Override
- public Listeners<Command.TransientListener> transientListeners()
- {
- return global.transientListeners();
- }
-
@Override
public void invalidate()
{
diff --git a/accord-core/src/main/java/accord/impl/SimpleProgressLog.java b/accord-core/src/main/java/accord/impl/SimpleProgressLog.java
deleted file mode 100644
index 2dc57a6d..00000000
--- a/accord-core/src/main/java/accord/impl/SimpleProgressLog.java
+++ /dev/null
@@ -1,729 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package accord.impl;
-
-import java.time.Duration;
-import java.util.List;
-import java.util.Map;
-import java.util.TreeMap;
-import java.util.concurrent.CopyOnWriteArrayList;
-import java.util.concurrent.TimeUnit;
-import java.util.function.BiConsumer;
-import javax.annotation.Nullable;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import accord.api.ProgressLog;
-import accord.coordinate.CoordinationFailed;
-import accord.coordinate.FetchData;
-import accord.coordinate.Outcome;
-import accord.local.Command;
-import accord.local.CommandStore;
-import accord.local.Node;
-import accord.local.SafeCommand;
-import accord.local.SafeCommandStore;
-import accord.local.SaveStatus;
-import accord.local.SaveStatus.LocalExecution;
-import accord.local.Status.Known;
-import accord.primitives.EpochSupplier;
-import accord.primitives.Participants;
-import accord.primitives.ProgressToken;
-import accord.primitives.Ranges;
-import accord.primitives.Route;
-import accord.primitives.Timestamp;
-import accord.primitives.TxnId;
-import accord.primitives.Unseekables;
-import accord.utils.IntrusiveLinkedList;
-import accord.utils.IntrusiveLinkedListNode;
-import accord.utils.Invariants;
-import accord.utils.async.AsyncChain;
-import accord.utils.async.AsyncResult;
-
-import static accord.api.ProgressLog.ProgressShard.Unsure;
-import static accord.coordinate.InformHomeOfTxn.inform;
-import static accord.impl.SimpleProgressLog.CoordinateStatus.NotStable;
-import static accord.impl.SimpleProgressLog.CoordinateStatus.ReadyToExecute;
-import static accord.impl.SimpleProgressLog.Progress.Done;
-import static accord.impl.SimpleProgressLog.Progress.Expected;
-import static accord.impl.SimpleProgressLog.Progress.Investigating;
-import static accord.impl.SimpleProgressLog.Progress.NoProgress;
-import static accord.impl.SimpleProgressLog.Progress.NoneExpected;
-import static accord.local.PreLoadContext.contextFor;
-import static accord.local.PreLoadContext.empty;
-import static accord.local.SaveStatus.LocalExecution.NotReady;
-import static accord.local.SaveStatus.LocalExecution.WaitingToApply;
-import static accord.local.Status.KnownRoute.Full;
-import static accord.local.Status.PreApplied;
-import static accord.utils.Invariants.illegalState;
-
-// TODO (desired, consider): consider propagating invalidations in the same way as we do applied
-// TODO (expected): report long-lived recurring transactions / operations
-public class SimpleProgressLog implements ProgressLog.Factory
-{
- private static final Logger logger = LoggerFactory.getLogger(SimpleProgressLog.class);
-
- enum Progress { NoneExpected, Expected, NoProgress, Investigating, Done }
-
- public static volatile boolean PAUSE_FOR_TEST = false;
-
- enum CoordinateStatus
- {
- NotWitnessed, NotStable, Stable, ReadyToExecute, Done;
-
- boolean isAtMostReadyToExecute()
- {
- return compareTo(CoordinateStatus.ReadyToExecute) <= 0;
- }
-
- boolean isAtLeastCommitted()
- {
- return compareTo(CoordinateStatus.Stable) >= 0;
- }
- }
-
- final Node node;
- final List<Instance> instances = new CopyOnWriteArrayList<>();
-
- public SimpleProgressLog(Node node)
- {
- this.node = node;
- }
-
- class Instance extends IntrusiveLinkedList<Instance.State.Monitoring> implements ProgressLog, Runnable
- {
- class State
- {
- abstract class Monitoring extends IntrusiveLinkedListNode
- {
- private Progress progress = NoneExpected;
-
- void setProgress(Progress newProgress)
- {
- Invariants.checkState(progress != Done);
-
- progress = newProgress;
- switch (newProgress)
- {
- default: throw new AssertionError();
- case NoneExpected:
- case Done:
- case Investigating:
- remove();
- Invariants.paranoid(isFree());
- break;
- case Expected:
- case NoProgress:
- if (isFree())
- addFirst(this);
- }
- }
-
- void ensureDone()
- {
- if (progress != Done)
- setProgress(Done);
- }
-
- boolean shouldRun()
- {
- switch (progress)
- {
- default: throw new AssertionError("Unexpected progress: " + progress);
- case NoneExpected:
- case Done:
- return false;
- case Investigating:
- throw illegalState("Unexpected progress: " + progress);
- case Expected:
- Invariants.paranoid(!isFree());
- progress = NoProgress;
- return false;
- case NoProgress:
- remove();
- return true;
- }
- }
-
- abstract void run(SafeCommandStore safeStore, SafeCommand safeCommand);
-
- Progress progress()
- {
- return progress;
- }
-
- TxnId txnId()
- {
- return txnId;
- }
- }
-
- // exists only on home shard
- // TODO (expected): should not take any prompt action if we're ourselves already coordinating the transaction
- class CoordinateState extends Monitoring
- {
- CoordinateStatus status = CoordinateStatus.NotWitnessed;
- ProgressToken token = ProgressToken.NONE;
-
- void ensureAtLeast(Command command, CoordinateStatus newStatus, Progress newProgress)
- {
- ensureAtLeast(newStatus, newProgress);
- updateMax(command);
- }
-
- void ensureAtLeast(CoordinateStatus newStatus, Progress newProgress)
- {
- if (newStatus.compareTo(status) > 0)
- {
- status = newStatus;
- setProgress(newProgress);
- }
- }
-
- void updateMax(Command command)
- {
- token = token.merge(new ProgressToken(command.durability(), command.status(), command.promised(), command.acceptedOrCommitted()));
- }
-
- void updateMax(ProgressToken ok)
- {
- // TODO (low priority): perhaps set localProgress back to Waiting if Investigating and we update anything?
- token = token.merge(ok);
- if (token.durability.isDurableOrInvalidated())
- durableGlobal();
- }
-
- void durableGlobal()
- {
- switch (status)
- {
- default: throw new IllegalStateException();
- case NotWitnessed:
- case NotStable:
- case Stable:
- case ReadyToExecute:
- status = CoordinateStatus.Done;
- setProgress(Done);
- case Done:
- }
- }
-
- @Override
- void run(SafeCommandStore safeStore, SafeCommand safeCommand)
- {
- Command command = safeCommand.current();
- Invariants.checkState(!safeStore.isTruncated(command), "Command %s is truncated", command);
- setProgress(Investigating);
- switch (status)
- {
- default: throw new AssertionError("Unexpected status: " + status);
- case NotWitnessed: // can't make progress if we haven't witnessed it yet
- case Stable: // can't make progress if we aren't yet ReadyToExecute
- case Done: // shouldn't be trying to make progress, as we're done
- throw illegalState("Unexpected status: " + status);
-
- case ReadyToExecute:
- // TODO (expected): we should have already exited this loop if this conditions is true
- if (command.durability().isDurableOrInvalidated())
- {
- // should not reach here with an invalidated state
- Invariants.checkState(command.durability().isDurable(), "Command %s is not durable", command);
- durableGlobal();
- return;
- }
-
- case NotStable:
- {
- Invariants.checkState(!token.durability.isDurableOrInvalidated(), "ProgressToken is durable invalidated, but we have not cleared the ProgressLog");
- Invariants.checkState(!command.durability().isDurableOrInvalidated(), "Command is durable or invalidated, but we have not cleared the ProgressLog");
-
- Route<?> route = Invariants.nonNull(command.route()).withHomeKey();
- node.withEpoch(txnId.epoch(), (ignored, withEpochFailure) -> {
- if (withEpochFailure != null)
- {
- node.agent().onUncaughtException(CoordinationFailed.wrap(withEpochFailure));
- commandStore.execute(contextFor(txnId), safeStore0 -> {
- if (status.isAtMostReadyToExecute() && progress() == Investigating)
- {
- setProgress(Expected);
- }
- }).begin(node.agent());
- return;
- }
-
- AsyncResult<? extends Outcome> recover = node.maybeRecover(txnId, route, token);
- recover.addCallback((success, fail) -> {
- // TODO (expected): callback should be on safeStore, and should provide safeStore as a parameter
- commandStore.execute(contextFor(txnId), safeStore0 -> {
- if (status.isAtMostReadyToExecute() && progress() == Investigating)
- {
- setProgress(Expected);
- if (fail != null)
- return;
-
- updateMax(success.asProgressToken());
- }
- }).begin(node.agent());
- });
- });
- }
- }
- }
-
- @Override
- public String toString()
- {
- return "{" + status + ',' + progress() + '}';
- }
- }
-
- class BlockingState extends State.Monitoring
- {
- LocalExecution blockedUntil = NotReady;
-
- Route<?> route;
- Participants<?> participants;
-
- @SuppressWarnings({"unchecked", "rawtypes"})
- void recordBlocking(LocalExecution blockedUntil, @Nullable Route<?> route, @Nullable Participants<?> participants)
- {
- Invariants.checkState(route != null || participants != null, "Route and participants are both undefined");
- Invariants.checkState(participants == null || !participants.isEmpty(), "participants is empty");
- Invariants.checkState(route == null || !route.isEmpty(), "Route %s is empty", route);
-
- this.route = Route.merge(this.route, (Route)route);
- this.participants = Participants.merge(this.participants, (Participants) participants);
- if (blockedUntil.compareTo(this.blockedUntil) > 0)
- {
- this.blockedUntil = blockedUntil;
- setProgress(Expected);
- }
- }
-
- void record(Known known)
- {
- // invalidation coordination callback may fire
- // before invalidation is committed locally
- if (progress() == Done)
- return;
-
- if (blockedUntil.isSatisfiedBy(known))
- setProgress(NoneExpected);
- }
-
- @Override
- void run(SafeCommandStore safeStore, SafeCommand safeCommand)
- {
- Command command = safeCommand.current();
- if (command.isAtLeast(blockedUntil))
- {
- setProgress(NoneExpected);
- return;
- }
-
- setProgress(Investigating);
- // first make sure we have enough information to obtain the command locally
- Timestamp executeAt = command.executeAtIfKnown();
- // we want to fetch a route if we have it, so that we can go to our neighbouring shards for info
- // (rather than the home shard, which may have GC'd its state if the result is durable)
- Unseekables<?> fetchKeys = maxContact(command);
- EpochSupplier forLocalEpoch = safeStore.ranges().latestEpochWithNewParticipants(txnId.epoch(), fetchKeys);
-
- BiConsumer<Known, Throwable> callback = (success, fail) -> {
- // TODO (expected): this should be invoked on this commandStore; also do not need to load txn unless in DEBUG mode
- commandStore.execute(contextFor(txnId), safeStore0 -> {
- if (progress() != Investigating)
- return;
-
- setProgress(Expected);
- // TODO (required): we might not be in the coordinating OR execution epochs if an accept round contacted us but recovery did not (quite hard to achieve)
- Invariants.checkState(fail != null || !blockedUntil.isSatisfiedBy(success.propagates()) || success.route != Full);
- }).begin(commandStore.agent());
- };
-
- node.withEpoch(blockedUntil.fetchEpoch(txnId, executeAt), (ignored, withEpochFailure) -> {
- if (withEpochFailure != null)
- {
- callback.accept(null, CoordinationFailed.wrap(withEpochFailure));
- return;
- }
- FetchData.fetch(blockedUntil.requires, node, txnId, fetchKeys, forLocalEpoch, executeAt, callback);
- });
- }
-
- @SuppressWarnings({"rawtypes", "unchecked"})
- private Unseekables<?> maxContact(Command command)
- {
- Route<?> route = Route.merge(command.route(), (Route)this.route);
- if (route != null) route = route.withHomeKey();
- return Unseekables.merge(route == null ? null : route.withHomeKey(), (Unseekables)participants);
- }
-
- @Override
- public String toString()
- {
- return progress().toString();
- }
- }
-
- class NonHomeState extends State.Monitoring
- {
- NonHomeState()
- {
- setProgress(Expected);
- }
-
- void setSafe()
- {
- if (progress() != Done)
- setProgress(Done);
- }
-
- @Override
- void run(SafeCommandStore safeStore, SafeCommand safeCommand)
- {
- Command command = safeCommand.current();
- // make sure a quorum of the home shard is aware of the transaction, so we can rely on it to ensure progress
- if (command.route() == null)
- throw new AssertionError(String.format("Attempted to inform but route is not known for command %s", command));
- AsyncChain<Void> inform = inform(node, txnId, command.route());
- inform.begin((success, fail) -> {
- commandStore.execute(empty(), ignore -> {
- if (progress() == Done)
- return;
-
- setProgress(fail != null ? Expected : Done);
- }).begin(commandStore.agent());
- });
- }
-
- @Override
- public String toString()
- {
- return progress() == Done ? "Safe" : "Unsafe";
- }
- }
-
- final TxnId txnId;
-
- CoordinateState coordinateState;
- NonHomeState nonHomeState;
- BlockingState blockingState;
-
- State(TxnId txnId)
- {
- this.txnId = txnId;
- }
-
- void recordBlocking(TxnId txnId, LocalExecution waitingFor, Route<?> route, Participants<?> participants)
- {
- Invariants.checkArgument(txnId.equals(this.txnId));
- if (blockingState == null)
- blockingState = new BlockingState();
- blockingState.recordBlocking(waitingFor, route, participants);
- }
-
- CoordinateState coordinate()
- {
- if (coordinateState == null)
- coordinateState = new CoordinateState();
- return coordinateState;
- }
-
- void ensureAtLeast(Command command, CoordinateStatus newStatus, Progress newProgress)
- {
- coordinate().ensureAtLeast(command, newStatus, newProgress);
- }
-
- void ensureAtLeast(CoordinateStatus newStatus, Progress newProgress)
- {
- coordinate().ensureAtLeast(newStatus, newProgress);
- }
-
- void touchNonHomeUnsafe()
- {
- if (nonHomeState == null)
- nonHomeState = new NonHomeState();
- }
-
- void setSafe()
- {
- if (nonHomeState == null)
- nonHomeState = new NonHomeState();
- nonHomeState.setSafe();
- }
-
- @Override
- public String toString()
- {
- return coordinateState != null ? coordinateState.toString()
- : nonHomeState != null
- ? nonHomeState.toString()
- : blockingState.toString();
- }
- }
-
- final CommandStore commandStore;
- final Map<TxnId, State> stateMap = new TreeMap<>();
- boolean isScheduled;
-
- Instance(CommandStore commandStore)
- {
- this.commandStore = commandStore;
- }
-
- State ensure(TxnId txnId)
- {
- Invariants.checkState(txnId.kind().isGloballyVisible());
- return stateMap.computeIfAbsent(txnId, ignored -> {
- node.agent().metricsEventsListener().onProgressLogSizeChange(txnId, 1);
- return new State(txnId);
- });
- }
-
- State ensure(TxnId txnId, State state)
- {
- return state != null ? state : ensure(txnId);
- }
-
- private void ensureSafeOrAtLeast(Command command, ProgressShard shard, CoordinateStatus newStatus, Progress newProgress)
- {
- Invariants.checkState(shard != Unsure);
-
- State state = null;
- assert newStatus.isAtMostReadyToExecute();
- if (newStatus.isAtLeastCommitted())
- state = recordStable(command.txnId());
-
- if (shard.isProgress())
- {
- state = ensure(command.txnId(), state);
-
- if (shard.isHome()) state.ensureAtLeast(command, newStatus, newProgress);
- else ensure(command.txnId()).setSafe();
- }
- }
-
- State recordStable(TxnId txnId)
- {
- State state = stateMap.get(txnId);
- if (state != null && state.blockingState != null)
- state.blockingState.record(SaveStatus.Stable.known);
- return state;
- }
-
- State recordApply(TxnId txnId)
- {
- State state = stateMap.get(txnId);
- if (state != null && state.blockingState != null)
- state.blockingState.record(SaveStatus.PreApplied.known);
- return state;
- }
-
- @Override
- public void unwitnessed(TxnId txnId, ProgressShard shard)
- {
- if (shard.isHome())
- ensure(txnId).ensureAtLeast(NotStable, Expected);
- }
-
- @Override
- public void preaccepted(Command command, ProgressShard shard)
- {
- Invariants.checkState(shard != Unsure);
-
- if (shard.isProgress())
- {
- State state = ensure(command.txnId());
- if (shard.isHome()) state.ensureAtLeast(command, NotStable, Expected);
- else state.touchNonHomeUnsafe();
- }
- }
-
- @Override
- public void accepted(Command command, ProgressShard shard)
- {
- ensureSafeOrAtLeast(command, shard, NotStable, Expected);
- }
-
- @Override
- public void precommitted(Command command)
- {
- State state = stateMap.get(command.txnId());
- if (state != null && state.blockingState != null)
- state.blockingState.record(SaveStatus.PreCommitted.known);
- }
-
- @Override
- public void stable(Command command, ProgressShard shard)
- {
- ensureSafeOrAtLeast(command, shard, CoordinateStatus.Stable, NoneExpected);
- }
-
- @Override
- public void readyToExecute(Command command)
- {
- State state = stateMap.get(command.txnId());
- if (state == null)
- return; // not progress shard, and nothing blocking
-
- Invariants.checkState(state.nonHomeState == null || state.nonHomeState.progress() == Done, "nonHomeState should have been set safe by call to committed");
- if (state.coordinateState != null)
- state.coordinateState.ensureAtLeast(command, ReadyToExecute, Expected);
- }
-
- @Override
- public void executed(Command command, ProgressShard shard)
- {
- recordApply(command.txnId());
- // this is the home shard's state ONLY, so we don't know it is fully durable locally
- ensureSafeOrAtLeast(command, shard, ReadyToExecute, Expected);
- }
-
- @Override
- public void clear(TxnId txnId)
- {
- State state = stateMap.remove(txnId);
- if (state == null)
- return;
-
- if (state.blockingState != null)
- state.blockingState.ensureDone();
- if (state.nonHomeState != null)
- state.nonHomeState.ensureDone();
- if (state.coordinateState != null)
- state.coordinateState.durableGlobal();
- }
-
- @Override
- public void durable(Command command)
- {
- State state = ensure(command.txnId());
- // if we participate in the transaction and its outcome hasn't been recorded locally then fetch it
- // TODO (expected): we should delete the in-memory state once we reach Done, and guard against backward progress with Command state
- // however, we need to be careful:
- // - we might participate in the execution epoch so need to be sure we have received a route covering both
- if (command.route() == null)
- return;
-
- Ranges coordinateRanges = commandStore.unsafeRangesForEpoch().allAt(command.txnId().epoch());
- if (!command.status().hasBeen(PreApplied) && command.route().participatesIn(coordinateRanges))
- state.recordBlocking(command.txnId(), WaitingToApply, command.route(), null);
- if (coordinateRanges.contains(command.route().homeKey()))
- state.coordinate().durableGlobal();
- }
-
- @Override
- public void waiting(SafeCommand blockedBy, LocalExecution blockedUntil, Route<?> blockedOnRoute, Participants<?> blockedOnParticipants)
- {
- if (!blockedBy.txnId().kind().isGloballyVisible())
- return;
-
- // ensure we have a record to work with later; otherwise may think has been truncated
- // TODO (expected): we shouldn't rely on this anymore
- blockedBy.initialise();
- if (blockedBy.current().has(blockedUntil.requires))
- return;
-
- // TODO (consider): consider triggering a preemption of existing coordinator (if any) in some circumstances;
- // today, an LWT can pre-empt more efficiently (i.e. instantly) a failed operation whereas Accord will
- // wait for some progress interval before taking over; there is probably some middle ground where we trigger
- // faster preemption once we're blocked on a transaction, while still offering some amount of time to complete.
- // TODO (desirable, efficiency): forward to local progress shard for processing (if known)
- // TODO (desirable, efficiency): if we are co-located with the home shard, don't need to do anything unless we're in a
- // later topology that wasn't covered by its coordination
- ensure(blockedBy.txnId()).recordBlocking(blockedBy.txnId(), blockedUntil, blockedOnRoute, blockedOnParticipants);
- }
-
- @Override
- public void waiting(TxnId blockedBy, LocalExecution blockedUntil, @Nullable Route<?> blockedOnRoute, @Nullable Participants<?> blockedOnParticipants)
- {
- if (!blockedBy.kind().isGloballyVisible())
- return;
-
- ensure(blockedBy).recordBlocking(blockedBy, blockedUntil, blockedOnRoute, blockedOnParticipants);
- }
-
- @Override
- public void addFirst(State.Monitoring add)
- {
- super.addFirst(add);
- ensureScheduled();
- }
-
- @Override
- public void addLast(State.Monitoring add)
- {
- throw new UnsupportedOperationException();
- }
-
- void ensureScheduled()
- {
- if (isScheduled)
- return;
-
- isScheduled = true;
- Duration delay = node.localConfig().getProgressLogScheduleDelay();
- node.scheduler().once(() -> commandStore.execute(empty(), ignore -> run()).begin(commandStore.agent()), delay.toNanos(), TimeUnit.NANOSECONDS);
- }
-
- @Override
- public void run()
- {
- isScheduled = false;
- try
- {
- if (PAUSE_FOR_TEST)
- return;
-
- for (State.Monitoring run : this)
- {
- if (run.shouldRun())
- {
- commandStore.execute(contextFor(run.txnId()), safeStore -> {
- if (run.shouldRun()) // could have been completed by a callback
- {
- SafeCommand safeCommand = safeStore.ifInitialised(run.txnId());
- if (safeCommand != null && run.shouldRun()) // could be truncated on access
- run.run(safeStore, safeCommand);
- }
- }).begin(commandStore.agent());
- }
- }
- }
- catch (Throwable t)
- {
- node.agent().onUncaughtException(t);
- }
- finally
- {
- if (!isEmpty())
- ensureScheduled();
- }
- }
- }
-
- @Override
- public Instance create(CommandStore commandStore)
- {
- Instance instance = new Instance(commandStore);
- instances.add(instance);
- return instance;
- }
-}
diff --git a/accord-core/src/main/java/accord/impl/progresslog/BaseTxnState.java b/accord-core/src/main/java/accord/impl/progresslog/BaseTxnState.java
new file mode 100644
index 00000000..0b3e1e4f
--- /dev/null
+++ b/accord-core/src/main/java/accord/impl/progresslog/BaseTxnState.java
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.impl.progresslog;
+
+import javax.annotation.Nullable;
+
+import accord.api.ProgressLog.BlockedUntil;
+import accord.local.SafeCommandStore;
+import accord.primitives.TxnId;
+import accord.utils.Invariants;
+import accord.utils.LogGroupTimers;
+
+import static accord.impl.progresslog.TxnStateKind.Home;
+import static accord.impl.progresslog.TxnStateKind.Waiting;
+
+/**
+ * We have a slightly odd class hierarchy here to cleanly group the logic, though TxnState can logically
+ * be read as a single class.
+ * <p>
+ * We have:
+ * - BaseTxnState defines the state that each of the child classes operate on and provides shared
+ * methods for updating it, particularly with relation to tracking timer scheduling
+ * - WaitingState defines the methods for updating the WaitingState state machine (i.e. for local commands that are waiting on a dependency's outcome)
+ * - HomeState defines the methods for updating the HomeState state machine (i.e. for coordinators ensuring a transaction completes)
+ * <p>
+ * TODO (required): unit test all bit fiddling methods
+ */
+abstract class BaseTxnState extends LogGroupTimers.Timer implements Comparable<BaseTxnState>
+{
+ private static final int PENDING_TIMER_SHIFT = 53;
+ private static final int PENDING_TIMER_BITS = 9;
+ private static final int PENDING_TIMER_LOW_BITS = PackedLogLinearInteger.validateLowBits(4, PENDING_TIMER_BITS);
+ private static final long PENDING_TIMER_MASK = (1L << PENDING_TIMER_BITS) - 1;
+ private static final long PENDING_TIMER_CLEAR_MASK = ~(PENDING_TIMER_MASK << PENDING_TIMER_SHIFT);
+ private static final int SCHEDULED_TIMER_SHIFT = 62;
+ private static final int CONTACT_ALL_SHIFT = 63;
+
+ public final TxnId txnId;
+
+ /**
+ * bits [0..40) encode WaitingState
+ * 2 bits for Progress
+ * 3 bits for BlockedUntil target
+ * 3 bits for BlockedUntil that home shard can satisfy
+ * 32 bits for remote progress key counter [note: if we need to in future we can safely and easily reclaim bits here]
+ * 3 bits for retry counter
+ * <p>
+ * bits [40..45) encode HomeState
+ * 2 bits for Progress
+ * 3 bits for CoordinatePhase
+ * 3 bits for retry counter
+ * <p>
+ * bits [53..62) for pending timer delay
+ * bit 62 for which kind of timer is scheduled
+ * bit 63 for whether we should contact all candidate replicas (rather than just our preferred group)
+ */
+ long encodedState;
+
+ BaseTxnState(TxnId txnId)
+ {
+ this.txnId = txnId;
+ }
+
+ @Override
+ public int compareTo(BaseTxnState that)
+ {
+ return this.txnId.compareTo(that.txnId);
+ }
+
+ boolean contactEveryone()
+ {
+ return ((encodedState >>> CONTACT_ALL_SHIFT) & 1L) == 1L;
+ }
+
+ void setContactEveryone(boolean newContactEveryone)
+ {
+ encodedState &= encodedState & ~(1L << CONTACT_ALL_SHIFT);
+ encodedState |= (newContactEveryone ? 1L : 0) << CONTACT_ALL_SHIFT;
+ }
+
+ /**
+ * The state that has a timer actively scheduled, if any
+ */
+ TxnStateKind scheduledTimer()
+ {
+ if (!isScheduled())
+ return null;
+ return wasScheduledTimer();
+ }
+
+ /**
+ * The state that last had a timer actively scheduled
+ */
+ TxnStateKind wasScheduledTimer()
+ {
+ return ((encodedState >>> SCHEDULED_TIMER_SHIFT) & 1) == 0 ? Waiting : Home;
+ }
+
+ /**
+ * Set the state that has a timer actively scheduled
+ */
+ void setScheduledTimer(TxnStateKind kind)
+ {
+ encodedState &= ~(1L << SCHEDULED_TIMER_SHIFT);
+ encodedState |= (long) kind.ordinal() << SCHEDULED_TIMER_SHIFT;
+ Invariants.checkState(wasScheduledTimer() == kind);
+ }
+
+ /**
+ * The state that has a timer pending, if any
+ */
+ TxnStateKind pendingTimer()
+ {
+ int pendingDelay = pendingTimerDelay();
+ if (pendingDelay == 0)
+ return null;
+ return ((encodedState >>> SCHEDULED_TIMER_SHIFT) & 1) == 0 ? Home : Waiting;
+ }
+
+ void clearPendingTimerDelay()
+ {
+ encodedState &= PENDING_TIMER_CLEAR_MASK;
+ }
+
+ void setPendingTimerDelay(int deltaMicros)
+ {
+ clearPendingTimerDelay();
+ encodedState |= (long)PackedLogLinearInteger.encode(deltaMicros, PENDING_TIMER_LOW_BITS, PENDING_TIMER_BITS) << PENDING_TIMER_SHIFT;
+ }
+
+ int pendingTimerDelay()
+ {
+ return PackedLogLinearInteger.decode((int)((encodedState >>> PENDING_TIMER_SHIFT) & PENDING_TIMER_MASK), PENDING_TIMER_LOW_BITS);
+ }
+
+ long pendingTimerDeadline(long scheduledDeadline)
+ {
+ long pendingTimerDelay = pendingTimerDelay();
+ return pendingTimerDelay == 0 ? 0 : scheduledDeadline + pendingTimerDelay;
+ }
+
+ long pendingTimerDeadline(DefaultProgressLog owner)
+ {
+ long pendingTimerDelay = pendingTimerDelay();
+ return pendingTimerDelay == 0 ? 0 : deadline() + pendingTimerDelay;
+ }
+
+ boolean isScheduled()
+ {
+ return isInHeap();
+ }
+
+ abstract void updateScheduling(SafeCommandStore safeStore, DefaultProgressLog instance, TxnStateKind updated, @Nullable BlockedUntil awaiting, Progress newProgress);
+
+ abstract void maybeRemove(DefaultProgressLog instance);
+}
diff --git a/accord-core/src/main/java/accord/messages/Request.java b/accord-core/src/main/java/accord/impl/progresslog/Callback.java
similarity index 72%
copy from accord-core/src/main/java/accord/messages/Request.java
copy to accord-core/src/main/java/accord/impl/progresslog/Callback.java
index 0c923716..fc530432 100644
--- a/accord-core/src/main/java/accord/messages/Request.java
+++ b/accord-core/src/main/java/accord/impl/progresslog/Callback.java
@@ -16,15 +16,13 @@
* limitations under the License.
*/
-package accord.messages;
+package accord.impl.progresslog;
-import accord.local.Node;
-import accord.local.Node.Id;
+import accord.local.SafeCommand;
+import accord.local.SafeCommandStore;
+import accord.primitives.TxnId;
-public interface Request extends Message
+interface Callback<P, V>
{
- default long waitForEpoch() { return 0; }
- void preProcess(Node on, Id from, ReplyContext replyContext);
- void process(Node on, Id from, ReplyContext replyContext);
-
+ void callback(SafeCommandStore safeStore, SafeCommand safeCommand, DefaultProgressLog instance, TxnId txnId, P param, V success, Throwable fail);
}
diff --git a/accord-core/src/main/java/accord/impl/progresslog/CallbackInvoker.java b/accord-core/src/main/java/accord/impl/progresslog/CallbackInvoker.java
new file mode 100644
index 00000000..afd6aea1
--- /dev/null
+++ b/accord-core/src/main/java/accord/impl/progresslog/CallbackInvoker.java
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.impl.progresslog;
+
+import java.util.function.BiConsumer;
+
+import accord.local.SafeCommand;
+import accord.primitives.TxnId;
+import accord.utils.Invariants;
+
+import static accord.impl.progresslog.TxnStateKind.Home;
+import static accord.impl.progresslog.TxnStateKind.Waiting;
+import static accord.local.PreLoadContext.contextFor;
+
+class CallbackInvoker<P, V> implements BiConsumer<V, Throwable>
+{
+ static <P, V> BiConsumer<V, Throwable> invokeWaitingCallback(DefaultProgressLog instance, TxnId txnId, P param, Callback<P, V> callback)
+ {
+ return invokeCallback(Waiting, instance, txnId, param, callback);
+ }
+
+ static <P, V> BiConsumer<V, Throwable> invokeHomeCallback(DefaultProgressLog owner, TxnId txnId, P param, Callback<P, V> callback)
+ {
+ return invokeCallback(Home, owner, txnId, param, callback);
+ }
+
+ static <P, V> BiConsumer<V, Throwable> invokeCallback(TxnStateKind kind, DefaultProgressLog owner, TxnId txnId, P param, Callback<P, V> callback)
+ {
+ CallbackInvoker<P, V> invoker = new CallbackInvoker<>(owner, kind, owner.nextInvokerId(), txnId, param, callback);
+ owner.registerActive(kind, txnId, invoker);
+ return invoker;
+ }
+
+ final DefaultProgressLog owner;
+ final boolean isHome;
+ final long id;
+ final TxnId txnId;
+ final P param;
+ final Callback<P, V> callback;
+
+ CallbackInvoker(DefaultProgressLog owner, TxnStateKind kind, long id, TxnId txnId, P param, Callback<P, V> callback)
+ {
+ this.owner = owner;
+ this.isHome = kind == Home;
+ this.id = id;
+ this.txnId = txnId;
+ this.param = param;
+ this.callback = callback;
+ }
+
+ private TxnStateKind kind()
+ {
+ return isHome ? Home : Waiting;
+ }
+
+ @Override
+ public void accept(V success, Throwable fail)
+ {
+ owner.commandStore.execute(contextFor(txnId), safeStore -> {
+
+ // we load safeCommand first so that if it clears the progress log we abandon the callback
+ SafeCommand safeCommand = safeStore.ifInitialised(txnId);
+ if (!owner.deregisterActive(kind(), this))
+ return;
+
+ Invariants.checkState(safeCommand != null);
+ callback.callback(safeStore, safeCommand, owner, txnId, param, success, fail);
+
+ }).begin(owner.commandStore.agent());
+ }
+
+ @Override
+ public boolean equals(Object obj)
+ {
+ if (obj == null) return false;
+ if (obj.getClass() == TxnId.class) return txnId.equals(obj);
+ if (obj.getClass() != getClass()) return false;
+ CallbackInvoker<?, ?> that = (CallbackInvoker<?, ?>) obj;
+ return id == that.id && callback == that.callback;
+ }
+
+ @Override
+ public int hashCode()
+ {
+ return txnId.hashCode();
+ }
+}
+
diff --git a/accord-core/src/main/java/accord/impl/progresslog/CoordinatePhase.java b/accord-core/src/main/java/accord/impl/progresslog/CoordinatePhase.java
new file mode 100644
index 00000000..3e63d31b
--- /dev/null
+++ b/accord-core/src/main/java/accord/impl/progresslog/CoordinatePhase.java
@@ -0,0 +1,68 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.impl.progresslog;
+
+// the phase of the distributed state machine
+enum CoordinatePhase
+{
+ /**
+ * This replica is not known to be a home shard of the transaction
+ */
+ NotInitialised,
+
+ /**
+ * not durably decided; if eligible to take over coordination, should see if coordination is stalled and if so
+ * take over recovery to ensure an execution decision is reached
+ */
+ Undecided,
+
+ /**
+ * durably decided, but replicas may not be ready to execute; should wait until we can expect to successfully
+ * execute the transaction before attempting recovery
+ * TODO (expected): this state is not effectively used today, we only wait for the home shard to be ready to execute,
+ * whereas we can perform a distributed wait using the WaitingState to ensure we can make progress
+ */
+ AwaitReadyToExecute,
+
+ /**
+ * some replicas of all shards ready to execute; we can expect that the transaction can be successfully executed,
+ * and we should now try to ensure this happens when it is our turn to do so
+ */
+ ReadyToExecute,
+
+ /**
+ * The transaction has been durably executed at a majority of replicas of all shards (but not necessarily ourselves)
+ */
+ Done
+ ;
+
+ private static final CoordinatePhase[] lookup = values();
+
+ boolean isAtMostReadyToExecute()
+ {
+ return compareTo(CoordinatePhase.ReadyToExecute) <= 0;
+ }
+
+ public static CoordinatePhase forOrdinal(int ordinal)
+ {
+ if (ordinal < 0 || ordinal > lookup.length)
+ throw new IndexOutOfBoundsException(ordinal);
+ return lookup[ordinal];
+ }
+}
diff --git a/accord-core/src/main/java/accord/impl/progresslog/DefaultProgressLog.java b/accord-core/src/main/java/accord/impl/progresslog/DefaultProgressLog.java
new file mode 100644
index 00000000..752a8684
--- /dev/null
+++ b/accord-core/src/main/java/accord/impl/progresslog/DefaultProgressLog.java
@@ -0,0 +1,536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.impl.progresslog;
+
+import java.util.Arrays;
+import java.util.Map;
+import java.util.concurrent.TimeUnit;
+import java.util.function.Consumer;
+import java.util.function.Function;
+import javax.annotation.Nullable;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import accord.api.ProgressLog;
+import accord.api.RoutingKey;
+import accord.local.Command;
+import accord.local.CommandStore;
+import accord.local.CommonAttributes;
+import accord.local.Node;
+import accord.local.PreLoadContext;
+import accord.local.SafeCommand;
+import accord.local.SafeCommandStore;
+import accord.local.SaveStatus;
+import accord.primitives.Participants;
+import accord.primitives.ProgressToken;
+import accord.primitives.Ranges;
+import accord.primitives.Route;
+import accord.primitives.TxnId;
+import accord.utils.Invariants;
+import accord.utils.LogGroupTimers;
+import accord.utils.btree.BTree;
+import accord.utils.btree.BTreeRemoval;
+import org.agrona.collections.Object2ObjectHashMap;
+import org.agrona.collections.ObjectHashSet;
+
+import static accord.api.ProgressLog.BlockedUntil.CanApply;
+import static accord.impl.progresslog.CoordinatePhase.AwaitReadyToExecute;
+import static accord.impl.progresslog.CoordinatePhase.ReadyToExecute;
+import static accord.impl.progresslog.CoordinatePhase.Undecided;
+import static accord.impl.progresslog.Progress.Awaiting;
+import static accord.impl.progresslog.Progress.NoneExpected;
+import static accord.impl.progresslog.Progress.Querying;
+import static accord.impl.progresslog.Progress.Queued;
+import static accord.impl.progresslog.TxnStateKind.Home;
+import static accord.impl.progresslog.TxnStateKind.Waiting;
+import static accord.local.PreLoadContext.contextFor;
+import static accord.local.Status.PreApplied;
+import static accord.utils.ArrayBuffers.cachedAny;
+import static accord.utils.btree.UpdateFunction.noOpReplace;
+import static java.util.concurrent.TimeUnit.MICROSECONDS;
+import static java.util.concurrent.TimeUnit.MILLISECONDS;
+import static java.util.concurrent.TimeUnit.SECONDS;
+
+public class DefaultProgressLog implements ProgressLog, Runnable
+{
+ private static final Logger logger = LoggerFactory.getLogger(DefaultProgressLog.class);
+
+ final Node node;
+ final CommandStore commandStore;
+
+ private Object[] stateMap = BTree.empty();
+ private Object[] progressTokenMap = BTree.empty();
+
+ private final LogGroupTimers<TxnState> timers = new LogGroupTimers<>(MICROSECONDS);
+ private final LogGroupTimers<TxnState>.Scheduling timerScheduling;
+
+ /**
+ * A collection of active callbacks (waiting remote replies) or submitted run invocations
+ * (perhaps waiting load from disk, or for the CommandStore thread to be available).
+ *
+ * These callbacks are required to have hashCode() == txnId.hashCode() and equals(txnId) == true,
+ * so that we can manage overriding callbacks on the relevant TxnState.
+ */
+ private final ObjectHashSet<Object> activeWaiting = new ObjectHashSet<>();
+ private final ObjectHashSet<Object> activeHome = new ObjectHashSet<>();
+
+ private final Map<TxnId, StackTraceElement[]> deleted = Invariants.debug() ? new Object2ObjectHashMap<>() : null;
+
+ private static final Object[] EMPTY_RUN_BUFFER = new Object[0];
+
+ // The tasks whose timers have elapsed and are going to be run
+ // The queue is drained here first before processing tasks so that tasks can modify the queue
+ private Object[] runBuffer;
+ private int runBufferCount;
+
+ private long nextInvokerId;
+
+ DefaultProgressLog(Node node, CommandStore commandStore)
+ {
+ this.node = node;
+ this.commandStore = commandStore;
+ Function<TxnState, Runnable> taskFactory = next -> {
+ PreLoadContext context = next == null ? PreLoadContext.empty() : PreLoadContext.contextFor(next.txnId);
+ return () -> commandStore.execute(context, safeStore -> run())
+ .begin(commandStore.agent());
+ };
+ this.timerScheduling = timers.new Scheduling(node.scheduler(), taskFactory,
+ MILLISECONDS.toMicros(10), 100, SECONDS.toMicros(1L));
+ }
+
+ Node node()
+ {
+ return node;
+ }
+
+ void update(long deadline, TxnState timer)
+ {
+ update(node.elapsed(MICROSECONDS), deadline, timer);
+ }
+
+ void update(long nowMicros, long deadline, TxnState timer)
+ {
+ timers.update(deadline, timer);
+ maybeReschedule(nowMicros, deadline);
+ }
+
+ void add(long nowMicros, long deadline, TxnState timer)
+ {
+ timers.add(deadline, timer);
+ maybeReschedule(nowMicros, deadline);
+ }
+
+ @VisibleForTesting
+ public @Nullable TxnState get(TxnId txnId)
+ {
+ Invariants.checkState(txnId.kind().isGloballyVisible());
+ return BTree.<TxnId, TxnState>find(stateMap, (id, state) -> id.compareTo(state.txnId), txnId);
+ }
+
+ TxnState ensure(TxnId txnId)
+ {
+ Invariants.checkState(txnId.kind().isGloballyVisible());
+ TxnState result = BTree.<TxnId, TxnState>find(stateMap, (id, state) -> id.compareTo(state.txnId), txnId);
+ if (result == null)
+ {
+ Invariants.checkState(deleted == null || !deleted.containsKey(txnId));
+ node.agent().metricsEventsListener().onProgressLogSizeChange(txnId, 1);
+ stateMap = BTree.update(stateMap, BTree.singleton(result = new TxnState(txnId)), TxnState::compareTo);
+ }
+ return result;
+ }
+
+ private TxnState insert(TxnId txnId)
+ {
+ Invariants.checkState(deleted == null || !deleted.containsKey(txnId));
+ node.agent().metricsEventsListener().onProgressLogSizeChange(txnId, 1);
+ TxnState result = new TxnState(txnId);
+ stateMap = BTree.update(stateMap, BTree.singleton(result), TxnState::compareTo);
+ return result;
+ }
+
+ ProgressToken savedProgressToken(TxnId txnId)
+ {
+ ProgressToken saved = BTree.<TxnId, SavedProgressToken>find(progressTokenMap, (id, e) -> id.compareTo(e.txnId), txnId);
+ if (saved == null)
+ return ProgressToken.NONE;
+
+ progressTokenMap = BTreeRemoval.<TxnId, SavedProgressToken>remove(progressTokenMap, (id, e) -> id.compareTo(e.txnId), txnId);
+ return saved;
+ }
+
+ void saveProgressToken(TxnId txnId, ProgressToken token)
+ {
+ SavedProgressToken save = new SavedProgressToken(txnId, token);
+ // we could save memory by setting ballot to ZERO when we have the same ballot in command (and can restore isAccepted)
+ // but this isn't likely to offer dramatic savings very often
+ progressTokenMap = BTree.update(progressTokenMap, BTree.singleton(save), SavedProgressToken::compare, noOpReplace());
+ }
+
+ void clearProgressToken(TxnId txnId)
+ {
+ progressTokenMap = BTreeRemoval.<TxnId, SavedProgressToken>remove(progressTokenMap, (id, e) -> id.compareTo(e.txnId), txnId);
+ }
+
+ @Override
+ public void update(SafeCommandStore safeStore, TxnId txnId, Command before, Command after)
+ {
+ if (!txnId.kind().isGloballyVisible())
+ return;
+
+ TxnState state = null;
+ if (before.route() == null && after.route() != null)
+ {
+ RoutingKey homeKey = after.homeKey();
+ Ranges coordinateRanges = safeStore.ranges().allAt(txnId.epoch());
+ boolean isHome = coordinateRanges.contains(homeKey);
+ state = get(txnId);
+ if (isHome)
+ {
+ if (state == null)
+ state = insert(txnId);
+
+ if (after.durability().isDurableOrInvalidated())
+ {
+ state.setHomeDoneAnyMaybeRemove(this);
+ if (!after.hasBeen(PreApplied))
+ state.waiting().setBlockedUntil(safeStore, this, CanApply);
+ }
+ else
+ state.set(safeStore, this, Undecided, Queued);
+ }
+ else if (state != null)
+ {
+ // not home shard
+ state.setHomeDone(this);
+ }
+ }
+ else if (after.durability().isDurableOrInvalidated() && !before.durability().isDurableOrInvalidated())
+ {
+ state = get(txnId);
+ if (state != null)
+ state.setHomeDoneAnyMaybeRemove(this);
+
+ if (!after.hasBeen(PreApplied))
+ {
+ // this command should be ready to apply locally, so fetch it
+ if (state == null)
+ state = insert(txnId);
+ state.waiting().setBlockedUntil(safeStore, this, CanApply);
+ }
+ }
+
+ SaveStatus beforeSaveStatus = before.saveStatus();
+ SaveStatus afterSaveStatus = after.saveStatus();
+ if (beforeSaveStatus == afterSaveStatus)
+ return;
+
+ if (state == null)
+ state = get(txnId);
+
+ if (state == null)
+ return;
+
+ state.waiting().record(this, afterSaveStatus);
+ if (state.isHomeInitialised())
+ {
+ switch (afterSaveStatus)
+ {
+ case Stable:
+ state.home().atLeast(safeStore, this, Undecided, NoneExpected);
+ break;
+ case ReadyToExecute:
+ state.home().atLeast(safeStore, this, AwaitReadyToExecute, Queued);
+ break;
+ case PreApplied:
+ state.home().atLeast(safeStore, this, ReadyToExecute, Queued);
+ break;
+ }
+ }
+ }
+
+ @Override
+ public void clear(TxnId txnId)
+ {
+ if (!txnId.kind().isGloballyVisible())
+ return;
+
+ TxnState state = get(txnId);
+ if (state != null)
+ clear(state);
+ }
+
+ private void clear(TxnState state)
+ {
+ state.setHomeDone(this);
+ state.setWaitingDone(this);
+ Invariants.checkState(!state.isScheduled());
+ remove(state.txnId);
+ }
+
+ void remove(TxnId txnId)
+ {
+ Object[] newStateMap = BTreeRemoval.<TxnId, TxnState>remove(stateMap, (id, s) -> id.compareTo(s.txnId), txnId);
+ if (stateMap != newStateMap)
+ node.agent().metricsEventsListener().onProgressLogSizeChange(txnId, -1);
+ stateMap = newStateMap;
+ if (deleted != null)
+ deleted.put(txnId, Thread.currentThread().getStackTrace());
+ }
+
+ @Override
+ public void remoteCallback(SafeCommandStore safeStore, SafeCommand safeCommand, SaveStatus remoteStatus, int callbackId, Node.Id from)
+ {
+ TxnState state = get(safeCommand.txnId());
+ if (state != null)
+ state.asynchronousAwaitCallback(this, safeStore, remoteStatus, from, callbackId);
+ }
+
+ @Override
+ public void waiting(BlockedUntil blockedUntil, SafeCommandStore safeStore, SafeCommand blockedBy, Route<?> blockedOnRoute, Participants<?> blockedOnParticipants)
+ {
+ if (!blockedBy.txnId().kind().isGloballyVisible())
+ return;
+
+ // ensure we have a record to work with later; otherwise may think has been truncated
+ // TODO (expected): we shouldn't rely on this anymore
+ blockedBy.initialise();
+ Invariants.checkState(blockedBy.current().saveStatus().compareTo(blockedUntil.minSaveStatus) < 0);
+
+ // first save the route/participant info into the Command if it isn't already there
+ CommonAttributes update = blockedBy.current();
+ Route<?> currentRoute = update.route();
+ if (currentRoute != null)
+ {
+ Route<?> updatedRoute = currentRoute;
+ if (blockedOnRoute != null) updatedRoute = currentRoute.with((Route) blockedOnRoute);
+ else if (blockedOnParticipants != null)
+ updatedRoute = currentRoute.with((Participants) blockedOnParticipants);
+ if (updatedRoute != currentRoute)
+ update = update.mutable().route(updatedRoute);
+ }
+ else if (blockedOnRoute != null) update = update.mutable().route(blockedOnRoute);
+ else if (blockedOnParticipants != null) update = update.mutable().participants(blockedOnParticipants);
+ if (update != blockedBy.current())
+ blockedBy.updateAttributes(safeStore, update);
+
+ // TODO (consider): consider triggering a preemption of existing coordinator (if any) in some circumstances;
+ // today, an LWT can pre-empt more efficiently (i.e. instantly) a failed operation whereas Accord will
+ // wait for some progress interval before taking over; there is probably some middle ground where we trigger
+ // faster preemption once we're blocked on a transaction, while still offering some amount of time to complete.
+ // TODO (desirable, efficiency): forward to local progress shard for processing (if known)
+ // TODO (desirable, efficiency): if we are co-located with the home shard, don't need to do anything unless we're in a
+ // later topology that wasn't covered by its coordination
+ TxnState state = ensure(blockedBy.txnId());
+ state.waiting().setBlockedUntil(safeStore, this, blockedUntil);
+ }
+
+ void ensureScheduled(long nowMicros)
+ {
+ timerScheduling.ensureScheduled(nowMicros);
+ }
+
+ void maybeReschedule(long nowMicros, long newDeadlineMicros)
+ {
+ timerScheduling.maybeReschedule(nowMicros, newDeadlineMicros);
+ }
+
+ @Override
+ public void run()
+ {
+ long nowMicros = node.elapsed(TimeUnit.MICROSECONDS);
+ try
+ {
+ if (DefaultProgressLogs.pauseForTest(this))
+ {
+ logger.info("Skipping progress log because it is paused for test");
+ return;
+ }
+
+ // drain to a buffer to avoid reentrancy
+ runBufferCount = 0;
+ runBuffer = EMPTY_RUN_BUFFER;
+ timers.advance(nowMicros, this, DefaultProgressLog::addToRunBuffer);
+ processRunBuffer(nowMicros);
+ cachedAny().forceDiscard(runBuffer, runBufferCount);
+ }
+ catch (Throwable t)
+ {
+ node.agent().onUncaughtException(t);
+ }
+ finally
+ {
+ if (!timers.isEmpty())
+ ensureScheduled(nowMicros);
+ }
+ }
+
+ private void addToRunBuffer(TxnState add)
+ {
+ if (runBufferCount == runBuffer.length)
+ runBuffer = cachedAny().resize(runBuffer, runBufferCount, Math.max(8, runBuffer.length * 2));
+ runBuffer[runBufferCount++] = add;
+ }
+
+ // TODO (expected): invoke immediately if the command is already loaded
+ private void processRunBuffer(long nowMicros)
+ {
+ for (int i = 0; i < runBufferCount; ++i)
+ {
+ TxnState run = (TxnState) runBuffer[i];
+ Invariants.checkState(!run.isScheduled());
+ TxnStateKind runKind = run.wasScheduledTimer();
+ validatePreRunState(run, runKind);
+
+ long pendingTimerDeadline = run.pendingTimerDeadline(DefaultProgressLog.this);
+ if (pendingTimerDeadline > 0)
+ {
+ run.clearPendingTimerDelay();
+ if (pendingTimerDeadline <= nowMicros)
+ {
+ invoke(run, runKind.other());
+ }
+ else
+ {
+ run.setScheduledTimer(runKind.other());
+ timers.add(pendingTimerDeadline, run);
+ }
+ }
+
+ invoke(run, runKind);
+ }
+
+ Arrays.fill(runBuffer, 0, runBufferCount, null);
+ runBufferCount = 0;
+ }
+
+ private void validatePreRunState(TxnState run, TxnStateKind kind)
+ {
+ Progress progress = kind == Waiting ? run.waiting().waitingProgress() : run.home().homeProgress();
+ Invariants.checkState(progress != NoneExpected && progress != Querying);
+ }
+
+ void invoke(TxnState run, TxnStateKind runKind)
+ {
+ RunInvoker invoker = new RunInvoker(nextInvokerId(), run, runKind);
+ registerActive(runKind, run.txnId, invoker);
+ node.withEpoch(run.txnId.epoch(), commandStore.agent(), () -> {
+ commandStore.execute(contextFor(run.txnId), invoker)
+ .begin(commandStore.agent());
+ });
+ }
+
+ class RunInvoker implements Consumer<SafeCommandStore>
+ {
+ final long id;
+ final TxnState run;
+ final TxnStateKind runKind;
+
+ RunInvoker(long id, TxnState run, TxnStateKind runKind)
+ {
+ this.id = id;
+ this.run = run;
+ this.runKind = runKind;
+ }
+
+ @Override
+ public void accept(SafeCommandStore safeStore)
+ {
+ // we have to read safeCommand first as it may become truncated on load, which may clear the progress log and invalidate us
+ SafeCommand safeCommand = safeStore.ifInitialised(run.txnId);
+ Invariants.checkState(safeCommand != null);
+
+ if (!deregisterActive(runKind, this))
+ return; // we've been cancelled
+
+ Invariants.checkState(get(run.txnId) == run);
+ Invariants.checkState(run.scheduledTimer() != runKind, "We are actively executing %s, but we are also scheduled to run this same TxnState later. This should not happen.", runKind);
+ Invariants.checkState(run.pendingTimer() != runKind, "We are actively executing %s, but we also have a pending scheduled task to run this same TxnState later. This should not happen.", runKind);
+
+ validatePreRunState(run, runKind);
+ if (runKind == Home)
+ {
+ boolean isRetry = run.homeProgress() == Awaiting;
+ if (isRetry) run.incrementHomeRetryCounter();
+ run.home().runHome(DefaultProgressLog.this, safeStore, safeCommand);
+ }
+ else
+ {
+ boolean isRetry = run.waitingProgress() == Awaiting;
+ if (isRetry) run.incrementWaitingRetryCounter();
+ run.runWaiting(safeStore, safeCommand, DefaultProgressLog.this);
+ }
+ }
+
+ @Override
+ public boolean equals(Object obj)
+ {
+ if (obj == null) return false;
+ if (obj.getClass() == TxnId.class) return run.txnId.equals(obj);
+ if (obj.getClass() != getClass()) return false;
+ RunInvoker that = (RunInvoker) obj;
+ return id == that.id && run.txnId.equals(that.run.txnId) && runKind.equals(that.runKind);
+ }
+
+ @Override
+ public int hashCode()
+ {
+ return run.txnId.hashCode();
+ }
+ }
+
+ long nextInvokerId()
+ {
+ return nextInvokerId++;
+ }
+
+ ObjectHashSet<Object> active(TxnStateKind kind)
+ {
+ return kind == Waiting ? activeWaiting : activeHome;
+ }
+
+ void registerActive(TxnStateKind kind, TxnId txnId, Object object)
+ {
+ ObjectHashSet<Object> active = active(kind);
+ Invariants.checkState(!active.contains(txnId));
+ active.add(object);
+ }
+
+ boolean hasActive(TxnStateKind kind, TxnId txnId)
+ {
+ return active(kind).contains(txnId);
+ }
+
+ boolean deregisterActive(TxnStateKind kind, Object object)
+ {
+ return active(kind).remove(object);
+ }
+
+ void clearActive(TxnStateKind kind, TxnId txnId)
+ {
+ active(kind).remove(txnId);
+ }
+
+ void unschedule(TxnState state)
+ {
+ timers.remove(state);
+ }
+}
diff --git a/accord-core/src/main/java/accord/impl/progresslog/DefaultProgressLogs.java b/accord-core/src/main/java/accord/impl/progresslog/DefaultProgressLogs.java
new file mode 100644
index 00000000..7f4fceaa
--- /dev/null
+++ b/accord-core/src/main/java/accord/impl/progresslog/DefaultProgressLogs.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.impl.progresslog;
+
+import java.util.List;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.concurrent.CopyOnWriteArrayList;
+
+import accord.api.ProgressLog;
+import accord.local.CommandStore;
+import accord.local.Node;
+
+// TODO (desired, consider): consider propagating invalidations in the same way as we do applied
+// TODO (expected): report transactions not making progress
+// TODO (required): evict to disk
+public class DefaultProgressLogs implements ProgressLog.Factory
+{
+ static ConcurrentHashMap<DefaultProgressLog, Runnable> PAUSE_FOR_TEST;
+ public static synchronized void unsafePauseForTesting(boolean pause)
+ {
+ if (pause)
+ {
+ PAUSE_FOR_TEST = new ConcurrentHashMap<>();
+ }
+ else
+ {
+ ConcurrentHashMap<DefaultProgressLog, Runnable> reschedule = PAUSE_FOR_TEST;
+ PAUSE_FOR_TEST = null;
+ if (reschedule != null)
+ reschedule.values().forEach(Runnable::run);
+ }
+ }
+
+ static boolean pauseForTest(DefaultProgressLog progressLog)
+ {
+ ConcurrentHashMap<DefaultProgressLog, Runnable> paused = PAUSE_FOR_TEST;
+ if (paused == null)
+ return false;
+
+ if (!paused.containsKey(progressLog))
+ paused.putIfAbsent(progressLog, () -> progressLog.commandStore.execute(progressLog));
+ return true;
+ }
+
+ final Node node;
+ final List<DefaultProgressLog> instances = new CopyOnWriteArrayList<>();
+
+ public DefaultProgressLogs(Node node)
+ {
+ this.node = node;
+ }
+
+ @Override
+ public DefaultProgressLog create(CommandStore commandStore)
+ {
+ DefaultProgressLog instance = new DefaultProgressLog(node, commandStore);
+ instances.add(instance);
+ return instance;
+ }
+}
diff --git a/accord-core/src/main/java/accord/impl/progresslog/HomeState.java b/accord-core/src/main/java/accord/impl/progresslog/HomeState.java
new file mode 100644
index 00000000..d9268b61
--- /dev/null
+++ b/accord-core/src/main/java/accord/impl/progresslog/HomeState.java
@@ -0,0 +1,219 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.impl.progresslog;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import accord.coordinate.MaybeRecover;
+import accord.coordinate.Outcome;
+import accord.local.Command;
+import accord.local.SafeCommand;
+import accord.local.SafeCommandStore;
+import accord.primitives.ProgressToken;
+import accord.primitives.TxnId;
+import accord.utils.Invariants;
+
+import static accord.api.ProgressLog.BlockedUntil.CanCoordinateExecution;
+import static accord.impl.progresslog.CallbackInvoker.invokeHomeCallback;
+import static accord.impl.progresslog.CoordinatePhase.Done;
+import static accord.impl.progresslog.CoordinatePhase.ReadyToExecute;
+import static accord.impl.progresslog.Progress.NoneExpected;
+import static accord.impl.progresslog.Progress.Querying;
+import static accord.impl.progresslog.Progress.Queued;
+import static accord.impl.progresslog.TxnStateKind.Home;
+
+/**
+ * TODO (required): describe state machine
+ *
+ * TODO (expected): do not attempt recovery every run; simply check the coordinator is still active
+ * TODO (expected): do not attempt execution until all shards are ready; use the WaitingState to achieve this
+ */
+abstract class HomeState extends WaitingState
+{
+ private static final int PROGRESS_SHIFT = WaitingState.WAITING_STATE_END_SHIFT;
+ private static final long PROGRESS_MASK = 0x3;
+ private static final int STATUS_SHIFT = PROGRESS_SHIFT + 2;
+ private static final long STATUS_MASK = 0x7;
+ private static final int RETRY_COUNTER_SHIFT = PROGRESS_SHIFT + 3;
+ private static final long RETRY_COUNTER_MASK = 0x7;
+ private static final long SET_MASK = ~((PROGRESS_MASK << PROGRESS_SHIFT)
+ | (STATUS_MASK << STATUS_SHIFT));
+ final int HOME_STATE_END_SHIFT = RETRY_COUNTER_SHIFT + 3;
+
+ HomeState(TxnId txnId)
+ {
+ super(txnId);
+ }
+
+ void set(SafeCommandStore safeStore, DefaultProgressLog instance, CoordinatePhase newCoordinatePhase, Progress newProgress)
+ {
+ encodedState &= SET_MASK;
+ encodedState |= ((long)newCoordinatePhase.ordinal() << STATUS_SHIFT)
+ | ((long)newProgress.ordinal() << PROGRESS_SHIFT);
+
+ if (newProgress == NoneExpected)
+ instance.clearProgressToken(txnId);
+ updateScheduling(safeStore, instance, Home, CanCoordinateExecution, newProgress);
+ }
+
+ @Nonnull CoordinatePhase phase()
+ {
+ return phase(encodedState);
+ }
+
+ final @Nonnull Progress homeProgress()
+ {
+ return homeProgress(encodedState);
+ }
+
+ private static @Nonnull CoordinatePhase phase(long encodedState)
+ {
+ return CoordinatePhase.forOrdinal((int) ((encodedState >>> STATUS_SHIFT) & STATUS_MASK));
+ }
+
+ private static @Nonnull Progress homeProgress(long encodedState)
+ {
+ return Progress.forOrdinal((int) ((encodedState >>> PROGRESS_SHIFT) & PROGRESS_MASK));
+ }
+
+ final int homeRetryCounter()
+ {
+ return (int) ((encodedState >>> RETRY_COUNTER_SHIFT) & RETRY_COUNTER_MASK);
+ }
+
+ final void incrementHomeRetryCounter()
+ {
+ long shiftedMask = RETRY_COUNTER_MASK << RETRY_COUNTER_SHIFT;
+ long current = encodedState & shiftedMask;
+ long updated = Math.min(shiftedMask, current + (1L << RETRY_COUNTER_SHIFT));
+ encodedState &= ~shiftedMask;
+ encodedState |= updated;
+ }
+
+ final void clearHomeRetryCounter()
+ {
+ long shiftedMask = RETRY_COUNTER_MASK << RETRY_COUNTER_SHIFT;
+ encodedState &= ~shiftedMask;
+ }
+
+
+ void atLeast(SafeCommandStore safeStore, DefaultProgressLog instance, CoordinatePhase newPhase, Progress newProgress)
+ {
+ if (phase() == Done)
+ return;
+
+ if (newPhase.compareTo(phase()) > 0)
+ {
+ instance.clearActive(Home, txnId);
+ clearHomeRetryCounter();
+ set(safeStore, instance, newPhase, newProgress);
+ }
+ }
+
+ final void runHome(DefaultProgressLog instance, SafeCommandStore safeStore, SafeCommand safeCommand)
+ {
+ Invariants.checkState(!isHomeDoneOrUninitialised());
+ Command command = safeCommand.current();
+ Invariants.checkState(!safeStore.isTruncated(command), "Command %s is truncated", command);
+
+ // TODO (expected): when invalidated, safer to maintain HomeState until known to be globally invalidated
+ // TODO (now): validate that we clear HomeState when we receive a Durable reply, to replace the token check logic
+ Invariants.checkState(!command.durability().isDurableOrInvalidated(), "Command is durable or invalidated, but we have not cleared the ProgressLog");
+
+ ProgressToken maxProgressToken = instance.savedProgressToken(txnId).merge(command);
+ MaybeRecover.maybeRecover(instance.node(), txnId, command.route(), maxProgressToken, invokeHomeCallback(instance, txnId, maxProgressToken, HomeState::recoverCallback));
+ set(safeStore, instance, ReadyToExecute, Querying);
+ }
+
+ static void recoverCallback(SafeCommandStore safeStore, SafeCommand safeCommand, DefaultProgressLog instance, TxnId txnId, @Nullable ProgressToken prevProgressToken, Outcome success, Throwable fail)
+ {
+ HomeState state = instance.get(txnId);
+ if (state == null)
+ return;
+
+ CoordinatePhase status = state.phase();
+ if (status.isAtMostReadyToExecute() && state.homeProgress() == Querying)
+ {
+ if (fail != null)
+ {
+ state.incrementHomeRetryCounter();
+ state.set(safeStore, instance, status, Queued);
+ }
+ else
+ {
+ Command command = safeCommand.current();
+ ProgressToken token = success.asProgressToken().merge(command);
+ if (prevProgressToken != null)
+ token = token.merge(prevProgressToken);
+
+ if (token.durability.isDurableOrInvalidated())
+ {
+ state.setHomeDoneAnyMaybeRemove(instance);
+ }
+ else
+ {
+ if (prevProgressToken != null && token.compareTo(command) > 0)
+ instance.saveProgressToken(command.txnId(), token);
+ state.set(safeStore, instance, status, Queued);
+ }
+ }
+ }
+ }
+
+ void setHomeDone(DefaultProgressLog instance)
+ {
+ set(null, instance, Done, NoneExpected);
+ clearHomeRetryCounter();
+ instance.clearActive(Home, txnId);
+ }
+
+ void setHomeDoneAnyMaybeRemove(DefaultProgressLog instance)
+ {
+ setHomeDone(instance);
+ maybeRemove(instance);
+ }
+
+ @Override
+ public String toStateString()
+ {
+ return (isHomeUninitialised() ? "" : isHomeDone() ? "Done; " : "{" + phase() + ',' + homeProgress() + "}; ") + super.toStateString();
+ }
+
+ boolean isHomeDone()
+ {
+ return phase() == Done;
+ }
+
+ boolean isHomeDoneOrUninitialised()
+ {
+ CoordinatePhase phase = phase();
+ return phase == Done || phase == CoordinatePhase.NotInitialised;
+ }
+
+ boolean isHomeInitialised()
+ {
+ return phase() != CoordinatePhase.NotInitialised;
+ }
+
+ private boolean isHomeUninitialised()
+ {
+ return phase() == CoordinatePhase.NotInitialised;
+ }
+}
diff --git a/accord-core/src/main/java/accord/impl/progresslog/PackedKeyTracker.java b/accord-core/src/main/java/accord/impl/progresslog/PackedKeyTracker.java
new file mode 100644
index 00000000..14619511
--- /dev/null
+++ b/accord-core/src/main/java/accord/impl/progresslog/PackedKeyTracker.java
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.impl.progresslog;
+
+import accord.local.Node;
+import accord.primitives.Ranges;
+import accord.primitives.Routables;
+import accord.primitives.Route;
+import accord.primitives.TxnId;
+import accord.primitives.Unseekables;
+import accord.utils.Invariants;
+
+/**
+ * Using a predefined number of bits, store state allowing us to iterate through participating keys in groups
+ * small enough we can maintain a bitset of the keys that have met our desired criteria.
+ *
+ * This permits us to register remote callbacks with minimal state on the sender and recipient and track a transaction's
+ * progress towards whatever local or distributed state we need to achieve.
+ *
+ * We split the bits dynamically between a bitSet and a roundIndex.
+ * The bitSet has a number of bits equal to the roundSize, and the roundIndex has enough bits
+ * to encode the total number of rounds we need of the given size.
+ * <p>
+ * In the typical case we have one round with enough bits for all keys.
+ * At the other extreme we have many rounds each fetching a single key because we have too many keys
+ * to represent in the number of bits available.
+ */
+public class PackedKeyTracker
+{
+ /**
+ * Calculate a packed round size for a given number of bits.
+ */
+ static int roundSize(int routeSize, int totalBits)
+ {
+ if (routeSize < totalBits)
+ return routeSize;
+
+ return Math.toIntExact((1L << (totalBits - 1)) / routeSize);
+ }
+
+ /**
+ * Generate a bitSet of those keys we would have requested from the responding replica for the callbackId.
+ * These bits can be unset from the state's bitSet, as their criteria have been fulfilled on the remote replica.
+ */
+ static int roundCallbackBitSet(DefaultProgressLog instance, TxnId txnId, Node.Id from, Route<?> route, int callbackId, int roundIndex, int roundSize)
+ {
+ if (callbackId != roundIndex)
+ {
+ Invariants.checkState(callbackId / roundSize < roundIndex);
+ return 0; // stale callback for earlier round
+ }
+
+ int start = roundSize * roundIndex;
+ int end = Math.min((roundSize + 1) * roundIndex, route.size());
+
+ Ranges sliceRanges = instance.node().topology().globalForEpoch(txnId.epoch()).rangesForNode(from);
+ Route<?> ready = route.slice(start, end)
+ .slice(sliceRanges);
+
+ return computeBitSet(route, ready, roundIndex, roundSize);
+ }
+
+ /**
+ * Read the packed bitSet for a round of the provided size
+ */
+ static int bitSet(long encodedState, int roundSize, int shift)
+ {
+ return (int) ((encodedState >>> shift) & ((1L << roundSize) - 1));
+ }
+
+ /**
+ * Set the packed round bitSet for a round of the provided size
+ */
+ static long setBitSet(long encodedState, int bitSet, int roundSize, int shift)
+ {
+ long mask = (1L << roundSize) - 1;
+ Invariants.checkArgument(Integer.highestOneBit(bitSet) <= mask);
+ encodedState &= ~(mask << shift);
+ encodedState |= (long) bitSet << shift;
+ return encodedState;
+ }
+
+ static long initialiseBitSet(long encodedState, Route<?> route, Unseekables<?> intersecting, int roundIndex, int roundSize, int shift)
+ {
+ int bitSet = computeBitSet(route, intersecting, roundIndex, roundSize);
+ return setBitSet(encodedState, bitSet, roundSize, shift);
+ }
+
+ static int computeBitSet(Route<?> route, Unseekables<?> intersecting, int roundIndex, int roundSize)
+ {
+ int bitSet = 0, start = roundSize * roundIndex, ai = start, bi = 0;
+ for (int i = 0; i < intersecting.size(); ++i)
+ {
+ long abi = route.findNextIntersection(ai, (Routables) intersecting, bi);
+ if (abi < 0)
+ break;
+
+ ai = (int) abi;
+ bi = (int) (abi >>> 32);
+ bitSet |= 1 << (ai - start);
+ }
+ return bitSet;
+ }
+
+ /**
+ * Read the packed round index for rounds of the provided size
+ */
+ static int roundIndex(long encodedState, int roundSize, int shift, long mask)
+ {
+ return (int) (((encodedState >>> shift) & mask) >>> roundSize);
+ }
+
+ /**
+ * Set the packed round index for a round of the provided size
+ */
+ static long setRoundIndexAndClearBitSet(long encodedState, int newIndex, int roundSize, int shift, long mask)
+ {
+ Invariants.checkArgument(roundSize > 0);
+ int counterBits = shift - roundSize;
+ Invariants.checkState(newIndex < (1L << counterBits));
+ int counterShift = shift + roundSize;
+ encodedState &= ~(mask << shift);
+ encodedState |= (long) newIndex << counterShift;
+ return encodedState;
+ }
+
+ /**
+ * Set the packed round index for a round of the provided size
+ */
+ static long setMaxRoundIndexAndClearBitSet(long encodedState, int roundSize, int shift, long mask)
+ {
+ Invariants.checkArgument(roundSize > 0);
+ int counterBits = shift - roundSize;
+ long maxIndex = (1L << counterBits) - 1;
+ int counterShift = shift + roundSize;
+ encodedState &= ~(mask << shift);
+ encodedState |= maxIndex << counterShift;
+ return encodedState;
+ }
+
+ static long clearRoundState(long encodedState, int shift, long mask)
+ {
+ return encodedState & ~(mask << shift);
+ }
+
+}
diff --git a/accord-core/src/main/java/accord/impl/progresslog/PackedLogLinearInteger.java b/accord-core/src/main/java/accord/impl/progresslog/PackedLogLinearInteger.java
new file mode 100644
index 00000000..78a5b30e
--- /dev/null
+++ b/accord-core/src/main/java/accord/impl/progresslog/PackedLogLinearInteger.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.impl.progresslog;
+
+import accord.utils.Invariants;
+
+/**
+ * A simple encoding scheme for very lossy storage of an integer, where the inaccuracy is proportional to the log scale
+ * of the value being stored. That is, a number is truncated to its nearest highest M bits, with the highest bit position
+ * being stored as an integer of log2(N) bits, and the remaining bits being stored directly - only that they may be
+ * incremented by 1, if the remaining bits of the original integer represent more than half of the value of the lowest
+ * stored bit (i.e. we round to the nearest, rather than truncating).
+ */
+public class PackedLogLinearInteger
+{
+ public static int validateLowBits(int numberOfLowBits, int numberOfBits)
+ {
+ Invariants.checkArgument(numberOfBits <= 30, "%s bits can store the raw input without packing", numberOfBits);
+ int numberOfHighBits = numberOfBits - numberOfLowBits;
+ Invariants.checkArgument(numberOfHighBits <= 5, numberOfHighBits + " bits is too many to represent the maximum bit position");
+ Invariants.checkArgument(numberOfHighBits > 0, numberOfHighBits + " bits cannot be zero; this produces a simple linear integer");
+ return numberOfLowBits;
+ }
+
+ public static int encode(int value, int numberOfLowBits, int numberOfBits)
+ {
+ Invariants.checkState(value >= 0);
+ if (value < 1 << numberOfLowBits)
+ return value;
+
+ int highestSetBit = Integer.highestOneBit(value);
+ int highBits = Math.max(0, 1 + (Integer.numberOfTrailingZeros(highestSetBit) & 31) - numberOfLowBits);
+ value ^= highestSetBit;
+ // include one additional bit in lowBits for rounding, and discount highest bit
+ if (highBits >= 2) value >>>= highBits - 2;
+ else value <<= 2 - highBits;
+ // exclude the rounding bit when packing
+ int result = (value >>> 1) | (highBits << numberOfLowBits);
+ // then increment by any rounding bit, permitting this to overflow and increment our highBits
+ result += value & 1;
+ // in case of overflow, saturate our high and low bits
+ if (result >= 1 << numberOfBits)
+ return -1 >>> (32 - numberOfBits);
+ return result;
+ }
+
+ public static int decode(int encoded, int lowBits)
+ {
+ int bitShift = encoded >>> lowBits;
+ if (bitShift == 0)
+ return encoded;
+
+ --bitShift;
+ int highestBit = 1 << lowBits;
+ int bits = highestBit | (encoded & (highestBit - 1));
+ return bits << bitShift;
+ }
+}
diff --git a/accord-core/src/main/java/accord/impl/progresslog/Progress.java b/accord-core/src/main/java/accord/impl/progresslog/Progress.java
new file mode 100644
index 00000000..952edb8b
--- /dev/null
+++ b/accord-core/src/main/java/accord/impl/progresslog/Progress.java
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.impl.progresslog;
+
+enum Progress
+{
+ /**
+ * We do not expect any progress for this state machine at present
+ */
+ NoneExpected,
+
+ /**
+ * We expect progress for this state machine, and have queued the TxnState to be processed
+ */
+ Queued,
+
+ /**
+ * The TxnState has recently been processed and sent queries to some replicas to attempt to progress out state machine.
+ * These are messages that expect a synchronous response, and we rely on the outcome of this action to proceed.
+ * We do not register any local timeouts to retry, as we expect the coordination of this query to have its own timeouts.
+ */
+ Querying,
+
+ /**
+ * The TxnState has recently been processed, and a synchronous remote query has completed without finding the
+ * relevant remote state to progress out local state machine, but a remote listener has been registered that
+ * will notify us when a remote replica reaches a point where we can advance our state machine.
+ *
+ * We also register a local timeout to retry, in case this remote reply or the remote listener itself is lost.
+ */
+ Awaiting;
+
+ private static final Progress[] lookup = values();
+
+ public static Progress forOrdinal(int ordinal)
+ {
+ if (ordinal < 0 || ordinal > lookup.length)
+ throw new IndexOutOfBoundsException(ordinal);
+ return lookup[ordinal];
+ }
+}
diff --git a/accord-core/src/main/java/accord/messages/Request.java b/accord-core/src/main/java/accord/impl/progresslog/SavedProgressToken.java
similarity index 63%
copy from accord-core/src/main/java/accord/messages/Request.java
copy to accord-core/src/main/java/accord/impl/progresslog/SavedProgressToken.java
index 0c923716..0b3fea45 100644
--- a/accord-core/src/main/java/accord/messages/Request.java
+++ b/accord-core/src/main/java/accord/impl/progresslog/SavedProgressToken.java
@@ -16,15 +16,23 @@
* limitations under the License.
*/
-package accord.messages;
+package accord.impl.progresslog;
-import accord.local.Node;
-import accord.local.Node.Id;
+import accord.primitives.ProgressToken;
+import accord.primitives.TxnId;
-public interface Request extends Message
+class SavedProgressToken extends ProgressToken
{
- default long waitForEpoch() { return 0; }
- void preProcess(Node on, Id from, ReplyContext replyContext);
- void process(Node on, Id from, ReplyContext replyContext);
+ final TxnId txnId;
+ public SavedProgressToken(TxnId txnId, ProgressToken copy)
+ {
+ super(copy.durability, copy.status, copy.promised, copy.isAccepted);
+ this.txnId = txnId;
+ }
+
+ public int compare(SavedProgressToken that)
+ {
+ return this.txnId.compareTo(that.txnId);
+ }
}
diff --git a/accord-core/src/main/java/accord/impl/progresslog/TxnState.java b/accord-core/src/main/java/accord/impl/progresslog/TxnState.java
new file mode 100644
index 00000000..95ef3845
--- /dev/null
+++ b/accord-core/src/main/java/accord/impl/progresslog/TxnState.java
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.impl.progresslog;
+
+import javax.annotation.Nullable;
+
+import com.google.common.primitives.Ints;
+
+import accord.api.ProgressLog.BlockedUntil;
+import accord.local.SafeCommandStore;
+import accord.primitives.TxnId;
+import accord.utils.Invariants;
+
+import static java.util.concurrent.TimeUnit.MICROSECONDS;
+
+final class TxnState extends HomeState
+{
+ TxnState(TxnId txnId)
+ {
+ super(txnId);
+ }
+
+ void updateScheduling(SafeCommandStore safeStore, DefaultProgressLog instance, TxnStateKind updated, @Nullable BlockedUntil blockedUntil, Progress newProgress)
+ {
+ long newDelay;
+ switch (newProgress)
+ {
+ default:
+ throw new AssertionError("Unhandled Progress: " + newProgress);
+ case NoneExpected:
+ case Querying:
+ newDelay = 0;
+ break;
+ case Queued:
+ switch (updated)
+ {
+ default: throw new AssertionError("Unhandled TxnStateKind: " + updated);
+ case Waiting:
+ newDelay = instance.commandStore.agent().seekProgressDelay(instance.node, safeStore, txnId, waitingRetryCounter(), blockedUntil, MICROSECONDS);
+ break;
+ case Home:
+ newDelay = instance.commandStore.agent().attemptCoordinationDelay(instance.node, safeStore, txnId, MICROSECONDS, homeRetryCounter());
+ }
+ Invariants.checkState(newDelay > 0);
+ break;
+ case Awaiting:
+ int retries = updated == TxnStateKind.Home ? homeRetryCounter() : waitingRetryCounter();
+ newDelay = instance.commandStore.agent().retryAwaitTimeout(instance.node, safeStore, txnId, retries, blockedUntil, MICROSECONDS);
+ Invariants.checkState(newDelay > 0);
+ break;
+ }
+
+ TxnStateKind scheduled = scheduledTimer();
+ if (scheduled == null)
+ {
+ Invariants.checkState(pendingTimer() == null);
+ }
+
+ // previousDeadline is the previous deadline of <updated>;
+ // otherDeadline is the active deadline (if any) of <updated.other()>
+ long previousDeadline, otherDeadline;
+ if (scheduled == updated)
+ {
+ previousDeadline = deadline();
+ otherDeadline = pendingTimerDeadline(previousDeadline);
+ }
+ else if (scheduled != null)
+ {
+ otherDeadline = deadline();
+ previousDeadline = pendingTimerDeadline(otherDeadline);
+ }
+ else
+ {
+ Invariants.checkState(pendingTimer() == null);
+ otherDeadline = previousDeadline = 0;
+ }
+
+ if (newDelay == 0)
+ {
+ if (otherDeadline > 0)
+ {
+ clearPendingTimerDelay();
+ setScheduledTimer(updated.other());
+ instance.update(otherDeadline, this);
+ }
+ else if (previousDeadline > 0)
+ {
+ instance.unschedule(this);
+ }
+ else
+ {
+ Invariants.checkState(!isScheduled());
+ }
+ }
+ else
+ {
+ long nowMicros = instance.node().elapsed(MICROSECONDS);
+ long newDeadline = nowMicros + newDelay;
+ if (otherDeadline == 0)
+ {
+ setScheduledTimer(updated);
+ if (previousDeadline > 0) instance.update(nowMicros, newDeadline, this);
+ else instance.add(nowMicros, newDeadline, this);
+ }
+ else if (newDeadline < otherDeadline)
+ {
+ setScheduledTimer(updated);
+ setPendingTimerDelay(Ints.saturatedCast(otherDeadline - newDeadline));
+ instance.update(nowMicros, newDeadline, this);
+ }
+ else
+ {
+ setScheduledTimer(updated.other());
+ setPendingTimerDelay(Ints.saturatedCast(Math.max(1, newDeadline - otherDeadline)));
+ instance.update(nowMicros, otherDeadline, this);
+ }
+ }
+ }
+
+ void maybeRemove(DefaultProgressLog instance)
+ {
+ if (isWaitingDone() && isHomeDoneOrUninitialised())
+ instance.remove(txnId);
+ }
+
+ HomeState home()
+ {
+ return this;
+ }
+
+ WaitingState waiting()
+ {
+ return this;
+ }
+
+ @Override
+ public String toString()
+ {
+ return txnId + ": " + toStateString();
+ }
+}
diff --git a/accord-core/src/main/java/accord/primitives/FullRoute.java b/accord-core/src/main/java/accord/impl/progresslog/TxnStateKind.java
similarity index 73%
copy from accord-core/src/main/java/accord/primitives/FullRoute.java
copy to accord-core/src/main/java/accord/impl/progresslog/TxnStateKind.java
index be0ba22a..382b662a 100644
--- a/accord-core/src/main/java/accord/primitives/FullRoute.java
+++ b/accord-core/src/main/java/accord/impl/progresslog/TxnStateKind.java
@@ -16,9 +16,17 @@
* limitations under the License.
*/
-package accord.primitives;
+package accord.impl.progresslog;
-public interface FullRoute<T extends Unseekable> extends Route<T>, Unseekables<T>
+/**
+ * The kinds of TxnState we support in the DefaultProgressLog. This is used to manage timers and other shared state.
+ */
+public enum TxnStateKind
{
- @Override default FullRoute<T> union(Route<T> route) { return this; }
+ Waiting, Home;
+
+ TxnStateKind other()
+ {
+ return this == Waiting ? Home : Waiting;
+ }
}
diff --git a/accord-core/src/main/java/accord/impl/progresslog/WaitingState.java b/accord-core/src/main/java/accord/impl/progresslog/WaitingState.java
new file mode 100644
index 00000000..792eae23
--- /dev/null
+++ b/accord-core/src/main/java/accord/impl/progresslog/WaitingState.java
@@ -0,0 +1,665 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package accord.impl.progresslog;
+
+import java.util.function.BiConsumer;
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import accord.api.ProgressLog.BlockedUntil;
+import accord.coordinate.AsynchronousAwait;
+import accord.coordinate.FetchData;
+import accord.local.Command;
+import accord.local.Node;
+import accord.local.SafeCommand;
+import accord.local.SafeCommandStore;
+import accord.local.SaveStatus;
+import accord.local.Status;
+import accord.primitives.EpochSupplier;
+import accord.primitives.Participants;
+import accord.primitives.Ranges;
+import accord.primitives.Routables;
+import accord.primitives.Route;
+import accord.primitives.Timestamp;
+import accord.primitives.TxnId;
+import accord.primitives.Unseekables;
+import accord.topology.Topologies;
+import accord.utils.Invariants;
+
+import static accord.api.ProgressLog.BlockedUntil.CanApply;
+import static accord.api.ProgressLog.BlockedUntil.Query.HOME;
+import static accord.api.ProgressLog.BlockedUntil.Query.SHARD;
+import static accord.impl.progresslog.CallbackInvoker.invokeWaitingCallback;
+import static accord.impl.progresslog.PackedKeyTracker.bitSet;
+import static accord.impl.progresslog.PackedKeyTracker.clearRoundState;
+import static accord.impl.progresslog.PackedKeyTracker.initialiseBitSet;
+import static accord.impl.progresslog.PackedKeyTracker.roundCallbackBitSet;
+import static accord.impl.progresslog.PackedKeyTracker.roundIndex;
+import static accord.impl.progresslog.PackedKeyTracker.roundSize;
+import static accord.impl.progresslog.PackedKeyTracker.setBitSet;
+import static accord.impl.progresslog.PackedKeyTracker.setMaxRoundIndexAndClearBitSet;
+import static accord.impl.progresslog.PackedKeyTracker.setRoundIndexAndClearBitSet;
+import static accord.impl.progresslog.Progress.Awaiting;
+import static accord.impl.progresslog.Progress.NoneExpected;
+import static accord.impl.progresslog.Progress.Querying;
+import static accord.impl.progresslog.Progress.Queued;
+import static accord.impl.progresslog.TxnStateKind.Waiting;
+import static accord.impl.progresslog.WaitingState.CallbackKind.AwaitHome;
+import static accord.impl.progresslog.WaitingState.CallbackKind.AwaitSlice;
+import static accord.impl.progresslog.WaitingState.CallbackKind.Fetch;
+import static accord.impl.progresslog.WaitingState.CallbackKind.FetchRoute;
+import static accord.primitives.EpochSupplier.constant;
+
+/**
+ * This represents a simple state machine encoded in a small number of bits for efficiently gathering
+ * distributed state we require locally to make progress.
+ * <p>
+ * The state machine consists of the following packed registers:
+ * - target BlockedUntil
+ * - The BlockUntil we know at least one home shard replica is able to satisfy
+ * - A packed bitset/counter for enumerating the relevant keys and awaiting
+ * remote replicas for the keys to be ready to satisfy our local requirements
+ *
+ */
+@SuppressWarnings("CodeBlock2Expr")
+abstract class WaitingState extends BaseTxnState
+{
+ private static final int PROGRESS_SHIFT = 0;
+ private static final long PROGRESS_MASK = 0x3;
+ private static final int BLOCKED_UNTIL_SHIFT = 2;
+ private static final long BLOCKED_UNTIL_MASK = 0x7;
+ private static final int HOME_SATISFIES_SHIFT = 5;
+ private static final long HOME_SATISFIES_MASK = 0x7;
+ private static final int AWAIT_BITS = 32;
+ private static final int AWAIT_SHIFT = 8;
+ private static final long AWAIT_MASK = (1L << AWAIT_BITS) - 1;
+ private static final long SET_MASK = ~((PROGRESS_MASK << PROGRESS_SHIFT) | (BLOCKED_UNTIL_MASK << BLOCKED_UNTIL_SHIFT));
+ private static final long INITIALISED_MASK = (PROGRESS_MASK << PROGRESS_SHIFT) | (BLOCKED_UNTIL_MASK << BLOCKED_UNTIL_SHIFT) | (HOME_SATISFIES_MASK << HOME_SATISFIES_SHIFT);
+
+ private static final int RETRY_COUNTER_SHIFT = 40;
+ private static final long RETRY_COUNTER_MASK = 0x7;
+ static final int WAITING_STATE_END_SHIFT = RETRY_COUNTER_SHIFT + 3;
+
+ // when awaiting shards we register callbacks numbered by the keys we're processing;
+ // we want to special-case the home key callback and its easiest to pick the highest integer
+ // so that we know it won't clash
+ public static final int AWAITING_HOME_KEY_CALLBACKID = Integer.MAX_VALUE;
+
+ WaitingState(TxnId txnId)
+ {
+ super(txnId);
+ }
+
+ private void set(SafeCommandStore safeStore, DefaultProgressLog owner, BlockedUntil newBlockedUntil, Progress newProgress)
+ {
+ encodedState &= SET_MASK;
+ encodedState |= ((long) newBlockedUntil.ordinal() << BLOCKED_UNTIL_SHIFT) | ((long) newProgress.ordinal() << PROGRESS_SHIFT);
+ updateScheduling(safeStore, owner, Waiting, newBlockedUntil, newProgress);
+ }
+
+ private void setHomeSatisfies(BlockedUntil homeStatus)
+ {
+ encodedState &= ~(HOME_SATISFIES_MASK << HOME_SATISFIES_SHIFT);
+ encodedState |= (long) homeStatus.ordinal() << HOME_SATISFIES_SHIFT;
+ }
+
+ boolean isUninitialised()
+ {
+ return 0 == (encodedState & INITIALISED_MASK);
+ }
+
+ @Nonnull BlockedUntil blockedUntil()
+ {
+ return blockedUntil(encodedState);
+ }
+
+ @Nonnull BlockedUntil homeSatisfies()
+ {
+ return homeSatisfies(encodedState);
+ }
+
+ final @Nonnull Progress waitingProgress()
+ {
+ return waitingProgress(encodedState);
+ }
+
+ private static @Nonnull BlockedUntil blockedUntil(long encodedState)
+ {
+ return BlockedUntil.forOrdinal((int) ((encodedState >>> BLOCKED_UNTIL_SHIFT) & BLOCKED_UNTIL_MASK));
+ }
+
+ private static @Nonnull BlockedUntil homeSatisfies(long encodedState)
+ {
+ return BlockedUntil.forOrdinal((int) ((encodedState >>> HOME_SATISFIES_SHIFT) & HOME_SATISFIES_MASK));
+ }
+
+ private static @Nonnull Progress waitingProgress(long encodedState)
+ {
+ return Progress.forOrdinal((int) ((encodedState >>> PROGRESS_SHIFT) & PROGRESS_MASK));
+ }
+
+ private static int awaitRoundSize(Route<?> slicedRoute)
+ {
+ return roundSize(slicedRoute.size(), AWAIT_BITS);
+ }
+
+ private void clearAwaitState()
+ {
+ encodedState = clearRoundState(encodedState, AWAIT_SHIFT, AWAIT_MASK);
+ }
+
+ private int awaitBitSet(int roundSize)
+ {
+ return bitSet(encodedState, roundSize, AWAIT_SHIFT);
+ }
+
+ private void initialiseAwaitBitSet(Route<?> route, Unseekables<?> notReady, int roundIndex, int roundSize)
+ {
+ encodedState = initialiseBitSet(encodedState, route, notReady, roundIndex, roundSize, AWAIT_SHIFT);
+ }
+
+ private void setAwaitBitSet(int bitSet, int roundSize)
+ {
+ encodedState = setBitSet(encodedState, bitSet, roundSize, AWAIT_SHIFT);
+ }
+
+ private int awaitRoundIndex(int roundSize)
+ {
+ return roundIndex(encodedState, roundSize, AWAIT_SHIFT, AWAIT_MASK);
+ }
+
+ private void updateAwaitRound(int newRoundIndex, int roundSize)
+ {
+ Invariants.checkArgument(roundSize <= AWAIT_BITS);
+ encodedState = setRoundIndexAndClearBitSet(encodedState, newRoundIndex, roundSize, AWAIT_SHIFT, AWAIT_MASK);
+ }
+
+ private void setAwaitDone(int roundSize)
+ {
+ Invariants.checkArgument(roundSize <= AWAIT_BITS);
+ encodedState = setMaxRoundIndexAndClearBitSet(encodedState, roundSize, AWAIT_SHIFT, AWAIT_MASK);
+ }
+
+ final int waitingRetryCounter()
+ {
+ return (int) ((encodedState >>> RETRY_COUNTER_SHIFT) & RETRY_COUNTER_MASK);
+ }
+
+ final void incrementWaitingRetryCounter()
+ {
+ long shiftedMask = RETRY_COUNTER_MASK << RETRY_COUNTER_SHIFT;
+ long current = encodedState & shiftedMask;
+ long updated = Math.min(shiftedMask, current + (1L << RETRY_COUNTER_SHIFT));
+ encodedState &= ~shiftedMask;
+ encodedState |= updated;
+ }
+
+ final void clearWaitingRetryCounter()
+ {
+ long shiftedMask = RETRY_COUNTER_MASK << RETRY_COUNTER_SHIFT;
+ encodedState &= ~shiftedMask;
+ }
+
+ Topologies contact(DefaultProgressLog owner, Unseekables<?> forKeys, long epoch)
+ {
+ Node node = owner.node();
+ Topologies topologies = node.topology().forEpoch(forKeys, epoch);
+ return node.agent().selectPreferred(node.id(), topologies);
+ }
+
+ /*
+ * Ranges may have moved between command stores locally so extend to a later epoch to invoke those command stores
+ */
+ private static EpochSupplier toLocalEpoch(SafeCommandStore safeStore, TxnId txnId, BlockedUntil blockedUntil, Command command, Timestamp executeAt)
+ {
+ long epoch = blockedUntil.fetchEpoch(txnId, executeAt);
+ if (command.route() != null)
+ epoch = Math.max(epoch, safeStore.ranges().latestEpochWithNewParticipants(epoch, command.route()));
+ if (command.additionalKeysOrRanges() != null)
+ epoch = Math.max(epoch, safeStore.ranges().latestEpochWithNewParticipants(epoch, command.additionalKeysOrRanges()));
+ return constant(epoch);
+ }
+
+ private static Route<?> slicedRoute(SafeCommandStore safeStore, TxnId txnId, Command command, BlockedUntil blockedUntil)
+ {
+ Timestamp executeAt = command.executeAtIfKnown();
+ EpochSupplier toLocalEpoch = toLocalEpoch(safeStore, txnId, blockedUntil, command, executeAt);
+
+ Ranges ranges = safeStore.ranges().allBetween(txnId.epoch(), toLocalEpoch);
+ return command.route().slice(ranges);
+ }
+
+ private static Route<?> slicedRoute(SafeCommandStore safeStore, TxnId txnId, Route<?> route, EpochSupplier toLocalEpoch)
+ {
+ Ranges ranges = safeStore.ranges().allBetween(txnId.epoch(), toLocalEpoch);
+ return route.slice(ranges);
+ }
+
+ private static Route<?> awaitRoute(Route<?> slicedRoute, BlockedUntil blockedUntil)
+ {
+ return blockedUntil.waitsOn == HOME ? slicedRoute.homeKeyOnlyRoute() : slicedRoute;
+ }
+
+ private static Route<?> fetchRoute(Route<?> slicedRoute, Route<?> awaitRoute, BlockedUntil blockedUntil)
+ {
+ return blockedUntil.waitsOn == blockedUntil.fetchFrom ? awaitRoute : slicedRoute;
+ }
+
+ void setWaitingDone(DefaultProgressLog owner)
+ {
+ set(null, owner, CanApply, NoneExpected);
+ owner.clearActive(Waiting, txnId);
+ clearWaitingRetryCounter();
+ }
+
+ void setBlockedUntil(SafeCommandStore safeStore, DefaultProgressLog owner, BlockedUntil blockedUntil)
+ {
+ BlockedUntil currentlyBlockedUntil = blockedUntil();
+ if (blockedUntil.compareTo(currentlyBlockedUntil) > 0 || isUninitialised())
+ {
+ clearAwaitState();
+ clearWaitingRetryCounter();
+ owner.clearActive(Waiting, txnId);
+ set(safeStore, owner, blockedUntil, Queued);
+ }
+ }
+
+ void record(DefaultProgressLog owner, SaveStatus newSaveStatus)
+ {
+ BlockedUntil currentlyBlockedUntil = blockedUntil();
+ if (currentlyBlockedUntil.minSaveStatus.compareTo(newSaveStatus) <= 0)
+ {
+ boolean isDone = newSaveStatus.hasBeen(Status.PreApplied);
+ set(null, owner, isDone ? CanApply : currentlyBlockedUntil, NoneExpected);
+ if (isDone)
+ maybeRemove(owner);
+ owner.clearActive(Waiting, txnId);
+ }
+ }
+
+ final void runWaiting(SafeCommandStore safeStore, SafeCommand safeCommand, DefaultProgressLog owner)
+ {
+ runInternal(safeStore, safeCommand, owner);
+ }
+
+ private void runInternal(SafeCommandStore safeStore, SafeCommand safeCommand, DefaultProgressLog owner)
+ {
+ BlockedUntil blockedUntil = blockedUntil();
+ Command command = safeCommand.current();
+ Invariants.checkState(!owner.hasActive(Waiting, txnId));
+ Invariants.checkState(command.saveStatus().compareTo(blockedUntil.minSaveStatus) < 0, "Command has met desired criteria but progress log entry has not been cancelled");
+
+ set(safeStore, owner, blockedUntil, Querying);
+ TxnId txnId = safeCommand.txnId();
+ // first make sure we have enough information to obtain the command locally
+ Timestamp executeAt = command.executeAtIfKnown();
+ EpochSupplier toLocalEpoch = toLocalEpoch(safeStore, txnId, blockedUntil, command, executeAt);
+ Participants<?> fetchKeys = Invariants.nonNull(command.maxContactable());
+
+ if (!Route.isRoute(fetchKeys))
+ {
+ fetchRoute(owner, blockedUntil, txnId, executeAt, toLocalEpoch, fetchKeys);
+ return;
+ }
+
+ Route<?> route = Route.castToRoute(fetchKeys);
+ if (homeSatisfies().compareTo(blockedUntil) < 0)
+ {
+ // first wait until the homeKey has progressed to a point where it can answer our query; we don't expect our shards to know until then anyway
+ awaitHomeKey(owner, blockedUntil, txnId, executeAt, route);
+ return;
+ }
+
+ Route<?> slicedRoute = slicedRoute(safeStore, txnId, route, toLocalEpoch);
+ if (!command.hasBeen(Status.PreCommitted))
+ {
+ // we know it has been decided one way or the other by the home shard at least, so we attempt a fetch
+ // including the home shard to get us to at least PreCommitted where we can safely wait on individual shards
+ fetch(owner, blockedUntil, txnId, executeAt, toLocalEpoch, slicedRoute, slicedRoute.withHomeKey());
+ return;
+ }
+
+ // the awaitRoute may be only the home shard, if that is sufficient to indicate the fetchRoute will be able to answer our query;
+ // the fetchRoute may also be only the home shard, if that is sufficient to answer our query (e.g. for executeAt)
+ Route<?> awaitRoute = awaitRoute(slicedRoute, blockedUntil);
+ Route<?> fetchRoute = fetchRoute(slicedRoute, awaitRoute, blockedUntil);
+
+ if (slicedRoute.size() == 0 || awaitRoute.isHomeKeyOnlyRoute())
+ {
+ // either we await only the home shard (which is already known to meet our criteria)
+ // or there are no specific local keys we want to update;
+ // at this point we can switch to polling as we know someone (i.e. home shard) has the relevant state
+ fetch(owner, blockedUntil, txnId, executeAt, toLocalEpoch, slicedRoute, fetchRoute);
+ return;
+ }
+
+ int roundSize = awaitRoundSize(awaitRoute);
+ int roundIndex = awaitRoundIndex(roundSize);
+ int roundStart = roundIndex * roundSize;
+ if (roundStart >= awaitRoute.size())
+ {
+ // all of the shards we are awaiting have been processed and found at least one replica that has the state needed to answer our query
+ // at this point we can switch to polling as we know someone has the relevant state
+ fetch(owner, blockedUntil, txnId, executeAt, toLocalEpoch, slicedRoute, fetchRoute);
+ return;
+ }
+
+ int roundEnd = Math.min(roundStart + roundSize, awaitRoute.size());
+ awaitRoute = awaitRoute.slice(roundStart, roundEnd);
+ // TODO (desired): use some mechanism (e.g. random chance or another counter)
+ // to either periodically fetch the whole remaining route or gradually increase the slice length
+ awaitSlice(owner, blockedUntil, txnId, executeAt, awaitRoute, (roundIndex << 1) | 1);
+ }
+
+ // note that ready and notReady may include keys not requested by this progressLog
+ static void awaitOrFetchCallback(CallbackKind kind, SafeCommandStore safeStore, SafeCommand safeCommand, DefaultProgressLog owner, TxnId txnId, BlockedUntil blockedUntil, Unseekables<?> ready, @Nullable Unseekables<?> notReady, @Nullable BlockedUntil upgrade, Throwable fail)
+ {
+ WaitingState state = owner.get(txnId);
+ Invariants.checkState(state != null, "State has been cleared but callback was not cancelled");
+
+ Invariants.checkState(state.waitingProgress() == Querying);
+ Invariants.checkState(state.blockedUntil() == blockedUntil);
+
+ Command command = safeCommand.current();
+ Route<?> route = command.route();
+
+ if (fail == null)
+ {
+ if (route == null)
+ {
+ Invariants.checkState(kind == FetchRoute);
+ state.retry(safeStore, safeCommand, owner, blockedUntil);
+ return;
+ }
+
+ if (ready.contains(route.homeKey()) && blockedUntil.compareTo(state.homeSatisfies()) > 0)
+ {
+ // TODO (expected): we can introduce an additional home state check that waits until DURABLE for execution;
+ // at this point it would even be redundant to wait for each separate shard for the WaitingState? Freeing up bits and simplifying.
+ BlockedUntil newHomeSatisfies = blockedUntil;
+ if (upgrade != null && upgrade.compareTo(newHomeSatisfies) > 0)
+ newHomeSatisfies = upgrade;
+ state.setHomeSatisfies(newHomeSatisfies);
+ }
+
+ EpochSupplier toLocalEpoch = toLocalEpoch(safeStore, txnId, blockedUntil, command, command.executeAtOrTxnId());
+ Route<?> slicedRoute = slicedRoute(safeStore, txnId, route, toLocalEpoch); // the actual local keys we care about
+ Route<?> awaitRoute = awaitRoute(slicedRoute, blockedUntil); // either slicedRoute or just the home key
+
+ int roundSize = awaitRoundSize(awaitRoute);
+ int roundIndex = state.awaitRoundIndex(roundSize);
+ int roundStart = roundIndex * roundSize;
+
+ switch (kind)
+ {
+ default: throw new AssertionError("Unhandled CallbackKind: " + kind);
+
+ case AwaitHome:
+ if (notReady == null)
+ {
+ // the home shard was found to already have the necessary state, with no distributed await;
+ // we can immediately progress the state machine
+ Invariants.checkState(0 == state.awaitRoundIndex(roundSize));
+ Invariants.checkState(0 == state.awaitBitSet(roundSize));
+ state.runInternal(safeStore, safeCommand, owner);
+ }
+ else
+ {
+ // the home shard is not ready to answer our query, but we have registered our remote callback so can wait for it to contact us
+ state.set(safeStore, owner, blockedUntil, Awaiting);
+ }
+ break;
+
+ case AwaitSlice:
+ Invariants.checkState(awaitRoute == slicedRoute);
+ if (notReady == null)
+ {
+ Invariants.checkState((int) awaitRoute.findNextIntersection(roundStart, (Routables) ready, 0) / roundSize == roundIndex);
+ // TODO (desired): in this case perhaps upgrade to fetch for next round?
+ state.updateAwaitRound(roundIndex + 1, roundSize);
+ state.runInternal(safeStore, safeCommand, owner);
+ }
+ else
+ {
+ Invariants.checkState((int) awaitRoute.findNextIntersection(roundStart, (Routables) notReady, 0) / roundSize == roundIndex);
+ // TODO (desired): would be nice to validate this is 0 in cases where we are starting a fresh round
+ // but have to be careful as cannot zero when we restart as we may have an async callback arrive while we're waiting that then advances state machine
+ state.initialiseAwaitBitSet(awaitRoute, notReady, roundIndex, roundSize);
+ state.set(safeStore, owner, blockedUntil, Awaiting);
+ }
+ break;
+
+ case FetchRoute:
+ if (state.homeSatisfies().compareTo(blockedUntil) < 0)
+ {
+ state.runInternal(safeStore, safeCommand, owner);
+ return;
+ }
+ // we may not have requested everything since we didn't have a Route, so calculate our own notReady and fall-through
+ notReady = slicedRoute.without(ready);
+
+ case Fetch:
+ {
+ Invariants.checkState(notReady != null, "Fetch was successful for all keys, but the WaitingState has not been cleared");
+ Invariants.checkState(notReady.intersects(slicedRoute), "Fetch was successful for all keys, but the WaitingState has not been cleared");
+ int nextIndex;
+ if (roundStart >= awaitRoute.size()) nextIndex = -1;
+ else if (slicedRoute == awaitRoute) nextIndex = (int) awaitRoute.findNextIntersection(roundStart, (Routables) notReady, 0);
+ else
+ {
+ Invariants.checkState(roundIndex == 0);
+ nextIndex = 0;
+ }
+
+ if (nextIndex < 0)
+ {
+ // we don't think we have anything to wait for, but we have encountered some notReady responses; queue up a retry
+ state.setAwaitDone(roundSize);
+ state.retry(safeStore, safeCommand, owner, blockedUntil);
+ }
+ else
+ {
+ Invariants.checkState(nextIndex >= roundStart);
+ roundIndex = nextIndex / roundSize;
+ state.updateAwaitRound(roundIndex, roundSize);
+ state.initialiseAwaitBitSet(awaitRoute, notReady, roundIndex, roundSize);
+ state.runInternal(safeStore, safeCommand, owner);
+ }
+ }
+ }
+ }
+ else
+ {
+ state.retry(safeStore, safeCommand, owner, blockedUntil);
+ }
+ }
+
+ static void fetchRouteCallback(SafeCommandStore safeStore, SafeCommand safeCommand, DefaultProgressLog owner, TxnId txnId, BlockedUntil blockedUntil, FetchData.FetchResult fetchResult, Throwable fail)
+ {
+ fetchCallback(FetchRoute, safeStore, safeCommand, owner, txnId, blockedUntil, fetchResult, fail);
+ }
+
+ static void fetchCallback(SafeCommandStore safeStore, SafeCommand safeCommand, DefaultProgressLog owner, TxnId txnId, BlockedUntil blockedUntil, FetchData.FetchResult fetchResult, Throwable fail)
+ {
+ fetchCallback(Fetch, safeStore, safeCommand, owner, txnId, blockedUntil, fetchResult, fail);
+ }
+
+ static void fetchCallback(CallbackKind kind, SafeCommandStore safeStore, SafeCommand safeCommand, DefaultProgressLog owner, TxnId txnId, BlockedUntil blockedUntil, FetchData.FetchResult fetchResult, Throwable fail)
+ {
+ Invariants.checkState(fetchResult != null || fail != null);
+ Unseekables<?> ready = fetchResult == null ? null : fetchResult.achievedTarget;
+ Unseekables<?> notReady = fetchResult == null ? null : fetchResult.didNotAchieveTarget;
+ BlockedUntil upgrade = fetchResult == null ? null : BlockedUntil.forSaveStatus(fetchResult.achieved.propagatesSaveStatus());
+ awaitOrFetchCallback(kind, safeStore, safeCommand, owner, txnId, blockedUntil, ready, notReady, upgrade, fail);
+ }
+
+ static void synchronousAwaitHomeCallback(SafeCommandStore safeStore, SafeCommand safeCommand, DefaultProgressLog owner, TxnId txnId, BlockedUntil blockedUntil, AsynchronousAwait.SynchronousResult awaitResult, Throwable fail)
+ {
+ synchronousAwaitCallback(AwaitHome, safeStore, safeCommand, owner, txnId, blockedUntil, awaitResult, fail);
+ }
+
+ static void synchronousAwaitSliceCallback(SafeCommandStore safeStore, SafeCommand safeCommand, DefaultProgressLog owner, TxnId txnId, BlockedUntil blockedUntil, AsynchronousAwait.SynchronousResult awaitResult, Throwable fail)
+ {
+ synchronousAwaitCallback(AwaitSlice, safeStore, safeCommand, owner, txnId, blockedUntil, awaitResult, fail);
+ }
+
+ static void synchronousAwaitCallback(CallbackKind kind, SafeCommandStore safeStore, SafeCommand safeCommand, DefaultProgressLog owner, TxnId txnId, BlockedUntil blockedUntil, AsynchronousAwait.SynchronousResult awaitResult, Throwable fail)
+ {
+ Unseekables<?> ready = awaitResult == null ? null : awaitResult.ready;
+ Unseekables<?> notReady = awaitResult == null ? null : awaitResult.notReady;
+ // TODO (desired): extract "upgrade" info from AsynchronousAwait
+ awaitOrFetchCallback(kind, safeStore, safeCommand, owner, txnId, blockedUntil, ready, notReady, null, fail);
+ }
+
+ void asynchronousAwaitCallback(DefaultProgressLog owner, SafeCommandStore safeStore, SaveStatus newStatus, Node.Id from, int callbackId)
+ {
+ if ((callbackId & 1) != 1)
+ return;
+
+ BlockedUntil blockedUntil = blockedUntil();
+ if (callbackId == AWAITING_HOME_KEY_CALLBACKID)
+ {
+ // homeKey reply
+ BlockedUntil currentHomeStatus = homeSatisfies();
+ BlockedUntil newHomeStatus = BlockedUntil.forSaveStatus(newStatus);
+ if (newHomeStatus.compareTo(currentHomeStatus) > 0)
+ setHomeSatisfies(newHomeStatus);
+
+ if (waitingProgress() != Awaiting)
+ return;
+
+ if (newHomeStatus.compareTo(blockedUntil) < 0 || currentHomeStatus.compareTo(blockedUntil) >= 0)
+ return;
+
+ SafeCommand safeCommand = safeStore.unsafeGet(txnId);
+ if (safeCommand != null)
+ runInternal(safeStore, safeCommand, owner);
+ }
+ else
+ {
+ if (waitingProgress() != Awaiting)
+ return;
+
+ callbackId >>= 1;
+ SafeCommand safeCommand = Invariants.nonNull(safeStore.unsafeGet(txnId));
+ Route<?> slicedRoute = slicedRoute(safeStore, txnId, safeCommand.current(), blockedUntil);
+
+ int roundSize = awaitRoundSize(slicedRoute);
+ int roundIndex = awaitRoundIndex(roundSize);
+ int updateBitSet = roundCallbackBitSet(owner, txnId, from, slicedRoute, callbackId, roundIndex, roundSize);
+ if (updateBitSet == 0)
+ return;
+
+ int bitSet = awaitBitSet(roundSize);
+ bitSet &= ~updateBitSet;
+ setAwaitBitSet(bitSet, roundSize);
+
+ if (bitSet == 0)
+ runInternal(safeStore, safeCommand, owner);
+ }
+ }
+
+ // TODO (expected): use back-off counter here
+ private void retry(SafeCommandStore safeStore, SafeCommand safeCommand, DefaultProgressLog owner, BlockedUntil blockedUntil)
+ {
+ if (!contactEveryone())
+ {
+ setContactEveryone(true);
+ // try again immediately with a query to all eligible replicas
+ runInternal(safeStore, safeCommand, owner);
+ }
+ else
+ {
+ // queue a retry
+ set(safeStore, owner, blockedUntil, Queued);
+ }
+ }
+
+ static void fetchRoute(DefaultProgressLog owner, BlockedUntil blockedUntil, TxnId txnId, Timestamp executeAt, EpochSupplier toLocalEpoch, Participants<?> fetchKeys)
+ {
+ // TODO (desired): fetch only the route
+ // we MUSt allocate before calling withEpoch to register cancellation, as async
+ BiConsumer<FetchData.FetchResult, Throwable> invoker = invokeWaitingCallback(owner, txnId, blockedUntil, WaitingState::fetchRouteCallback);
+ owner.node().withEpoch(blockedUntil.fetchEpoch(txnId, executeAt), invoker, () -> {
+ FetchData.fetch(blockedUntil.minSaveStatus.known, owner.node(), txnId, fetchKeys, toLocalEpoch, executeAt, invoker);
+ });
+ }
+
+ static void fetch(DefaultProgressLog owner, BlockedUntil blockedUntil, TxnId txnId, Timestamp executeAt, EpochSupplier toLocalEpoch, Route<?> slicedRoute, Route<?> fetchRoute)
+ {
+ Invariants.checkState(!slicedRoute.isEmpty());
+ // we MUSt allocate before calling withEpoch to register cancellation, as async
+ BiConsumer<FetchData.FetchResult, Throwable> invoker = invokeWaitingCallback(owner, txnId, blockedUntil, WaitingState::fetchCallback);
+ owner.node().withEpoch(blockedUntil.fetchEpoch(txnId, executeAt), invoker, () -> {
+ FetchData.fetchSpecific(blockedUntil.minSaveStatus.known, owner.node(), txnId, fetchRoute, slicedRoute, toLocalEpoch, executeAt, invoker);
+ });
+ }
+
+ void awaitHomeKey(DefaultProgressLog owner, BlockedUntil blockedUntil, TxnId txnId, Timestamp executeAt, Route<?> route)
+ {
+ // TODO (expected): special-case when this shard is home key to avoid remote messages
+ await(owner, blockedUntil, txnId, executeAt, route.homeKeyOnlyRoute(), AWAITING_HOME_KEY_CALLBACKID, WaitingState::synchronousAwaitHomeCallback);
+ }
+
+ void awaitSlice(DefaultProgressLog owner, BlockedUntil blockedUntil, TxnId txnId, Timestamp executeAt, Route<?> route, int callbackId)
+ {
+ Invariants.checkState(blockedUntil.waitsOn == SHARD);
+ // TODO (expected): special-case when this shard is home key to avoid remote messages
+ await(owner, blockedUntil, txnId, executeAt, route, callbackId, WaitingState::synchronousAwaitSliceCallback);
+ }
+
+ void await(DefaultProgressLog owner, BlockedUntil blockedUntil, TxnId txnId, Timestamp executeAt, Route<?> route, int callbackId, Callback<BlockedUntil, AsynchronousAwait.SynchronousResult> callback)
+ {
+ long epoch = blockedUntil.fetchEpoch(txnId, executeAt);
+ // we MUST allocate the invoker before invoking withEpoch as this may be asynchronous and we must first register our callback for cancellation
+ BiConsumer<AsynchronousAwait.SynchronousResult, Throwable> invoker = invokeWaitingCallback(owner, txnId, blockedUntil, callback);
+ owner.node().withEpoch(epoch, invoker, () -> {
+ AsynchronousAwait.awaitAny(owner.node(), contact(owner, route, epoch), txnId, route, blockedUntil, callbackId, invoker);
+ });
+ }
+
+ String toStateString()
+ {
+ BlockedUntil blockedUntil = blockedUntil();
+ Progress progress = waitingProgress();
+ switch (progress)
+ {
+ default:
+ throw new AssertionError("Unhandled Progress: " + progress);
+ case NoneExpected:
+ return blockedUntil == CanApply ? "Done" : "NotWaiting";
+ case Queued:
+ return "Queued(" + blockedUntil + ")";
+ case Querying:
+ return "Querying(" + blockedUntil + ")";
+ case Awaiting:
+ return "Awaiting(" + blockedUntil + ")";
+ }
+ }
+
+ boolean isWaitingDone()
+ {
+ return waitingProgress() == NoneExpected && blockedUntil() == CanApply;
+ }
+
+ enum CallbackKind
+ {
+ Fetch, FetchRoute, AwaitHome, AwaitSlice
+ }
+}
diff --git a/accord-core/src/main/java/accord/local/Bootstrap.java b/accord-core/src/main/java/accord/local/Bootstrap.java
index 9cbeb742..08bf14a2 100644
--- a/accord-core/src/main/java/accord/local/Bootstrap.java
+++ b/accord-core/src/main/java/accord/local/Bootstrap.java
@@ -171,7 +171,7 @@ class Bootstrap
if (!valid.intersects(invalidate))
return;
- valid = valid.subtract(invalidate);
+ valid = valid.without(invalidate);
abort = fetch;
// only cancel the outer future if we have no more ranges to fetch
cancel = valid.isEmpty() ? this.cancel : null;
@@ -334,7 +334,7 @@ class Bootstrap
if (newFailures.isEmpty())
return;
- valid = valid.subtract(newFailures);
+ valid = valid.without(newFailures);
hasFailed = valid.isEmpty();
}
@@ -357,7 +357,7 @@ class Bootstrap
if (state.safeToReadAt == null)
return;
- Ranges newDone = fetched.slice(state.ranges.subtract(fetchedAndSafeToRead), Minimal);
+ Ranges newDone = fetched.slice(state.ranges.without(fetchedAndSafeToRead), Minimal);
if (newDone.isEmpty())
return;
@@ -367,7 +367,7 @@ class Bootstrap
// then we are not a definitive bound for safely starting reads, so remove the range
if (overlap.startedAt > state.startedAt)
{
- newDone = newDone.subtract(overlap.ranges);
+ newDone = newDone.without(overlap.ranges);
if (newDone.isEmpty())
return;
}
@@ -413,7 +413,7 @@ class Bootstrap
// normalise fetched and fetchedAndSafeToRead against remaining valid ranges before completion
fetched = fetched.slice(valid, Minimal);
fetchedAndSafeToRead = fetchedAndSafeToRead.slice(valid, Minimal);
- retry = valid.subtract(fetchedAndSafeToRead);
+ retry = valid.without(fetchedAndSafeToRead);
completed = true;
}
@@ -472,7 +472,7 @@ class Bootstrap
Invariants.checkArgument(inProgress.contains(attempt));
Invariants.checkArgument(attempt.fetched.equals(attempt.fetchedAndSafeToRead));
inProgress.remove(attempt);
- remaining = remaining.subtract(attempt.fetched);
+ remaining = remaining.without(attempt.fetched);
if (inProgress.isEmpty() && remaining.isEmpty())
{
// TODO (now): this waits too long?
@@ -486,8 +486,8 @@ class Bootstrap
// distinct from abort as triggered by ourselves when we no longer own the range
synchronized void invalidate(Ranges invalidate)
{
- allValid = allValid.subtract(invalidate);
- remaining = remaining.subtract(invalidate);
+ allValid = allValid.without(invalidate);
+ remaining = remaining.without(invalidate);
for (Attempt attempt : inProgress)
attempt.invalidate(invalidate);
}
diff --git a/accord-core/src/main/java/accord/local/Cleanup.java b/accord-core/src/main/java/accord/local/Cleanup.java
index 36714b4f..baf96cb0 100644
--- a/accord-core/src/main/java/accord/local/Cleanup.java
+++ b/accord-core/src/main/java/accord/local/Cleanup.java
@@ -29,6 +29,7 @@ import static accord.local.RedundantBefore.NO_UPPER_BOUND;
import static accord.local.RedundantBefore.PreBootstrapOrStale.FULLY;
import static accord.local.RedundantStatus.NOT_OWNED;
import static accord.local.SaveStatus.Erased;
+import static accord.local.SaveStatus.Invalidated;
import static accord.local.SaveStatus.TruncatedApply;
import static accord.local.SaveStatus.TruncatedApplyWithOutcome;
import static accord.local.SaveStatus.Uninitialised;
@@ -48,6 +49,7 @@ public enum Cleanup
NO(Uninitialised),
TRUNCATE_WITH_OUTCOME(TruncatedApplyWithOutcome),
TRUNCATE(TruncatedApply),
+ INVALIDATE(Invalidated),
ERASE(Erased);
public final SaveStatus appliesIfNot;
@@ -98,7 +100,7 @@ public enum Cleanup
if (durableBefore.min(txnId) == UniversalOrInvalidated)
{
- if (status.hasBeen(PreCommitted) && !status.hasBeen(Applied)) // TODO (expected): may be stale
+ if (status.hasBeen(PreCommitted) && !status.hasBeen(Applied)) // TODO (expected): either stale or pre-bootstrap
illegalState("Loading universally-durable command that has been PreCommitted but not Applied");
return Cleanup.ERASE;
}
@@ -106,6 +108,8 @@ public enum Cleanup
if (!Route.isFullRoute(route))
return Cleanup.NO;
+ // TODO (required): should we apply additionalKeysOrRanges() to calculations here?
+ // TODO (required): enrich with additional epochs we know the command applies to
// We first check if the command is redundant locally, i.e. whether it has been applied to all non-faulty replicas of the local shard
// If not, we don't want to truncate its state else we may make catching up for these other replicas much harder
RedundantStatus redundant = redundantBefore.status(txnId, toEpoch, route.participants());
@@ -134,9 +138,12 @@ public enum Cleanup
return Cleanup.NO;
case SHARD_REDUNDANT:
if (enforceInvariants && status.hasBeen(PreCommitted) && !status.hasBeen(Applied) && redundantBefore.preBootstrapOrStale(txnId, toEpoch, route.participants()) != FULLY)
- illegalState("Loading redundant command that has been PreCommitted but not Applied");
+ illegalState("Loading redundant command that has been PreCommitted but not Applied.");
}
+ if (!status.hasBeen(PreCommitted))
+ return INVALIDATE;
+
Durability min = durableBefore.min(txnId, route);
switch (min)
{
@@ -145,6 +152,7 @@ public enum Cleanup
throw new AssertionError("Unexpected durability: " + min);
case NotDurable:
+ // TODO (required): disambiguate INVALIDATE and TRUNCATE_WITH_OUTCOME, and impose stronger invariants on former
if (durability == Majority)
return Cleanup.TRUNCATE;
return Cleanup.TRUNCATE_WITH_OUTCOME;
diff --git a/accord-core/src/main/java/accord/local/Command.java b/accord-core/src/main/java/accord/local/Command.java
index 89952577..47b1c51c 100644
--- a/accord-core/src/main/java/accord/local/Command.java
+++ b/accord-core/src/main/java/accord/local/Command.java
@@ -37,6 +37,7 @@ import accord.primitives.KeyDeps;
import accord.primitives.Keys;
import accord.primitives.PartialDeps;
import accord.primitives.PartialTxn;
+import accord.primitives.Participants;
import accord.primitives.RangeDeps;
import accord.primitives.Ranges;
import accord.primitives.Routable;
@@ -56,7 +57,7 @@ import accord.utils.SimpleBitSet;
import javax.annotation.Nullable;
import static accord.local.Command.AbstractCommand.validate;
-import static accord.local.Listeners.Immutable.EMPTY;
+import static accord.local.SaveStatus.AcceptedInvalidate;
import static accord.local.SaveStatus.ErasedOrInvalidOrVestigial;
import static accord.local.SaveStatus.Uninitialised;
import static accord.local.Status.Durability.Local;
@@ -77,72 +78,6 @@ import static java.lang.String.format;
public abstract class Command implements CommonAttributes
{
- public interface Listener
- {
- void onChange(SafeCommandStore safeStore, SafeCommand safeCommand);
-
- /**
- * Scope needed to run onChange
- */
- PreLoadContext listenerPreLoadContext(TxnId caller);
- }
-
- public interface DurableAndIdempotentListener extends Listener
- {
- }
-
- public interface TransientListener extends Listener
- {
- }
-
- public static class ProxyListener implements DurableAndIdempotentListener
- {
- protected final TxnId listenerId;
-
- public ProxyListener(TxnId listenerId)
- {
- this.listenerId = listenerId;
- }
-
- @Override
- public boolean equals(Object o)
- {
- if (this == o) return true;
- if (o == null || getClass() != o.getClass()) return false;
- ProxyListener that = (ProxyListener) o;
- return listenerId.equals(that.listenerId);
- }
-
- @Override
- public int hashCode()
- {
- return Objects.hash(listenerId);
- }
-
- @Override
- public String toString()
- {
- return "ListenerProxy{" + listenerId + '}';
- }
-
- public TxnId txnId()
- {
- return listenerId;
- }
-
- @Override
- public void onChange(SafeCommandStore safeStore, SafeCommand safeCommand)
- {
- Commands.listenerUpdate(safeStore, safeStore.get(listenerId), safeCommand);
- }
-
- @Override
- public PreLoadContext listenerPreLoadContext(TxnId caller)
- {
- return PreLoadContext.contextFor(listenerId, caller, Keys.EMPTY);
- }
- }
-
private static Durability durability(Durability durability, SaveStatus status)
{
if (durability == NotDurable && status.compareTo(SaveStatus.PreApplied) >= 0 && status.compareTo(ErasedOrInvalidOrVestigial) < 0)
@@ -168,6 +103,11 @@ public abstract class Command implements CommonAttributes
return Accepted.accepted(common, status, executeAt, promised, accepted);
}
+ public static AcceptedInvalidateWithoutDefinition acceptedInvalidateWithoutDefinition(CommonAttributes common, Ballot promised, Ballot accepted)
+ {
+ return AcceptedInvalidateWithoutDefinition.acceptedInvalidate(common, promised, accepted);
+ }
+
public static Committed committed(CommonAttributes common, SaveStatus status, Timestamp executeAt, Ballot promised, Ballot accepted, WaitingOn waitingOn)
{
return Committed.committed(common, status, executeAt, promised, accepted, waitingOn);
@@ -178,9 +118,9 @@ public abstract class Command implements CommonAttributes
return Executed.executed(common, status, executeAt, promised, accepted, waitingOn, writes, result);
}
- public static Truncated invalidated(TxnId txnId, Listeners.Immutable durableListeners)
+ public static Truncated invalidated(TxnId txnId)
{
- return Truncated.invalidated(txnId, durableListeners);
+ return Truncated.invalidated(txnId);
}
public static Truncated truncatedApply(CommonAttributes common, SaveStatus saveStatus, Timestamp executeAt, Writes writes, Result result)
@@ -213,9 +153,10 @@ public abstract class Command implements CommonAttributes
return validateCommandClass(status, NotDefined.class, klass);
case PreAccepted:
return validateCommandClass(status, PreAccepted.class, klass);
+ case AcceptedInvalidate:
+ return validateCommandClass(status, AcceptedInvalidateWithoutDefinition.class, klass);
case Accepted:
case AcceptedWithDefinition:
- case AcceptedInvalidate:
case AcceptedInvalidateWithDefinition:
case PreCommitted:
case PreCommittedWithAcceptedDeps:
@@ -253,9 +194,8 @@ public abstract class Command implements CommonAttributes
// TODO (expected): more consistent handling of keys for transactions that only *may* intersect a command store, as have both additionalKeysOrRanges AND forLocalEpoch to separately handle the problem
private final @Nullable Seekables<?, ?> additionalKeysOrRanges;
private final Ballot promised;
- private final Listeners.Immutable listeners;
- private AbstractCommand(TxnId txnId, SaveStatus status, Durability durability, @Nullable Route<?> route, @Nullable Seekables<?, ?> additionalKeysOrRanges, Ballot promised, Listeners.Immutable listeners)
+ private AbstractCommand(TxnId txnId, SaveStatus status, Durability durability, @Nullable Route<?> route, @Nullable Seekables<?, ?> additionalKeysOrRanges, Ballot promised)
{
this.txnId = txnId;
this.status = validateCommandClass(txnId, status, getClass());
@@ -263,7 +203,6 @@ public abstract class Command implements CommonAttributes
this.route = route;
this.additionalKeysOrRanges = additionalKeysOrRanges;
this.promised = promised;
- this.listeners = listeners;
}
private AbstractCommand(CommonAttributes common, SaveStatus status, Ballot promised)
@@ -274,10 +213,8 @@ public abstract class Command implements CommonAttributes
this.route = common.route();
this.additionalKeysOrRanges = common.additionalKeysOrRanges();
this.promised = promised;
- this.listeners = common.durableListeners();
}
-
@Override
public boolean equals(Object o)
{
@@ -289,8 +226,7 @@ public abstract class Command implements CommonAttributes
&& durability() == command.durability()
&& Objects.equals(route(), command.route())
&& Objects.equals(additionalKeysOrRanges, command.additionalKeysOrRanges())
- && Objects.equals(promised(), command.promised())
- && Objects.equals(durableListeners(), command.durableListeners());
+ && Objects.equals(promised(), command.promised());
}
@Override
@@ -323,12 +259,6 @@ public abstract class Command implements CommonAttributes
return Command.durability(durability, saveStatus());
}
- @Override
- public Listeners.Immutable durableListeners()
- {
- return listeners == null ? EMPTY : listeners;
- }
-
@Override
public final SaveStatus saveStatus()
{
@@ -463,11 +393,25 @@ public abstract class Command implements CommonAttributes
public abstract Route<?> route();
/**
- * The command may have an incomplete route when this is false
+ * We require that this is a FullRoute for all states where isDefinitionKnown().
+ * In some cases, the home shard will contain an arbitrary slice of the Route where !isDefinitionKnown(),
+ * i.e. when a non-home shard informs the home shards of a transaction to ensure forward progress.
+ *
+ * If hasBeen(Committed) this must contain the keys for both txnId.epoch and executeAt.epoch
+ *
+ * TODO (required): audit uses; do not assume non-null means it is a complete route for the shard;
+ * preferably introduce two variations so callers can declare whether they need the full shard's route
+ * or any route will do
*/
- public boolean hasFullRoute()
+ @Override
+ @Nullable
+ public abstract Participants<?> participants();
+
+ public Participants<?> maxContactable()
{
- return route() != null && route().kind().isFullRoute();
+ Route<?> route = route();
+ if (route != null) return route.withHomeKey();
+ else return participants();
}
/**
@@ -488,8 +432,6 @@ public abstract class Command implements CommonAttributes
public abstract Ballot promised();
@Override
public abstract Durability durability();
- @Override
- public abstract Listeners.Immutable<DurableAndIdempotentListener> durableListeners();
public abstract SaveStatus saveStatus();
/**
@@ -593,21 +535,11 @@ public abstract class Command implements CommonAttributes
return known.isSatisfiedBy(saveStatus().known);
}
- public boolean isAtLeast(SaveStatus.LocalExecution execution)
- {
- return saveStatus().execution.compareTo(execution) >= 0;
- }
-
public boolean is(Status status)
{
return status() == status;
}
- public final ProxyListener asListener()
- {
- return new ProxyListener(txnId());
- }
-
public final boolean isDefined()
{
if (status().hasBeen(Status.Truncated))
@@ -633,7 +565,7 @@ public abstract class Command implements CommonAttributes
public final boolean isCommitted()
{
SaveStatus saveStatus = saveStatus();
- return saveStatus.hasBeen(Status.Committed) && !saveStatus.hasBeen(Invalidated);
+ return saveStatus.hasBeen(Status.Committed) && !saveStatus.is(Invalidated);
}
public final boolean isStable()
@@ -644,7 +576,7 @@ public abstract class Command implements CommonAttributes
public static boolean isStable(SaveStatus saveStatus)
{
- return saveStatus.hasBeen(Status.Stable) && !saveStatus.hasBeen(Invalidated);
+ return saveStatus.hasBeen(Status.Stable) && !saveStatus.is(Invalidated);
}
public final Committed asCommitted()
@@ -669,16 +601,20 @@ public abstract class Command implements CommonAttributes
return updateAttributes(this, promised);
}
- public static final class NotDefined extends AbstractCommand
+ public static class NotDefined extends AbstractCommand
{
- NotDefined(TxnId txnId, SaveStatus status, Durability durability, @Nullable Route<?> route, @Nullable Seekables<?, ?> additionalKeysOrRanges, Ballot promised, Listeners.Immutable listeners)
+ final Participants<?> participants;
+ NotDefined(TxnId txnId, SaveStatus status, Durability durability, @Nullable Route<?> route, @Nullable Participants<?> participants, @Nullable Seekables<?, ?> additionalKeysOrRanges, Ballot promised)
{
- super(txnId, status, durability, route, additionalKeysOrRanges, promised, listeners);
+ super(txnId, status, durability, route, additionalKeysOrRanges, promised);
+ this.participants = participants;
+ Invariants.checkState(route == participants || route == null);
}
NotDefined(CommonAttributes common, SaveStatus status, Ballot promised)
{
super(common, status, promised);
+ participants = common.participants();
}
@Override
@@ -694,20 +630,20 @@ public abstract class Command implements CommonAttributes
public static NotDefined uninitialised(TxnId txnId)
{
- return validate(new NotDefined(txnId, Uninitialised, NotDurable, null, null, Ballot.ZERO, null));
+ return validate(new NotDefined(txnId, Uninitialised, NotDurable, null, null, null, Ballot.ZERO));
}
+ @Nullable
@Override
- public Timestamp executeAt()
+ public Participants<?> participants()
{
- return null;
+ return participants;
}
@Override
- public Ballot promised()
+ public Timestamp executeAt()
{
- // we can be preacceptedInvalidated, so can be promised even if not witnessed
- return super.promised;
+ return null;
}
@Override
@@ -754,9 +690,9 @@ public abstract class Command implements CommonAttributes
this.result = result;
}
- public Truncated(TxnId txnId, SaveStatus saveStatus, Durability durability, @Nullable Route<?> route, @Nullable Timestamp executeAt, Listeners.Immutable listeners, @Nullable Writes writes, @Nullable Result result)
+ public Truncated(TxnId txnId, SaveStatus saveStatus, Durability durability, @Nullable Route<?> route, @Nullable Timestamp executeAt, @Nullable Writes writes, @Nullable Result result)
{
- super(txnId, saveStatus, durability, route, null, Ballot.MAX, listeners);
+ super(txnId, saveStatus, durability, route, null, Ballot.MAX);
this.executeAt = executeAt;
this.writes = writes;
this.result = result;
@@ -782,7 +718,7 @@ public abstract class Command implements CommonAttributes
public static Truncated erased(TxnId txnId, Status.Durability durability, Route<?> route)
{
- return validate(new Truncated(txnId, SaveStatus.Erased, durability, route, null, EMPTY, null, null));
+ return validate(new Truncated(txnId, SaveStatus.Erased, durability, route, null, null, null));
}
public static Truncated erasedOrInvalidOrVestigial(Command command)
@@ -792,7 +728,7 @@ public abstract class Command implements CommonAttributes
public static Truncated erasedOrInvalidOrVestigial(TxnId txnId, Status.Durability durability, Route<?> route)
{
- return validate(new Truncated(txnId, SaveStatus.ErasedOrInvalidOrVestigial, durability, route, null, EMPTY, null, null));
+ return validate(new Truncated(txnId, SaveStatus.ErasedOrInvalidOrVestigial, durability, route, null, null, null));
}
public static Truncated truncatedApply(Command command)
@@ -808,24 +744,24 @@ public abstract class Command implements CommonAttributes
if (command.txnId().kind().awaitsOnlyDeps())
{
Timestamp executesAtLeast = command.hasBeen(Stable) ? command.executesAtLeast() : null;
- return validate(new TruncatedAwaitsOnlyDeps(command.txnId(), SaveStatus.TruncatedApply, durability, route, command.executeAt(), EMPTY, null, null, executesAtLeast));
+ return validate(new TruncatedAwaitsOnlyDeps(command.txnId(), SaveStatus.TruncatedApply, durability, route, command.executeAt(), null, null, executesAtLeast));
}
- return validate(new Truncated(command.txnId(), SaveStatus.TruncatedApply, durability, route, command.executeAt(), EMPTY, null, null));
+ return validate(new Truncated(command.txnId(), SaveStatus.TruncatedApply, durability, route, command.executeAt(), null, null));
}
public static Truncated truncatedApplyWithOutcome(Executed command)
{
Durability durability = Durability.mergeAtLeast(command.durability(), ShardUniversal);
if (command.txnId().kind().awaitsOnlyDeps())
- return validate(new TruncatedAwaitsOnlyDeps(command.txnId(), SaveStatus.TruncatedApplyWithOutcome, durability, command.route(), command.executeAt(), EMPTY, command.writes, command.result, command.executesAtLeast()));
- return validate(new Truncated(command.txnId(), SaveStatus.TruncatedApplyWithOutcome, durability, command.route(), command.executeAt(), EMPTY, command.writes, command.result));
+ return validate(new TruncatedAwaitsOnlyDeps(command.txnId(), SaveStatus.TruncatedApplyWithOutcome, durability, command.route(), command.executeAt(), command.writes, command.result, command.executesAtLeast()));
+ return validate(new Truncated(command.txnId(), SaveStatus.TruncatedApplyWithOutcome, durability, command.route(), command.executeAt(), command.writes, command.result));
}
public static Truncated truncatedApply(CommonAttributes common, SaveStatus saveStatus, Timestamp executeAt, Writes writes, Result result)
{
Invariants.checkArgument(!common.txnId().kind().awaitsOnlyDeps());
Durability durability = checkTruncatedApplyInvariants(common, saveStatus, executeAt);
- return validate(new Truncated(common.txnId(), saveStatus, durability, common.route(), executeAt, EMPTY, writes, result));
+ return validate(new Truncated(common.txnId(), saveStatus, durability, common.route(), executeAt, writes, result));
}
public static Truncated truncatedApply(CommonAttributes common, SaveStatus saveStatus, Timestamp executeAt, Writes writes, Result result, @Nullable Timestamp dependencyExecutesAt)
@@ -836,7 +772,7 @@ public abstract class Command implements CommonAttributes
return truncatedApply(common, saveStatus, executeAt, writes, result);
}
Durability durability = checkTruncatedApplyInvariants(common, saveStatus, executeAt);
- return validate(new TruncatedAwaitsOnlyDeps(common.txnId(), saveStatus, durability, common.route(), executeAt, EMPTY, writes, result, dependencyExecutesAt));
+ return validate(new TruncatedAwaitsOnlyDeps(common.txnId(), saveStatus, durability, common.route(), executeAt, writes, result, dependencyExecutesAt));
}
private static Durability checkTruncatedApplyInvariants(CommonAttributes common, SaveStatus saveStatus, Timestamp executeAt)
@@ -849,15 +785,21 @@ public abstract class Command implements CommonAttributes
public static Truncated invalidated(Command command)
{
Invariants.checkState(!command.hasBeen(Status.PreCommitted));
- // TODO (now): we shouldn't need to propagate these
- return invalidated(command.txnId(), command.durableListeners());
+ return invalidated(command.txnId());
}
- public static Truncated invalidated(TxnId txnId, Listeners.Immutable durableListeners)
+ public static Truncated invalidated(TxnId txnId)
{
// TODO (expected): migrate to using null for executeAt when invalidated
// TODO (expected): is UniversalOrInvalidated correct here? Should we have a lower implication pure Invalidated?
- return validate(new Truncated(txnId, SaveStatus.Invalidated, UniversalOrInvalidated, null, Timestamp.NONE, durableListeners, null, null));
+ return validate(new Truncated(txnId, SaveStatus.Invalidated, UniversalOrInvalidated, null, Timestamp.NONE, null, null));
+ }
+
+ @Nullable
+ @Override
+ public final Participants<?> participants()
+ {
+ return route();
}
@Override
@@ -906,7 +848,7 @@ public abstract class Command implements CommonAttributes
public Command updateAttributes(CommonAttributes attrs, Ballot promised)
{
// TODO (now): invoke listeners precisely once when we adopt this state, then we can simply return `this`
- return validate(new Truncated(txnId(), saveStatus(), attrs.durability(), attrs.route(), executeAt, attrs.durableListeners(), writes, result));
+ return validate(new Truncated(txnId(), saveStatus(), attrs.durability(), attrs.route(), executeAt, writes, result));
}
}
@@ -924,9 +866,9 @@ public abstract class Command implements CommonAttributes
this.executesAtLeast = executesAtLeast;
}
- public TruncatedAwaitsOnlyDeps(TxnId txnId, SaveStatus saveStatus, Durability durability, @Nullable Route<?> route, @Nullable Timestamp executeAt, Listeners.Immutable listeners, @Nullable Writes writes, @Nullable Result result, @Nullable Timestamp executesAtLeast)
+ public TruncatedAwaitsOnlyDeps(TxnId txnId, SaveStatus saveStatus, Durability durability, @Nullable Route<?> route, @Nullable Timestamp executeAt, @Nullable Writes writes, @Nullable Result result, @Nullable Timestamp executesAtLeast)
{
- super(txnId, saveStatus, durability, route, executeAt, listeners, writes, result);
+ super(txnId, saveStatus, durability, route, executeAt, writes, result);
this.executesAtLeast = executesAtLeast;
}
@@ -945,6 +887,19 @@ public abstract class Command implements CommonAttributes
public static class PreAccepted extends AbstractCommand
{
+ public static PreAccepted preAccepted(CommonAttributes common, Timestamp executeAt, Ballot promised)
+ {
+ return validate(new PreAccepted(common, SaveStatus.PreAccepted, promised, executeAt));
+ }
+
+ public static PreAccepted preAccepted(PreAccepted command, CommonAttributes common, Ballot promised)
+ {
+ checkPromised(command, promised);
+ checkSameClass(command, PreAccepted.class, "Cannot update");
+ Invariants.checkArgument(command.getClass() == PreAccepted.class);
+ return preAccepted(common, command.executeAt(), promised);
+ }
+
private final @Nullable Timestamp executeAt;
private final PartialTxn partialTxn;
private final @Nullable PartialDeps partialDeps;
@@ -962,35 +917,11 @@ public abstract class Command implements CommonAttributes
this.partialDeps = partialDeps;
}
+ @Nullable
@Override
- public Command updateAttributes(CommonAttributes attrs, Ballot promised)
+ public final Participants<?> participants()
{
- return new PreAccepted(attrs, saveStatus(), promised, executeAt());
- }
-
- @Override
- public boolean equals(Object o)
- {
- if (this == o) return true;
- if (o == null || getClass() != o.getClass()) return false;
- if (!super.equals(o)) return false;
- PreAccepted that = (PreAccepted) o;
- return Objects.equals(executeAt, that.executeAt)
- && Objects.equals(partialTxn, that.partialTxn)
- && Objects.equals(partialDeps, that.partialDeps);
- }
-
- public static PreAccepted preAccepted(CommonAttributes common, Timestamp executeAt, Ballot promised)
- {
- return validate(new PreAccepted(common, SaveStatus.PreAccepted, promised, executeAt));
- }
-
- public static PreAccepted preAccepted(PreAccepted command, CommonAttributes common, Ballot promised)
- {
- checkPromised(command, promised);
- checkSameClass(command, PreAccepted.class, "Cannot update");
- Invariants.checkArgument(command.getClass() == PreAccepted.class);
- return preAccepted(common, command.executeAt(), promised);
+ return route();
}
@Override
@@ -1024,6 +955,62 @@ public abstract class Command implements CommonAttributes
{
return partialDeps;
}
+
+ @Override
+ public Command updateAttributes(CommonAttributes attrs, Ballot promised)
+ {
+ return new PreAccepted(attrs, saveStatus(), promised, executeAt());
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+ if (this == o) return true;
+ if (o == null || getClass() != o.getClass()) return false;
+ if (!super.equals(o)) return false;
+ PreAccepted that = (PreAccepted) o;
+ return Objects.equals(executeAt, that.executeAt)
+ && Objects.equals(partialTxn, that.partialTxn)
+ && Objects.equals(partialDeps, that.partialDeps);
+ }
+ }
+
+ public static class AcceptedInvalidateWithoutDefinition extends NotDefined
+ {
+ public static AcceptedInvalidateWithoutDefinition acceptedInvalidate(CommonAttributes common, Ballot promised, Ballot accepted)
+ {
+ return new AcceptedInvalidateWithoutDefinition(common, promised, accepted);
+ }
+
+ private final Ballot accepted;
+
+ private AcceptedInvalidateWithoutDefinition(CommonAttributes common, Ballot promised, Ballot accepted)
+ {
+ super(common, SaveStatus.AcceptedInvalidate, promised);
+ this.accepted = accepted;
+ }
+
+ @Override
+ public Ballot acceptedOrCommitted()
+ {
+ return accepted;
+ }
+
+ @Override
+ public Command updateAttributes(CommonAttributes attrs, Ballot promised)
+ {
+ return new AcceptedInvalidateWithoutDefinition(attrs, promised, accepted);
+ }
+
+ @Override
+ public boolean equals(Object o)
+ {
+ if (this == o) return true;
+ if (o == null || getClass() != o.getClass()) return false;
+ if (!super.equals(o)) return false;
+ AcceptedInvalidateWithoutDefinition that = (AcceptedInvalidateWithoutDefinition) o;
+ return Objects.equals(accepted, that.accepted);
+ }
}
public static class Accepted extends PreAccepted
@@ -1461,7 +1448,7 @@ public abstract class Command implements CommonAttributes
return directRangeDeps.indexOf(txnId);
if (!CommandsForKey.managesExecution(txnId))
return directRangeDeps.txnIdCount() + directKeyDeps.indexOf(txnId);
- throw new IllegalArgumentException("WaitingOn does not track this kind of TxnId: " + txnId);
+ throw illegalArgument("WaitingOn does not track this kind of TxnId: " + txnId);
}
public int txnIdCount()
@@ -1726,18 +1713,6 @@ public abstract class Command implements CommonAttributes
}
}
- static Command addListener(Command command, DurableAndIdempotentListener listener)
- {
- CommonAttributes attrs = command.mutable().addListener(listener);
- return command.updateAttributes(attrs);
- }
-
- static Command removeListener(Command command, Listener listener)
- {
- CommonAttributes attrs = command.mutable().removeListener(listener);
- return command.updateAttributes(attrs);
- }
-
static Command.Committed updateWaitingOn(Committed command, WaitingOn.Update waitingOn)
{
if (!waitingOn.hasChanges())
@@ -1756,9 +1731,8 @@ public abstract class Command implements CommonAttributes
}
else if (command.status() == Status.AcceptedInvalidate && command.executeAt() == null)
{
- // TODO (now): reconsider this special-casing
- Command.Accepted accepted = command.asAccepted();
- return Command.Accepted.accepted(attrs, SaveStatus.enrich(accepted.saveStatus(), SaveStatus.PreAccepted.known), executeAt, ballot, accepted.acceptedOrCommitted());
+ // TODO (desired): reconsider this special-casing
+ return Command.Accepted.accepted(attrs, SaveStatus.enrich(command.saveStatus(), SaveStatus.PreAccepted.known), executeAt, ballot, command.acceptedOrCommitted());
}
else
{
@@ -1770,7 +1744,7 @@ public abstract class Command implements CommonAttributes
static Command.Accepted markDefined(Command command, CommonAttributes attributes, Ballot promised)
{
if (Command.isSameClass(command, Command.Accepted.class))
- return Command.Accepted.accepted(command.asAccepted(), attributes, SaveStatus.enrich(command.saveStatus(), Known.DefinitionOnly), promised);
+ return Command.Accepted.accepted(command.asAccepted(), attributes, SaveStatus.withDefinition(command.saveStatus()), promised);
return (Command.Accepted) command.updateAttributes(attributes, promised);
}
@@ -1779,10 +1753,12 @@ public abstract class Command implements CommonAttributes
return validate(new Command.Accepted(attrs, SaveStatus.get(Status.Accepted, command.known()), ballot, executeAt, ballot));
}
- static Command.Accepted acceptInvalidated(Command command, Ballot ballot)
+ static Command acceptInvalidated(Command command, Ballot ballot)
{
SaveStatus saveStatus = SaveStatus.get(Status.AcceptedInvalidate, command.known());
- // TODO (desired): This should be NotDefined, but AcceptedInvalidated is represented by Command.Accepted because there's no acceptedOrCommitted register in NotDefined
+ if (saveStatus == AcceptedInvalidate)
+ return validate(new Command.AcceptedInvalidateWithoutDefinition(command, ballot, ballot));
+
return validate(new Command.Accepted(command, saveStatus, ballot, command.executeAt(), command.partialTxn(), null, ballot));
}
diff --git a/accord-core/src/main/java/accord/local/CommandStore.java b/accord-core/src/main/java/accord/local/CommandStore.java
index fb97cf42..f993974b 100644
--- a/accord-core/src/main/java/accord/local/CommandStore.java
+++ b/accord-core/src/main/java/accord/local/CommandStore.java
@@ -18,10 +18,11 @@
package accord.local;
+import accord.api.LocalListeners;
import accord.api.ProgressLog;
import accord.api.DataStore;
import accord.api.VisibleForImplementationTesting;
-import accord.coordinate.CollectDeps;
+import accord.coordinate.CollectCalculatedDeps;
import accord.local.Command.WaitingOn;
import javax.annotation.Nullable;
@@ -135,6 +136,7 @@ public abstract class CommandStore implements AgentExecutor
Agent agent,
DataStore store,
ProgressLog.Factory progressLogFactory,
+ LocalListeners.Factory listenersFactory,
EpochUpdateHolder rangesForEpoch);
}
@@ -145,19 +147,18 @@ public abstract class CommandStore implements AgentExecutor
protected final Agent agent;
protected final DataStore store;
protected final ProgressLog progressLog;
+ protected final LocalListeners listeners;
protected final EpochUpdateHolder epochUpdateHolder;
// TODO (expected): schedule regular pruning of these collections
// bootstrapBeganAt and shardDurableAt are both canonical data sets mostly used for debugging / constructing
private NavigableMap<TxnId, Ranges> bootstrapBeganAt = ImmutableSortedMap.of(TxnId.NONE, Ranges.EMPTY); // additive (i.e. once inserted, rolled-over until invalidated, and the floor entry contains additions)
- // TODO (required): updates to these variables should be buffered like with Command updates in SafeCommandStore, to be rolled-back in event of a failure processing the command.
private RedundantBefore redundantBefore = RedundantBefore.EMPTY;
// TODO (expected): store this only once per node
private DurableBefore durableBefore = DurableBefore.EMPTY;
private MaxConflicts maxConflicts = MaxConflicts.EMPTY;
protected RangesForEpoch rangesForEpoch;
- // TODO (desired): merge with redundantBefore?
/**
* safeToRead is related to RedundantBefore, but a distinct concept.
* While bootstrappedAt defines the txnId bounds we expect to maintain data for locally,
@@ -171,19 +172,21 @@ public abstract class CommandStore implements AgentExecutor
* We also update safeToRead when we go stale, to remove ranges we may have bootstrapped but that are now known to
* be incomplete. In this case we permit transactions to execute in any order for the unsafe key ranges.
* But they may still be ordered for other key ranges they participate in.
+ *
+ * TODO (expected): merge with redundantBefore
*/
private NavigableMap<Timestamp, Ranges> safeToRead = ImmutableSortedMap.of(Timestamp.NONE, Ranges.EMPTY);
private final Set<Bootstrap> bootstraps = Collections.synchronizedSet(new DeterministicIdentitySet<>());
- // TODO (required): we must synchronise this on joining, prior to marking ourselves ready to coordinate
@Nullable private ReducingRangeMap<Timestamp> rejectBefore;
- protected CommandStore(int id, NodeTimeService time, Agent agent, DataStore store, ProgressLog.Factory progressLogFactory, EpochUpdateHolder epochUpdateHolder)
+ protected CommandStore(int id, NodeTimeService time, Agent agent, DataStore store, ProgressLog.Factory progressLogFactory, LocalListeners.Factory listenersFactory, EpochUpdateHolder epochUpdateHolder)
{
this.id = id;
this.time = time;
this.agent = agent;
this.store = store;
this.progressLog = progressLogFactory.create(this);
+ this.listeners = listenersFactory.create(this);
this.epochUpdateHolder = epochUpdateHolder;
}
@@ -465,7 +468,7 @@ public abstract class CommandStore implements AgentExecutor
Timestamp before = Timestamp.minForEpoch(epoch);
FullRoute<?> route = node.computeRoute(id, ranges);
// TODO (required): we need to ensure anyone we receive a reply from proposes newer timestamps for anything we don't see
- CollectDeps.withDeps(node, id, route, route, ranges, before, (deps, fail) -> {
+ CollectCalculatedDeps.withCalculatedDeps(node, id, route, route, ranges, before, (deps, fail) -> {
if (fail != null)
{
fetchMajorityDeps(coordination, node, epoch, ranges);
@@ -490,7 +493,7 @@ public abstract class CommandStore implements AgentExecutor
AsyncResult<Void> done = this.<Void>submit(empty(), safeStore -> {
for (Bootstrap prev : bootstraps)
{
- Ranges abort = prev.allValid.subtract(removedRanges);
+ Ranges abort = prev.allValid.without(removedRanges);
if (!abort.isEmpty())
prev.invalidate(abort);
}
@@ -532,6 +535,8 @@ public abstract class CommandStore implements AgentExecutor
}
// TODO (expected): we can immediately truncate dependencies locally once an exclusiveSyncPoint applies, we don't need to wait for the whole shard
+ // TODO (required): integrate validation of staleness with implementation (e.g. C* should know it has been marked stale)
+ // also: we no longer expect epochs that are losing a range to be marked stale, make sure logic reflects this
public void markShardStale(SafeCommandStore safeStore, Timestamp staleSince, Ranges ranges, boolean isSincePrecise)
{
Timestamp staleUntilAtLeast = staleSince;
@@ -617,7 +622,7 @@ public abstract class CommandStore implements AgentExecutor
Keys remaining = prev;
if (remaining == null) remaining = builder.directKeyDeps.participatingKeys(txnIdx);
else Invariants.checkState(!remaining.isEmpty());
- remaining = remaining.subtract(range);
+ remaining = remaining.without(range);
if (prev == null) Invariants.checkState(!remaining.isEmpty());
partiallyBootstrapping.put(txnIdx, remaining);
return remaining.isEmpty();
@@ -682,7 +687,7 @@ public abstract class CommandStore implements AgentExecutor
Ranges remaining = prev;
if (remaining == null) remaining = builder.directRangeDeps.ranges(rangeTxnIdx);
else Invariants.checkState(!remaining.isEmpty());
- remaining = remaining.subtract(Ranges.of(range));
+ remaining = remaining.without(Ranges.of(range));
if (prev == null) Invariants.checkState(!remaining.isEmpty());
partiallyBootstrapping.put(rangeTxnIdx, remaining);
return remaining.isEmpty();
@@ -757,7 +762,7 @@ public abstract class CommandStore implements AgentExecutor
private static <T extends Timestamp> ImmutableSortedMap<T, Ranges> purgeAndInsert(NavigableMap<T, Ranges> in, T insertAt, Ranges insert)
{
TreeMap<T, Ranges> build = new TreeMap<>(in);
- build.headMap(insertAt, false).entrySet().forEach(e -> e.setValue(e.getValue().subtract(insert)));
+ build.headMap(insertAt, false).entrySet().forEach(e -> e.setValue(e.getValue().without(insert)));
build.tailMap(insertAt, true).entrySet().forEach(e -> e.setValue(e.getValue().union(MERGE_ADJACENT, insert)));
build.entrySet().removeIf(e -> e.getKey().compareTo(Timestamp.NONE) > 0 && e.getValue().isEmpty());
Map.Entry<T, Ranges> prev = build.floorEntry(insertAt);
@@ -780,7 +785,7 @@ public abstract class CommandStore implements AgentExecutor
private static <T extends Timestamp> Map.Entry<T, Ranges> without(Map.Entry<T, Ranges> in, Ranges remove)
{
- Ranges without = in.getValue().subtract(remove);
+ Ranges without = in.getValue().without(remove);
if (without == in.getValue())
return in;
return new SimpleImmutableEntry<>(in.getKey(), without);
diff --git a/accord-core/src/main/java/accord/local/CommandStores.java b/accord-core/src/main/java/accord/local/CommandStores.java
index 231f1b05..9d6660ff 100644
--- a/accord-core/src/main/java/accord/local/CommandStores.java
+++ b/accord-core/src/main/java/accord/local/CommandStores.java
@@ -36,6 +36,7 @@ import accord.api.Agent;
import accord.api.ConfigurationService.EpochReady;
import accord.api.DataStore;
import accord.api.Key;
+import accord.api.LocalListeners;
import accord.api.ProgressLog;
import accord.api.RoutingKey;
import accord.local.CommandStore.EpochUpdateHolder;
@@ -48,7 +49,6 @@ import accord.primitives.Route;
import accord.primitives.RoutingKeys;
import accord.primitives.Timestamp;
import accord.primitives.TxnId;
-import accord.primitives.Unseekables;
import accord.topology.Topology;
import accord.utils.Invariants;
import accord.utils.MapReduce;
@@ -57,7 +57,6 @@ import accord.utils.RandomSource;
import accord.utils.async.AsyncChain;
import accord.utils.async.AsyncChains;
import javax.annotation.Nonnull;
-import javax.annotation.Nullable;
import org.agrona.collections.Hashing;
import org.agrona.collections.Int2ObjectHashMap;
@@ -67,7 +66,6 @@ import org.slf4j.LoggerFactory;
import static accord.api.ConfigurationService.EpochReady.done;
import static accord.local.PreLoadContext.empty;
-import static accord.primitives.EpochSupplier.constant;
import static accord.primitives.Routables.Slice.Minimal;
import static accord.utils.Invariants.checkArgument;
import static accord.utils.Invariants.illegalState;
@@ -84,11 +82,12 @@ public abstract class CommandStores
public interface Factory
{
CommandStores create(NodeTimeService time,
- Agent agent,
- DataStore store,
- RandomSource random,
- ShardDistributor shardDistributor,
- ProgressLog.Factory progressLogFactory);
+ Agent agent,
+ DataStore store,
+ RandomSource random,
+ ShardDistributor shardDistributor,
+ ProgressLog.Factory progressLogFactory,
+ LocalListeners.Factory listenersFactory);
}
private static class StoreSupplier
@@ -97,22 +96,24 @@ public abstract class CommandStores
private final Agent agent;
private final DataStore store;
private final ProgressLog.Factory progressLogFactory;
+ private final LocalListeners.Factory listenersFactory;
private final CommandStore.Factory shardFactory;
private final RandomSource random;
- StoreSupplier(NodeTimeService time, Agent agent, DataStore store, RandomSource random, ProgressLog.Factory progressLogFactory, CommandStore.Factory shardFactory)
+ StoreSupplier(NodeTimeService time, Agent agent, DataStore store, RandomSource random, ProgressLog.Factory progressLogFactory, LocalListeners.Factory listenersFactory, CommandStore.Factory shardFactory)
{
this.time = time;
this.agent = agent;
this.store = store;
this.random = random;
this.progressLogFactory = progressLogFactory;
+ this.listenersFactory = listenersFactory;
this.shardFactory = shardFactory;
}
CommandStore create(int id, EpochUpdateHolder rangesForEpoch)
{
- return shardFactory.create(id, time, agent, this.store, progressLogFactory, rangesForEpoch);
+ return shardFactory.create(id, time, agent, this.store, progressLogFactory, listenersFactory, rangesForEpoch);
}
}
@@ -179,7 +180,7 @@ public abstract class CommandStores
public @Nonnull Ranges unsafeToReadAt(Timestamp at)
{
- return allAt(at).subtract(store.safeToReadAt(at));
+ return allAt(at).without(store.safeToReadAt(at));
}
public @Nonnull Ranges allAt(Timestamp at)
@@ -307,19 +308,18 @@ public abstract class CommandStores
.collect(Collectors.joining(", "));
}
- public @Nullable EpochSupplier latestEpochWithNewParticipants(long sinceEpoch, Unseekables<?> keysOrRanges)
+ public long latestEpochWithNewParticipants(long sinceEpoch, Routables<?> keysOrRanges)
{
int i = floorIndex(sinceEpoch);
- long latest = -1;
+ long latest = sinceEpoch;
Ranges existing = i < 0 ? Ranges.EMPTY : ranges[i];
while (++i < ranges.length)
{
- if (ranges[i].subtract(existing).intersects(keysOrRanges))
+ if (ranges[i].without(existing).intersects(keysOrRanges))
latest = epochs[i];
existing = existing.with(ranges[i]);
}
-
- return latest == -1 ? null : constant(latest);
+ return latest;
}
public Ranges removed(long presentIn, long removedByInclusive)
@@ -329,7 +329,7 @@ public abstract class CommandStores
Ranges removed = Ranges.EMPTY;
while (i < maxi)
{
- removed = removed.with(ranges[i - 1].subtract(ranges[i]));
+ removed = removed.with(ranges[i - 1].without(ranges[i]));
++i;
}
return removed;
@@ -367,9 +367,9 @@ public abstract class CommandStores
}
public CommandStores(NodeTimeService time, Agent agent, DataStore store, RandomSource random, ShardDistributor shardDistributor,
- ProgressLog.Factory progressLogFactory, CommandStore.Factory shardFactory)
+ ProgressLog.Factory progressLogFactory, LocalListeners.Factory listenersFactory, CommandStore.Factory shardFactory)
{
- this(new StoreSupplier(time, agent, store, random, progressLogFactory, shardFactory), shardDistributor);
+ this(new StoreSupplier(time, agent, store, random, progressLogFactory, listenersFactory, shardFactory), shardDistributor);
}
public Topology local()
@@ -408,7 +408,7 @@ public abstract class CommandStores
return new TopologyUpdate(prev, () -> done(epoch));
Topology newLocalTopology = newTopology.forNode(supplier.time.id()).trim();
- Ranges addedGlobal = newTopology.ranges().subtract(prev.global.ranges());
+ Ranges addedGlobal = newTopology.ranges().without(prev.global.ranges());
if (!addedGlobal.isEmpty())
{
for (ShardHolder shard : prev.shards)
@@ -417,8 +417,8 @@ public abstract class CommandStores
}
}
- Ranges added = newLocalTopology.ranges().subtract(prev.local.ranges());
- Ranges subtracted = prev.local.ranges().subtract(newLocalTopology.ranges());
+ Ranges added = newLocalTopology.ranges().without(prev.local.ranges());
+ Ranges subtracted = prev.local.ranges().without(newLocalTopology.ranges());
if (added.isEmpty() && subtracted.isEmpty())
{
Supplier<EpochReady> epochReady = () -> done(epoch);
@@ -438,7 +438,7 @@ public abstract class CommandStores
if (!removeRanges.isEmpty())
{
// TODO (required): This is updating the a non-volatile field in the previous Snapshot, why modify it at all, even with volatile the guaranteed visibility is weak even with mutual exclusion
- shard.ranges = shard.ranges().withRanges(newTopology.epoch(), current.subtract(subtracted));
+ shard.ranges = shard.ranges().withRanges(newTopology.epoch(), current.without(subtracted));
shard.store.epochUpdateHolder.remove(epoch, shard.ranges, removeRanges);
bootstrapUpdates.add(shard.store.unbootstrap(epoch, removeRanges));
}
diff --git a/accord-core/src/main/java/accord/local/Commands.java b/accord-core/src/main/java/accord/local/Commands.java
index bb8c8ba4..43b8d77f 100644
--- a/accord-core/src/main/java/accord/local/Commands.java
+++ b/accord-core/src/main/java/accord/local/Commands.java
@@ -28,18 +28,14 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import accord.api.Key;
-import accord.api.ProgressLog.ProgressShard;
import accord.api.Result;
-import accord.api.RoutingKey;
import accord.coordinate.Infer;
-import accord.local.Command.ProxyListener;
import accord.local.Command.WaitingOn;
import accord.primitives.Ballot;
import accord.primitives.Deps;
import accord.primitives.EpochSupplier;
import accord.primitives.FullRoute;
import accord.primitives.PartialDeps;
-import accord.primitives.PartialRoute;
import accord.primitives.PartialTxn;
import accord.primitives.Participants;
import accord.primitives.Ranges;
@@ -54,12 +50,12 @@ import accord.utils.Invariants;
import accord.utils.async.AsyncChain;
import accord.utils.async.AsyncChains;
-import static accord.api.ProgressLog.ProgressShard.Home;
-import static accord.api.ProgressLog.ProgressShard.Local;
-import static accord.api.ProgressLog.ProgressShard.No;
+import static accord.api.ProgressLog.BlockedUntil.CanApply;
+import static accord.api.ProgressLog.BlockedUntil.HasDecidedExecuteAt;
import static accord.local.Cleanup.ERASE;
import static accord.local.Cleanup.shouldCleanup;
import static accord.local.Command.Truncated.erased;
+import static accord.local.Command.Truncated.invalidated;
import static accord.local.Command.Truncated.truncatedApply;
import static accord.local.Command.Truncated.truncatedApplyWithOutcome;
import static accord.local.KeyHistory.TIMESTAMPS;
@@ -67,8 +63,6 @@ import static accord.local.PreLoadContext.contextFor;
import static accord.local.RedundantStatus.PRE_BOOTSTRAP_OR_STALE;
import static accord.local.SaveStatus.Applying;
import static accord.local.SaveStatus.Erased;
-import static accord.local.SaveStatus.LocalExecution.ReadyToExclude;
-import static accord.local.SaveStatus.LocalExecution.WaitingToApply;
import static accord.local.SaveStatus.TruncatedApply;
import static accord.local.Status.Applied;
import static accord.local.Status.Committed;
@@ -77,7 +71,6 @@ import static accord.local.Status.Invalidated;
import static accord.local.Status.KnownExecuteAt.ExecuteAtKnown;
import static accord.local.Status.KnownRoute.Full;
import static accord.local.Status.NotDefined;
-import static accord.local.Status.PreAccepted;
import static accord.local.Status.PreApplied;
import static accord.local.Status.PreCommitted;
import static accord.local.Status.Stable;
@@ -95,35 +88,22 @@ public class Commands
{
}
- private static boolean hasQuery(PartialTxn txn)
- {
- return txn != null && txn.query() != null;
- }
-
- /**
- * true iff this commandStore owns the given key on the given epoch
- */
- public static boolean owns(SafeCommandStore safeStore, long epoch, RoutingKey someKey)
- {
- return safeStore.ranges().allAt(epoch).contains(someKey);
- }
-
public enum AcceptOutcome { Success, Redundant, RejectedBallot, Truncated }
- public static AcceptOutcome preaccept(SafeCommandStore safeStore, SafeCommand safeCommand, TxnId txnId, long acceptEpoch, PartialTxn partialTxn, FullRoute<?> route, @Nullable RoutingKey progressKey)
+ public static AcceptOutcome preaccept(SafeCommandStore safeStore, SafeCommand safeCommand, TxnId txnId, long acceptEpoch, PartialTxn partialTxn, FullRoute<?> route)
{
- return preacceptOrRecover(safeStore, safeCommand, txnId, acceptEpoch, partialTxn, route, progressKey, Ballot.ZERO);
+ return preacceptOrRecover(safeStore, safeCommand, txnId, acceptEpoch, partialTxn, route, Ballot.ZERO);
}
- public static AcceptOutcome recover(SafeCommandStore safeStore, SafeCommand safeCommand, TxnId txnId, PartialTxn partialTxn, FullRoute<?> route, @Nullable RoutingKey progressKey, Ballot ballot)
+ public static AcceptOutcome recover(SafeCommandStore safeStore, SafeCommand safeCommand, TxnId txnId, PartialTxn partialTxn, FullRoute<?> route, Ballot ballot)
{
// for recovery we only ever propose either the original epoch or an Accept that we witness; otherwise we invalidate
- return preacceptOrRecover(safeStore, safeCommand, txnId, txnId.epoch(), partialTxn, route, progressKey, ballot);
+ return preacceptOrRecover(safeStore, safeCommand, txnId, txnId.epoch(), partialTxn, route, ballot);
}
- private static AcceptOutcome preacceptOrRecover(SafeCommandStore safeStore, SafeCommand safeCommand, TxnId txnId, long acceptEpoch, PartialTxn partialTxn, FullRoute<?> route, @Nullable RoutingKey progressKey, Ballot ballot)
+ private static AcceptOutcome preacceptOrRecover(SafeCommandStore safeStore, SafeCommand safeCommand, TxnId txnId, long acceptEpoch, PartialTxn partialTxn, FullRoute<?> route, Ballot ballot)
{
- Command command = safeCommand.current();
+ final Command command = safeCommand.current();
if (command.hasBeen(Truncated))
{
@@ -151,7 +131,6 @@ public class Commands
Ranges coordinateRanges = coordinateRanges(safeStore, txnId);
Ranges acceptRanges = acceptRanges(safeStore, txnId, acceptEpoch, coordinateRanges);
Invariants.checkState(!acceptRanges.isEmpty());
- ProgressShard shard = progressShard(route, progressKey, coordinateRanges);
Invariants.checkState(validate(SaveStatus.PreAccepted, command, acceptRanges, route, partialTxn, null, null));
// FIXME: this should go into a consumer method
@@ -165,8 +144,7 @@ public class Commands
// invalidate any transactions that were not completed by their initial coordinator
// TODO (desired): limit preaccept to keys we include, to avoid inflating unnecessary state
Timestamp executeAt = safeStore.commandStore().preaccept(txnId, attrs.partialTxn().keys(), safeStore, ballot.equals(Ballot.ZERO));
- command = safeCommand.preaccept(safeStore, attrs, executeAt, ballot);
- safeStore.progressLog().preaccepted(command, shard);
+ safeCommand.preaccept(safeStore, attrs, executeAt, ballot);
}
else
{
@@ -174,7 +152,7 @@ public class Commands
safeCommand.markDefined(safeStore, attrs, ballot);
}
- safeStore.notifyListeners(safeCommand);
+ safeStore.notifyListeners(safeCommand, command);
return AcceptOutcome.Success;
}
@@ -199,10 +177,10 @@ public class Commands
return true;
}
- public static AcceptOutcome accept(SafeCommandStore safeStore, TxnId txnId, Ballot ballot, PartialRoute<?> route, Seekables<?, ?> keysOrRanges, @Nullable RoutingKey progressKey, Timestamp executeAt, PartialDeps partialDeps)
+ public static AcceptOutcome accept(SafeCommandStore safeStore, TxnId txnId, Ballot ballot, Route<?> route, Seekables<?, ?> keysOrRanges, Timestamp executeAt, PartialDeps partialDeps)
{
SafeCommand safeCommand = safeStore.get(txnId, executeAt, route);
- Command command = safeCommand.current();
+ final Command command = safeCommand.current();
if (command.hasBeen(PreCommitted))
{
if (command.is(Invalidated))
@@ -231,7 +209,6 @@ public class Commands
Ranges acceptRanges = acceptRanges(safeStore, txnId, executeAt, coordinateRanges);
Invariants.checkState(!acceptRanges.isEmpty());
- ProgressShard shard = progressShard(route, progressKey, coordinateRanges);
Invariants.checkState(validate(SaveStatus.Accepted, command, acceptRanges, route, null, partialDeps, null));
// TODO (desired, clarity/efficiency): we don't need to set the route here, and perhaps we don't even need to
@@ -240,9 +217,8 @@ public class Commands
CommonAttributes attrs = set(SaveStatus.Accepted, command, command, acceptRanges, ballot, route, null, partialDeps);
keysOrRanges = keysOrRanges.slice(acceptRanges);
- command = safeCommand.accept(safeStore, keysOrRanges, attrs, executeAt, ballot);
- safeStore.progressLog().accepted(command, shard);
- safeStore.notifyListeners(safeCommand);
+ safeCommand.accept(safeStore, keysOrRanges, attrs, executeAt, ballot);
+ safeStore.notifyListeners(safeCommand, command);
return AcceptOutcome.Success;
}
@@ -250,7 +226,7 @@ public class Commands
public static AcceptOutcome acceptInvalidate(SafeCommandStore safeStore, SafeCommand safeCommand, Ballot ballot)
{
// TODO (expected): save some partial route we can use to determine if it can be GC'd
- Command command = safeCommand.current();
+ final Command command = safeCommand.current();
if (command.hasBeen(PreCommitted))
{
if (command.is(Invalidated))
@@ -278,7 +254,7 @@ public class Commands
logger.trace("{}: accepted invalidated", command.txnId());
safeCommand.acceptInvalidated(safeStore, ballot);
- safeStore.notifyListeners(safeCommand);
+ safeStore.notifyListeners(safeCommand, command);
return AcceptOutcome.Success;
}
@@ -286,9 +262,9 @@ public class Commands
// relies on mutual exclusion for each key
- public static CommitOutcome commit(SafeCommandStore safeStore, SafeCommand safeCommand, SaveStatus newStatus, Ballot ballot, TxnId txnId, Route<?> route, @Nullable RoutingKey progressKey, @Nullable PartialTxn partialTxn, Timestamp executeAt, PartialDeps partialDeps)
+ public static CommitOutcome commit(SafeCommandStore safeStore, SafeCommand safeCommand, SaveStatus newStatus, Ballot ballot, TxnId txnId, Route<?> route, @Nullable PartialTxn partialTxn, Timestamp executeAt, PartialDeps partialDeps)
{
- Command command = safeCommand.current();
+ final Command command = safeCommand.current();
SaveStatus curStatus = command.saveStatus();
Invariants.checkArgument(newStatus == SaveStatus.Committed || newStatus == SaveStatus.Stable);
@@ -320,7 +296,6 @@ public class Commands
Ranges coordinateRanges = coordinateRanges(safeStore, txnId);
Ranges acceptRanges = acceptRanges(safeStore, txnId, executeAt, coordinateRanges);
- ProgressShard shard = progressShard(route, progressKey, coordinateRanges);
if (!validate(newStatus, command, acceptRanges, route, partialTxn, partialDeps, null))
return CommitOutcome.Insufficient;
@@ -328,22 +303,21 @@ public class Commands
CommonAttributes attrs = set(newStatus, command, command, acceptRanges, ballot, route, partialTxn, partialDeps);
logger.trace("{}: committed with executeAt: {}, deps: {}", txnId, executeAt, partialDeps);
+ final Command.Committed committed;
if (newStatus == SaveStatus.Stable)
{
- safeStore.progressLog().stable(command, shard);
WaitingOn waitingOn = initialiseWaitingOn(safeStore, txnId, attrs, executeAt, attrs.route());
- command = safeCommand.stable(safeStore, attrs, Ballot.max(command.acceptedOrCommitted(), ballot), executeAt, waitingOn);
- safeStore.agent().metricsEventsListener().onStable(command);
+ committed = safeCommand.stable(safeStore, attrs, Ballot.max(command.acceptedOrCommitted(), ballot), executeAt, waitingOn);
+ safeStore.agent().metricsEventsListener().onStable(committed);
// TODO (expected, safety): introduce intermediate status to avoid reentry when notifying listeners (which might notify us)
maybeExecute(safeStore, safeCommand, true, true);
}
else
{
- safeStore.progressLog().precommitted(command);
Invariants.checkArgument(command.acceptedOrCommitted().compareTo(ballot) <= 0);
- command = safeCommand.commit(safeStore, attrs, ballot, executeAt);
- safeStore.notifyListeners(safeCommand);
- safeStore.agent().metricsEventsListener().onCommitted(command);
+ committed = safeCommand.commit(safeStore, attrs, ballot, executeAt);
+ safeStore.notifyListeners(safeCommand, committed);
+ safeStore.agent().metricsEventsListener().onCommitted(committed);
}
return CommitOutcome.Success;
@@ -352,7 +326,8 @@ public class Commands
// relies on mutual exclusion for each key
public static CommitOutcome precommit(SafeCommandStore safeStore, SafeCommand safeCommand, TxnId txnId, Timestamp executeAt, Route<?> route)
{
- Command command = safeCommand.current();
+ Invariants.checkState(Route.isFullRoute(route));
+ final Command command = safeCommand.current();
if (command.hasBeen(PreCommitted))
{
if (command.is(Truncated))
@@ -371,12 +346,11 @@ public class Commands
}
CommonAttributes attrs = command;
- if (command.route() == null || !command.route().kind().isFullRoute())
+ if (command.route() == null || !Route.isFullRoute(command.route()))
attrs = updateRoute(command, route);
safeCommand.precommit(safeStore, attrs, executeAt);
- safeStore.progressLog().precommitted(command);
- safeStore.notifyListeners(safeCommand);
+ safeStore.notifyListeners(safeCommand, command);
logger.trace("{}: precommitted with executeAt: {}", txnId, executeAt);
return CommitOutcome.Success;
}
@@ -384,7 +358,7 @@ public class Commands
public static void createBootstrapCompleteMarkerTransaction(SafeCommandStore safeStore, TxnId localSyncId, Seekables<?, ?> keys)
{
SafeCommand safeCommand = safeStore.get(localSyncId);
- Command command = safeCommand.current();
+ final Command command = safeCommand.current();
Invariants.checkState(!command.hasBeen(Committed));
FullRoute<?> route = keys.toRoute(keys.get(0).someIntersectingRoutingKey(null));
@@ -396,7 +370,7 @@ public class Commands
Invariants.checkState(validate(SaveStatus.Stable, command, coordinateRanges, route, partialTxn, none, null));
CommonAttributes newAttributes = set(SaveStatus.Stable, command, command, coordinateRanges, Ballot.ZERO, route, partialTxn, none);
safeCommand.stable(safeStore, newAttributes, Ballot.ZERO, localSyncId, WaitingOn.empty(emptyTxn.keys().domain()));
- safeStore.notifyListeners(safeCommand);
+ safeStore.notifyListeners(safeCommand, command);
}
public static void ephemeralRead(SafeCommandStore safeStore, SafeCommand safeCommand, Route<?> route, TxnId txnId, PartialTxn partialTxn, PartialDeps partialDeps, long executeAtEpoch)
@@ -433,7 +407,7 @@ public class Commands
// TODO (expected, ?): commitInvalidate may need to update cfks _if_ possible
public static void commitInvalidate(SafeCommandStore safeStore, SafeCommand safeCommand, Unseekables<?> scope)
{
- Command command = safeCommand.current();
+ final Command command = safeCommand.current();
if (command.hasBeen(PreCommitted))
{
if (command.is(Truncated))
@@ -454,12 +428,12 @@ public class Commands
safeCommand.commitInvalidated(safeStore);
safeStore.progressLog().clear(command.txnId());
logger.trace("{}: committed invalidated", safeCommand.txnId());
- safeStore.notifyListeners(safeCommand, command, command.durableListeners(), safeCommand.transientListeners());
+ safeStore.notifyListeners(safeCommand, command);
}
public enum ApplyOutcome { Success, Redundant, Insufficient }
- public static ApplyOutcome apply(SafeCommandStore safeStore, SafeCommand safeCommand, TxnId txnId, Route<?> route, @Nullable RoutingKey progressKey, Timestamp executeAt, @Nullable PartialDeps partialDeps, @Nullable PartialTxn partialTxn, Writes writes, Result result)
+ public static ApplyOutcome apply(SafeCommandStore safeStore, SafeCommand safeCommand, TxnId txnId, Route<?> route, Timestamp executeAt, @Nullable PartialDeps partialDeps, @Nullable PartialTxn partialTxn, Writes writes, Result result)
{
Command command = safeCommand.current();
if (command.hasBeen(PreApplied) && executeAt.equals(command.executeAt()))
@@ -476,7 +450,6 @@ public class Commands
Ranges coordinateRanges = coordinateRanges(safeStore, txnId);
Ranges acceptRanges = acceptRanges(safeStore, txnId, executeAt, coordinateRanges);
- ProgressShard shard = progressShard(route, progressKey, coordinateRanges);
if (!validate(SaveStatus.PreApplied, command, acceptRanges, route, partialTxn, partialDeps, safeStore))
return ApplyOutcome.Insufficient; // TODO (expected, consider): this should probably be an assertion failure if !TrySet
@@ -484,11 +457,12 @@ public class Commands
CommonAttributes attrs = set(SaveStatus.PreApplied, command, command, acceptRanges, null, route, partialTxn, partialDeps);
WaitingOn waitingOn = !command.hasBeen(Stable) ? initialiseWaitingOn(safeStore, txnId, attrs, executeAt, attrs.route()) : command.asCommitted().waitingOn();
+
safeCommand.preapplied(safeStore, attrs, executeAt, waitingOn, writes, result);
logger.trace("{}: apply, status set to Executed with executeAt: {}, deps: {}", txnId, executeAt, partialDeps);
+ // must signal preapplied first, else we may be applied (and have cleared progress log state) already before maybeExecute exits
maybeExecute(safeStore, safeCommand, true, true);
- safeStore.progressLog().executed(safeCommand.current(), shard);
safeStore.agent().metricsEventsListener().onExecuted(command);
return ApplyOutcome.Success;
@@ -504,7 +478,6 @@ public class Commands
// TODO (desired): would be nice to ensure these are deregistered explicitly, but would be costly
Invariants.checkState(listener.saveStatus().isUninitialised() || listener.is(Truncated), "Listener status expected to be Uninitialised or Truncated, but was %s", listener.saveStatus());
Invariants.checkState(updated.is(NotDefined) || updated.hasBeen(Truncated) || !updated.asCommitted().waitingOn().isWaitingOn(listener.txnId()), "Updated status expected to be Applied or NotDefined, but was %s", updated);
- safeUpdated.removeListener(listener.asListener());
return;
}
@@ -536,8 +509,9 @@ public class Commands
{
logger.trace("{} applied, setting status to Applied and notifying listeners", txnId);
SafeCommand safeCommand = safeStore.get(txnId);
+ final Command original = safeCommand.current();
safeCommand.applied(safeStore);
- safeStore.notifyListeners(safeCommand);
+ safeStore.notifyListeners(safeCommand, original);
}
/**
@@ -616,14 +590,14 @@ public class Commands
private static boolean maybeExecute(SafeCommandStore safeStore, SafeCommand safeCommand, boolean alwaysNotifyListeners, boolean notifyWaitingOn)
{
- Command command = safeCommand.current();
+ final Command command = safeCommand.current();
if (logger.isTraceEnabled())
logger.trace("{}: Maybe executing with status {}. Will notify listeners on noop: {}", command.txnId(), command.status(), alwaysNotifyListeners);
if (command.status() != Stable && command.status() != PreApplied)
{
if (alwaysNotifyListeners)
- safeStore.notifyListeners(safeCommand);
+ safeStore.notifyListeners(safeCommand, command);
return false;
}
@@ -631,7 +605,7 @@ public class Commands
if (waitingOn.isWaiting())
{
if (alwaysNotifyListeners)
- safeStore.notifyListeners(safeCommand);
+ safeStore.notifyListeners(safeCommand, command);
if (notifyWaitingOn && waitingOn.isWaitingOnCommand())
new NotifyWaitingOn(safeCommand).accept(safeStore);
@@ -645,10 +619,12 @@ public class Commands
{
case Stable:
// TODO (desirable, efficiency): maintain distinct ReadyToRead and ReadyToWrite states
- command = safeCommand.readyToExecute(safeStore);
+ // TODO (required): we can have dangling transactions in some cases when proposing in a future epoch but
+ // later deciding on an earlier epoch. We should probably turn this into an erased vestigial command,
+ // but we should tighten up our semantics there in general.
+ safeCommand.readyToExecute(safeStore);
logger.trace("{}: set to ReadyToExecute", command.txnId());
- safeStore.progressLog().readyToExecute(command);
- safeStore.notifyListeners(safeCommand);
+ safeStore.notifyListeners(safeCommand, command);
return true;
case PreApplied:
@@ -660,7 +636,7 @@ public class Commands
{
// TODO (now): we should set applying within apply to avoid applying multiple times
safeCommand.applying(safeStore);
- safeStore.notifyListeners(safeCommand);
+ safeStore.notifyListeners(safeCommand, command);
logger.trace("{}: applying", command.txnId());
apply(safeStore, executed);
return true;
@@ -677,7 +653,7 @@ public class Commands
ranges = command.route().slice(ranges, Minimal).participants().toRanges();
safeStore.commandStore().markExclusiveSyncPointLocallyApplied(safeStore, command.txnId(), ranges);
}
- safeStore.notifyListeners(safeCommand);
+ safeStore.notifyListeners(safeCommand, command);
return true;
}
default:
@@ -743,24 +719,22 @@ public class Commands
case Invalidated:
logger.trace("{}: {} is invalidated. Stop listening and removing from waiting on commit set.", waitingId, dependencyId);
}
- dependencySafeCommand.removeListener(new ProxyListener(waitingId));
return waitingOn.setAppliedOrInvalidated(dependencyId);
}
else if (dependency.executeAt().compareTo(waitingExecuteAt) > 0 && !waitingId.kind().awaitsOnlyDeps())
{
// dependency cannot be a predecessor if it executes later
- logger.trace("{}: {} executes after us. Stop listening and removing from waiting on apply set.", waitingId, dependencyId);
- dependencySafeCommand.removeListener(new ProxyListener(waitingId));
+ logger.trace("{}: {} executes after us. Removing from waiting on apply set.", waitingId, dependencyId);
return waitingOn.removeWaitingOn(dependencyId);
}
else if (dependency.hasBeen(Applied))
{
- logger.trace("{}: {} has been applied. Stop listening and removing from waiting on apply set.", waitingId, dependencyId);
- dependencySafeCommand.removeListener(new ProxyListener(waitingId));
+ logger.trace("{}: {} has been applied. Removing from waiting on apply set.", waitingId, dependencyId);
return waitingOn.setAppliedAndPropagate(dependencyId, dependency.asCommitted().waitingOn());
}
else if (waitingOn.isWaitingOn(dependencyId))
{
+ safeStore.registerListener(dependencySafeCommand, SaveStatus.Applied, Full, waitingId);
return false;
}
else if (safeStore.isFullyPreBootstrapOrStale(dependency, waiting.partialDeps().participants(dependency.txnId())))
@@ -795,8 +769,8 @@ public class Commands
if (pred.hasBeen(PreCommitted))
{
TxnId nextWaitingOn = command.waitingOn().nextWaitingOn();
- if (nextWaitingOn != null && nextWaitingOn.equals(pred.txnId()))
- safeStore.progressLog().waiting(predecessor, WaitingToApply, pred.route(), null);
+ if (nextWaitingOn != null && nextWaitingOn.equals(pred.txnId()) && !pred.hasBeen(PreApplied))
+ safeStore.progressLog().waiting(CanApply, safeStore, predecessor, pred.route(), null);
}
}
}
@@ -858,9 +832,7 @@ public class Commands
public static void setErased(SafeCommandStore safeStore, SafeCommand safeCommand)
{
- Listeners.Immutable durableListeners = safeCommand.current().durableListeners();
- Command command = purge(safeStore, safeCommand, null, null, ERASE, true);
- safeStore.notifyListeners(safeCommand, command, durableListeners, safeCommand.transientListeners());
+ purge(safeStore, safeCommand, null, null, ERASE, true);
}
/**
@@ -868,7 +840,7 @@ public class Commands
*/
public static Command purge(SafeCommandStore safeStore, SafeCommand safeCommand, @Nullable Seekables<?, ?> keysOrRanges, @Nullable Unseekables<?> maybeFullRoute, Cleanup cleanup, boolean notifyListeners)
{
- Command command = safeCommand.current();
+ final Command command = safeCommand.current();
// 1) a command has been applied; or
// 2) has been coordinated but *will not* be applied (we just haven't witnessed the invalidation yet); or
@@ -883,21 +855,21 @@ public class Commands
switch (cleanup)
{
default: throw new AssertionError("Unexpected cleanup: " + cleanup);
+ case INVALIDATE:
+ Invariants.checkArgument(!command.hasBeen(PreCommitted));
+ result = invalidated(command);
+ break;
+
case TRUNCATE_WITH_OUTCOME:
Invariants.checkArgument(!command.hasBeen(Truncated));
- if (command.hasBeen(PreApplied))
- {
- result = truncatedApplyWithOutcome(command.asExecuted());
- break;
- }
- // TODO (expected, consider): should we downgrade to no truncation in this case? Or are we stale?
+ Invariants.checkState(command.hasBeen(PreApplied));
+ result = truncatedApplyWithOutcome(command.asExecuted());
+ break;
case TRUNCATE:
- // TODO (expected): consider passing through any information we have about the reason for loading, so we can infer APPLIED if !PreCommitted
Invariants.checkState(command.saveStatus().compareTo(TruncatedApply) < 0);
- if (!command.hasBeen(PreCommitted)) result = Command.Truncated.erasedOrInvalidOrVestigial(command);
- else result = truncatedApply(command, Route.tryCastToFullRoute(maybeFullRoute));
- safeStore.progressLog().clear(command.txnId());
+ Invariants.checkState(command.hasBeen(PreApplied));
+ result = truncatedApply(command, Route.tryCastToFullRoute(maybeFullRoute));
break;
case ERASE:
@@ -909,7 +881,7 @@ public class Commands
safeCommand.update(safeStore, result);
safeStore.progressLog().clear(safeCommand.txnId());
if (notifyListeners)
- safeStore.notifyListeners(safeCommand, result, command.durableListeners(), safeCommand.transientListeners());
+ safeStore.notifyListeners(safeCommand, command);
return result;
}
@@ -931,7 +903,7 @@ public class Commands
public static Command setDurability(SafeCommandStore safeStore, SafeCommand safeCommand, Durability durability, @Nullable Route<?> route, @Nullable Timestamp executeAt, @Nullable EpochSupplier toEpoch)
{
- Command command = safeCommand.current();
+ final Command command = safeCommand.current();
if (command.is(Truncated))
return command;
@@ -942,19 +914,13 @@ public class Commands
if (executeAt != null && command.status().hasBeen(Committed) && !command.executeAt().equals(executeAt))
safeStore.agent().onInconsistentTimestamp(command, command.asCommitted().executeAt(), executeAt);
attrs = attrs.mutable().durability(durability);
- command = safeCommand.updateAttributes(attrs);
+ Command updated = safeCommand.updateAttributes(safeStore, attrs);
if (maybeCleanup(safeStore, safeCommand, command, toEpoch, route))
- return safeCommand.current();
+ updated = safeCommand.current();
- safeStore.progressLog().durable(command);
- safeStore.notifyListeners(safeCommand);
- return command;
- }
-
- public static Command setDurability(SafeCommandStore safeStore, SafeCommand safeCommand, Durability durability)
- {
- return setDurability(safeStore, safeCommand, durability, null, null);
+ safeStore.notifyListeners(safeCommand, command);
+ return updated;
}
static class NotifyWaitingOn implements PreLoadContext, Consumer<SafeCommandStore>
@@ -1066,8 +1032,8 @@ public class Commands
case LIVE:
case PARTIALLY_PRE_BOOTSTRAP_OR_STALE:
if (logger.isTraceEnabled()) logger.trace("{} blocked on {} until ReadyToExclude", waitingId, dep.txnId());
- depSafe.addListener(waiting.asListener());
- safeStore.progressLog().waiting(depSafe, ReadyToExclude, null, participants);
+ safeStore.registerListener(depSafe, HasDecidedExecuteAt.minSaveStatus, Full, waitingId);
+ safeStore.progressLog().waiting(HasDecidedExecuteAt, safeStore, depSafe, null, participants);
return;
case LOCALLY_REDUNDANT:
@@ -1089,19 +1055,23 @@ public class Commands
case CleaningUp: throw illegalState("Invalid LocalExecution (should already be handled): " + depStatus.execution);
case NotReady:
+ safeStore.registerListener(depSafe, HasDecidedExecuteAt.minSaveStatus, Full, waitingId);
+ safeStore.progressLog().waiting(HasDecidedExecuteAt, safeStore, depSafe, dep.route(), null);
+ return;
+
case ReadyToExclude:
case WaitingToExecute:
case ReadyToExecute:
- safeStore.progressLog().waiting(depSafe, WaitingToApply, dep.route(), null);
+ safeStore.progressLog().waiting(CanApply, safeStore, depSafe, dep.route(), null);
case Applying:
- depSafe.addListener(waiting.asListener());
+ safeStore.registerListener(depSafe, SaveStatus.Applied, Full, waitingId);
return;
case WaitingToApply:
if (dep.asCommitted().isWaitingOnDependency())
{
- depSafe.addListener(waiting.asListener());
+ safeStore.registerListener(depSafe, SaveStatus.Applied, Full, waitingId);
return;
}
else
@@ -1111,7 +1081,7 @@ public class Commands
{
default: throw illegalState("Invalid child status after attempt to execute: " + depSafe.current().saveStatus());
case Applying:
- depSafe.addListener(waiting.asListener());
+ safeStore.registerListener(depSafe, SaveStatus.Applied, Full, waitingId);
return;
case Applied:
@@ -1157,17 +1127,16 @@ public class Commands
}
@SuppressWarnings({"unchecked", "rawtypes"})
- public static Command informHome(SafeCommandStore safeStore, SafeCommand safeCommand, Route<?> someRoute)
+ public static void ensureHomeIsMonitoring(SafeCommandStore safeStore, SafeCommand safeCommand, Command command, Unseekables<?> maybeRoute)
{
- Command command = safeCommand.current();
- Invariants.checkState(owns(safeStore, command.txnId().epoch(), someRoute.homeKey()));
- // TODO (expected): tighten up definition of what we know about a Route (pack it into Known) and whether we've been witnessed to decide our action here
- if (command.hasBeen(PreAccepted))
- return command;
+ if (command.known().isDefinitionKnown() || command.hasBeen(Truncated))
+ return;
- Command result = safeCommand.updateAttributes(command.mutable().route(Route.merge((Route)someRoute, command.route())));
- safeStore.progressLog().unwitnessed(safeCommand.txnId(), Home);
- return result;
+ Route<?> route = Route.tryCastToRoute(maybeRoute);
+ if (route == null)
+ return;
+
+ safeCommand.updateAttributes(safeStore, command.mutable().route(Route.merge((Route)route, command.route())));
}
/**
@@ -1182,21 +1151,34 @@ public class Commands
@SuppressWarnings({"unchecked", "rawtypes"})
public static CommonAttributes updateRoute(Command command, Route<?> route)
{
- if (command.route() == null || !command.route().containsAll(route))
+ if (command.route() == null || !Route.isFullRoute(command.route()))
return command.mutable().route(Route.merge((Route)route, command.route()));
return command;
}
- private static ProgressShard progressShard(Route<?> route, @Nullable RoutingKey progressKey, Ranges coordinates)
+ public static Command updateRouteOrParticipants(SafeCommandStore safeStore, SafeCommand safeCommand, Unseekables<?> participants)
{
- if (progressKey == null)
- return No;
+ Command current = safeCommand.current();
+ if (current.hasBeen(Invalidated))
+ return current;
+
+ CommonAttributes updated = current;
+ if (Route.isRoute(participants))
+ {
+ updated = updateRoute(current, Route.castToRoute(participants));
+ }
+ else
+ {
+ Route<?> existingRoute = current.route();
+ if (existingRoute != null && !existingRoute.containsAll(participants))
+ updated = current.mutable().route(existingRoute.with((Unseekables) participants));
+ }
- if (!coordinates.contains(progressKey))
- return No;
+ if (current == updated)
+ return current;
- return progressKey.equals(route.homeKey()) ? Home : Local;
+ return safeCommand.updateAttributes(safeStore, updated);
}
@SuppressWarnings({"unchecked", "rawtypes"})
@@ -1252,7 +1234,7 @@ public class Commands
if (haveKnown.definition.isKnown())
{
// TODO (desired): avoid converting to participants before subtracting
- Participants<?> extraScope = acceptScope.subtract(cur.partialTxn().keys().toParticipants());
+ Participants<?> extraScope = acceptScope.without(cur.partialTxn().keys().toParticipants());
if (!containsAll(addPartialTxn, PartialTxn::covers, extraScope, permitStaleMissing))
return false;
}
@@ -1284,7 +1266,7 @@ public class Commands
// TODO (required, later): in the event we are depending on a stale key for an insert into a non-stale key, we cannot proceed and must mark the new key stale
// I think today this is unsupported in practice, but must be addressed before we improve efficiency of result handling
Ranges staleRanges = permitStaleMissing.commandStore().redundantBefore().staleRanges();
- required = required.subtract(staleRanges);
+ required = required.without(staleRanges);
return adding == null ? required.isEmpty() : covers.test(adding, required);
}
diff --git a/accord-core/src/main/java/accord/local/CommonAttributes.java b/accord-core/src/main/java/accord/local/CommonAttributes.java
index 46c38588..4612266f 100644
--- a/accord-core/src/main/java/accord/local/CommonAttributes.java
+++ b/accord-core/src/main/java/accord/local/CommonAttributes.java
@@ -20,22 +20,24 @@ package accord.local;
import javax.annotation.Nullable;
-import accord.api.VisibleForImplementation;
import accord.primitives.PartialDeps;
import accord.primitives.PartialTxn;
+import accord.primitives.Participants;
import accord.primitives.Route;
import accord.primitives.Seekables;
import accord.primitives.TxnId;
+import accord.utils.Invariants;
public interface CommonAttributes
{
TxnId txnId();
Status.Durability durability();
Route<?> route();
+ // if we don't know a route, we may know some participants; if we know a route, this must return the same object as route()
+ Participants<?> participants();
PartialTxn partialTxn();
@Nullable Seekables<?, ?> additionalKeysOrRanges();
PartialDeps partialDeps();
- Listeners.Immutable durableListeners();
default Mutable mutable()
{
@@ -47,10 +49,10 @@ public interface CommonAttributes
private TxnId txnId;
private Status.Durability durability;
private Route<?> route;
+ private Participants<?> participants; // if route != null, route == participants
private PartialTxn partialTxn;
private PartialDeps partialDeps;
private Seekables<?, ?> additionalKeysOrRanges;
- private Listeners listeners;
public Mutable(TxnId txnId)
{
@@ -62,10 +64,11 @@ public interface CommonAttributes
this.txnId = attributes.txnId();
this.durability = attributes.durability();
this.route = attributes.route();
+ this.participants = attributes.participants();
+ Invariants.checkState(participants == route || route == null);
this.partialTxn = attributes.partialTxn();
this.partialDeps = attributes.partialDeps();
this.additionalKeysOrRanges = attributes.additionalKeysOrRanges();
- this.listeners = attributes.durableListeners();
}
@Override
@@ -107,6 +110,20 @@ public interface CommonAttributes
public Mutable route(Route<?> route)
{
this.route = route;
+ this.participants = route;
+ return this;
+ }
+
+ @Override
+ public Participants<?> participants()
+ {
+ return participants;
+ }
+
+ public Mutable participants(Participants<?> participants)
+ {
+ Invariants.checkState(route == null);
+ this.participants = participants;
return this;
}
@@ -122,15 +139,15 @@ public interface CommonAttributes
this.partialTxn = partialTxn;
if (prev != null || additionalKeysOrRanges != null)
{
- Seekables<?, ?> removed = prev == null ? null : ((Seekables) prev.keys()).subtract(partialTxn.keys());
+ Seekables<?, ?> removed = prev == null ? null : ((Seekables) prev.keys()).without(partialTxn.keys());
if (prev != null && !removed.isEmpty())
{
if (additionalKeysOrRanges == null) additionalKeysOrRanges = removed;
- else additionalKeysOrRanges = ((Seekables)additionalKeysOrRanges).subtract(partialTxn.keys()).with(removed);
+ else additionalKeysOrRanges = ((Seekables)additionalKeysOrRanges).without(partialTxn.keys()).with(removed);
}
else if (additionalKeysOrRanges != null)
{
- additionalKeysOrRanges = ((Seekables)additionalKeysOrRanges).subtract(partialTxn.keys());
+ additionalKeysOrRanges = ((Seekables)additionalKeysOrRanges).without(partialTxn.keys());
}
}
return this;
@@ -164,43 +181,5 @@ public interface CommonAttributes
this.additionalKeysOrRanges = additionalKeysOrRanges;
return this;
}
-
-
- @Override
- public Listeners.Immutable durableListeners()
- {
- if (listeners == null || listeners.isEmpty())
- return Listeners.Immutable.EMPTY;
- if (listeners instanceof Listeners.Immutable)
- return (Listeners.Immutable) listeners;
- return new Listeners.Immutable(listeners);
- }
-
- public Mutable addListener(Command.DurableAndIdempotentListener listener)
- {
- if (listeners == null)
- listeners = new Listeners();
- else if (listeners instanceof Listeners.Immutable)
- listeners = new Listeners(listeners);
- listeners.add(listener);
- return this;
- }
-
- public Mutable removeListener(Command.Listener listener)
- {
- if (listener == null || listeners.isEmpty())
- return this;
- if (listeners instanceof Listeners.Immutable)
- listeners = new Listeners(listeners);
- listeners.remove(listener);
- return this;
- }
-
- @VisibleForImplementation
- public Mutable setListeners(Listeners.Immutable listeners)
- {
- this.listeners = listeners;
- return this;
- }
}
}
diff --git a/accord-core/src/main/java/accord/local/Listeners.java b/accord-core/src/main/java/accord/local/Listeners.java
deleted file mode 100644
index b4059620..00000000
--- a/accord-core/src/main/java/accord/local/Listeners.java
+++ /dev/null
@@ -1,95 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package accord.local;
-
-import java.util.Collection;
-import java.util.function.Predicate;
-
-import accord.utils.DeterministicSet;
-
-// TODO (expected): these are immutable collections over sortable keys, so should simply use BTree (or arrays)
-public class Listeners<L extends Command.Listener> extends DeterministicSet<L>
-{
- public static Listeners EMPTY = new Listeners<>();
-
- public Listeners()
- {
- }
-
- public Listeners(Listeners<L> copy)
- {
- super(copy);
- }
-
- public static class Immutable<L extends Command.Listener> extends Listeners<L>
- {
- public static final Immutable EMPTY = new Immutable();
-
- private Immutable()
- {
- super();
- }
-
- public Immutable(Listeners<L> listeners)
- {
- super(listeners);
- }
-
- public Listeners<L> mutable()
- {
- return new Listeners<>(this);
- }
-
- @Override
- public boolean add(L item)
- {
- throw new UnsupportedOperationException("Cannot modify immutable set");
- }
-
- @Override
- public boolean remove(Object item)
- {
- throw new UnsupportedOperationException("Cannot modify immutable set");
- }
-
- @Override
- public boolean removeAll(Collection<?> c)
- {
- throw new UnsupportedOperationException("Cannot modify immutable set");
- }
-
- @Override
- public boolean addAll(Collection<? extends L> c)
- {
- throw new UnsupportedOperationException("Cannot modify immutable set");
- }
-
- @Override
- public boolean retainAll(Collection<?> c)
- {
- throw new UnsupportedOperationException("Cannot modify immutable set");
- }
-
- @Override
- public boolean removeIf(Predicate<? super L> filter)
- {
- throw new UnsupportedOperationException("Cannot modify immutable set");
- }
- }
-}
diff --git a/accord-core/src/main/java/accord/local/Node.java b/accord-core/src/main/java/accord/local/Node.java
index 8450ec49..f008edf1 100644
--- a/accord-core/src/main/java/accord/local/Node.java
+++ b/accord-core/src/main/java/accord/local/Node.java
@@ -46,13 +46,16 @@ import accord.api.ConfigurationService;
import accord.api.ConfigurationService.EpochReady;
import accord.api.DataStore;
import accord.api.Key;
+import accord.api.LocalConfig;
+import accord.api.LocalListeners;
import accord.api.MessageSink;
import accord.api.ProgressLog;
+import accord.api.RemoteListeners;
+import accord.api.RequestTimeouts;
import accord.api.Result;
import accord.api.RoutingKey;
import accord.api.Scheduler;
import accord.api.TopologySorter;
-import accord.config.LocalConfig;
import accord.coordinate.CoordinateEphemeralRead;
import accord.coordinate.CoordinateTransaction;
import accord.coordinate.CoordinationAdapter;
@@ -62,7 +65,6 @@ import accord.coordinate.MaybeRecover;
import accord.coordinate.Outcome;
import accord.coordinate.RecoverWithRoute;
import accord.messages.Callback;
-import accord.messages.LocalRequest;
import accord.messages.Reply;
import accord.messages.ReplyContext;
import accord.messages.Request;
@@ -88,7 +90,6 @@ import accord.utils.Invariants;
import accord.utils.MapReduceConsume;
import accord.utils.RandomSource;
import accord.utils.async.AsyncChain;
-import accord.utils.async.AsyncChains;
import accord.utils.async.AsyncExecutor;
import accord.utils.async.AsyncResult;
import accord.utils.async.AsyncResults;
@@ -150,9 +151,10 @@ public class Node implements ConfigurationService.Listener, NodeTimeService
private final Id id;
private final MessageSink messageSink;
- private final LocalRequest.Handler localRequestHandler;
private final ConfigurationService configService;
private final TopologyManager topology;
+ private final RemoteListeners listeners;
+ private final RequestTimeouts timeouts;
private final CommandStores commandStores;
private final CoordinationAdapter.Factory coordinationAdapters;
@@ -169,27 +171,28 @@ public class Node implements ConfigurationService.Listener, NodeTimeService
// TODO (expected, liveness): monitor the contents of this collection for stalled coordination, and excise them
private final Map<TxnId, AsyncResult<? extends Outcome>> coordinating = new ConcurrentHashMap<>();
- public Node(Id id, MessageSink messageSink, LocalRequest.Handler localRequestHandler,
+ public Node(Id id, MessageSink messageSink,
ConfigurationService configService, LongSupplier nowSupplier, ToLongFunction<TimeUnit> elapsed,
Supplier<DataStore> dataSupplier, ShardDistributor shardDistributor, Agent agent, RandomSource random, Scheduler scheduler, TopologySorter.Supplier topologySorter,
- Function<Node, ProgressLog.Factory> progressLogFactory, CommandStores.Factory factory, CoordinationAdapter.Factory coordinationAdapters,
+ Function<Node, RemoteListeners> remoteListenersFactory, Function<Node, RequestTimeouts> requestTimeoutsFactory, Function<Node, ProgressLog.Factory> progressLogFactory, Function<Node, LocalListeners.Factory> localListenersFactory, CommandStores.Factory factory, CoordinationAdapter.Factory coordinationAdapters,
LocalConfig localConfig)
{
this.id = id;
+ this.scheduler = scheduler; // we set scheduler first so that e.g. requestTimeoutsFactory and progressLogFactory can take references to it
this.localConfig = localConfig;
this.messageSink = messageSink;
- this.localRequestHandler = localRequestHandler;
this.configService = configService;
this.coordinationAdapters = coordinationAdapters;
this.topology = new TopologyManager(topologySorter, agent, id, scheduler, elapsed, localConfig);
topology.scheduleTopologyUpdateWatchdog();
+ this.listeners = remoteListenersFactory.apply(this);
+ this.timeouts = requestTimeoutsFactory.apply(this);
this.nowSupplier = nowSupplier;
this.elapsed = elapsed;
this.now = new AtomicReference<>(Timestamp.fromValues(topology.epoch(), nowSupplier.getAsLong(), id));
this.agent = agent;
this.random = random;
- this.scheduler = scheduler;
- this.commandStores = factory.create(this, agent, dataSupplier.get(), random.fork(), shardDistributor, progressLogFactory.apply(this));
+ this.commandStores = factory.create(this, agent, dataSupplier.get(), random.fork(), shardDistributor, progressLogFactory.apply(this), localListenersFactory.apply(this));
// TODO review these leak a reference to an object that hasn't finished construction, possibly to other threads
configService.registerListener(this);
}
@@ -307,6 +310,22 @@ public class Node implements ConfigurationService.Listener, NodeTimeService
}
}
+ public void withEpoch(long epoch, BiConsumer<?, Throwable> ifFailure, Runnable ifSuccess)
+ {
+ if (topology.hasEpoch(epoch))
+ {
+ ifSuccess.run();
+ }
+ else
+ {
+ configService.fetchTopologyForEpoch(epoch);
+ topology.awaitEpoch(epoch).begin((success, fail) -> {
+ if (fail != null) ifFailure.accept(null, CoordinationFailed.wrap(fail));
+ else ifSuccess.run();;
+ });
+ }
+ }
+
@Inline
public <T> AsyncChain<T> withEpoch(long epoch, Supplier<? extends AsyncChain<T>> supplier)
{
@@ -526,23 +545,6 @@ public class Node implements ConfigurationService.Listener, NodeTimeService
messageSink.send(to, send);
}
- public <R> void localRequest(LocalRequest<R> message, BiConsumer<? super R, Throwable> callback)
- {
- localRequestHandler.handle(message, callback, this);
- }
-
- public <R> AsyncChain<R> localRequest(LocalRequest<R> message)
- {
- return new AsyncChains.Head<R>()
- {
- @Override
- protected void start(BiConsumer<? super R, Throwable> callback)
- {
- localRequest(message, callback);
- }
- };
- }
-
public void reply(Id replyingToNode, ReplyContext replyContext, Reply send, Throwable failure)
{
if (failure != null)
@@ -623,45 +625,6 @@ public class Node implements ConfigurationService.Listener, NodeTimeService
return keysOrRanges.get(random.nextInt(keysOrRanges.size())).someIntersectingRoutingKey(null);
}
- public RoutingKey selectProgressKey(TxnId txnId, Route<?> route, RoutingKey homeKey)
- {
- return selectProgressKey(txnId.epoch(), route, homeKey);
- }
-
- public RoutingKey selectProgressKey(long epoch, Route<?> route, RoutingKey homeKey)
- {
- RoutingKey progressKey = trySelectProgressKey(epoch, route, homeKey);
- if (progressKey == null)
- throw new IllegalStateException();
- return progressKey;
- }
-
- public RoutingKey trySelectProgressKey(TxnId txnId, Route<?> route)
- {
- return trySelectProgressKey(txnId, route, route.homeKey());
- }
-
- public RoutingKey trySelectProgressKey(TxnId txnId, Route<?> route, RoutingKey homeKey)
- {
- return trySelectProgressKey(txnId.epoch(), route, homeKey);
- }
-
- public RoutingKey trySelectProgressKey(long epoch, Route<?> route, RoutingKey homeKey)
- {
- return trySelectProgressKey(this.topology.localForEpoch(epoch), route, homeKey);
- }
-
- private static RoutingKey trySelectProgressKey(Topology topology, Route<?> route, RoutingKey homeKey)
- {
- if (topology.ranges().contains(homeKey))
- return homeKey;
-
- int i = (int)route.findNextIntersection(0, topology.ranges(), 0);
- if (i < 0)
- return null;
- return route.get(i).someIntersectingRoutingKey(topology.ranges());
- }
-
static class RecoverFuture<T> extends AsyncResults.SettableResult<T> implements BiConsumer<T, Throwable>
{
@Override
@@ -745,6 +708,16 @@ public class Node implements ConfigurationService.Listener, NodeTimeService
return agent;
}
+ public RemoteListeners remoteListeners()
+ {
+ return listeners;
+ }
+
+ public RequestTimeouts requestTimeouts()
+ {
+ return timeouts;
+ }
+
@Override
public Id id()
{
diff --git a/accord-core/src/main/java/accord/local/NodeTimeService.java b/accord-core/src/main/java/accord/local/NodeTimeService.java
index 017e11c0..015390f3 100644
--- a/accord-core/src/main/java/accord/local/NodeTimeService.java
+++ b/accord-core/src/main/java/accord/local/NodeTimeService.java
@@ -37,8 +37,9 @@ public interface NodeTimeService
long now();
/**
- * Return the current time since some arbitrary epoch in the specified time unit. May still be simulated time and not
- * real time. The time returned by this will be monotonic.
+ * Return the current time since some arbitrary epoch in the specified time unit. May be simulated time and not
+ * real time. This clock should not go backwards, nor should it generally correct for clock skew - it should
+ * only track time elapsed between two points.
*/
long elapsed(TimeUnit unit);
@@ -46,7 +47,8 @@ public interface NodeTimeService
static ToLongFunction<TimeUnit> elapsedWrapperFromMonotonicSource(TimeUnit sourceUnit, LongSupplier monotonicNowSupplier)
{
- return resultUnit -> resultUnit.convert(monotonicNowSupplier.getAsLong(), sourceUnit);
+ long epoch = Math.max(0, monotonicNowSupplier.getAsLong() - sourceUnit.convert(1L, TimeUnit.DAYS));
+ return resultUnit -> resultUnit.convert(monotonicNowSupplier.getAsLong() - epoch, sourceUnit);
}
/**
@@ -56,7 +58,7 @@ public interface NodeTimeService
@VisibleForTesting
static ToLongFunction<TimeUnit> elapsedWrapperFromNonMonotonicSource(TimeUnit sourceUnit, LongSupplier nonMonotonicNowSupplier)
{
- return elapsedWrapperFromMonotonicSource(sourceUnit, new MonotonicWrapper(nonMonotonicNowSupplier));
+ return elapsedWrapperFromMonotonicSource(sourceUnit, new MonotonicWrapper(sourceUnit, nonMonotonicNowSupplier));
}
class MonotonicWrapper implements LongSupplier
@@ -66,11 +68,13 @@ public interface NodeTimeService
private long delta = 0;
// Use an arbitrary epoch
- private long epoch = Long.MAX_VALUE / 4;
+ private final long epoch;
- private MonotonicWrapper(LongSupplier nowSupplier)
+ private MonotonicWrapper(TimeUnit nowUnit, LongSupplier nowSupplier)
{
this.nowSupplier = nowSupplier;
+ // pick an arbitrary epoch we can safely convert back to any time unit from the source unit
+ this.epoch = nowUnit.convert(Long.MAX_VALUE / 4, TimeUnit.NANOSECONDS);
}
@Override
diff --git a/accord-core/src/main/java/accord/local/RedundantBefore.java b/accord-core/src/main/java/accord/local/RedundantBefore.java
index dbdfa000..26942bf6 100644
--- a/accord-core/src/main/java/accord/local/RedundantBefore.java
+++ b/accord-core/src/main/java/accord/local/RedundantBefore.java
@@ -70,9 +70,25 @@ public class RedundantBefore extends ReducingRangeMap<RedundantBefore.Entry>
/**
* Represents the maximum TxnId we know to have fully executed until locally for the range in question.
* Unless we are stale or pre-bootstrap, in which case no such guarantees can be made.
+ *
+ * We maintain locallyAppliedOrInvalidatedBefore that were reached prior to a new bootstrap exceeding them,
+ * as these were reached correctly.
*/
public final @Nonnull TxnId locallyAppliedOrInvalidatedBefore;
+ /**
+ * Represents the maximum TxnId we know to have fully executed until locally for the range in question,
+ * and for which we guarantee that any distributed decision that might need to be consulted is also recorded
+ * locally (i.e. it is known to be Stable locally, or else it did not execute)
+ *
+ * We maintain locallyDecidedAndExecutedBefore that were reached prior to a new bootstrap exceeding them,
+ * as these were reached correctly and can be used for GC.
+ *
+ * However, once a bootstrap has begun we cannot safely advance until shardAppliedOrInvalidatedBefore goes
+ * ahead of the bootstrappedAt, because we cannot guarantee to have any intervening decision recorded locally.
+ */
+ public final @Nonnull TxnId locallyDecidedAndAppliedOrInvalidatedBefore;
+
/**
* Represents the maximum TxnId we know to have fully executed until across all healthy replicas for the range in question.
* Unless we are stale or pre-bootstrap, in which case no such guarantees can be made.
@@ -101,15 +117,22 @@ public class RedundantBefore extends ReducingRangeMap<RedundantBefore.Entry>
public final @Nullable Timestamp staleUntilAtLeast;
public Entry(Range range, long startEpoch, long endEpoch, @Nonnull TxnId locallyAppliedOrInvalidatedBefore, @Nonnull TxnId shardAppliedOrInvalidatedBefore, @Nonnull TxnId bootstrappedAt, @Nullable Timestamp staleUntilAtLeast)
+ {
+ this(range, startEpoch, endEpoch, locallyAppliedOrInvalidatedBefore, locallyAppliedOrInvalidatedBefore, shardAppliedOrInvalidatedBefore, bootstrappedAt, staleUntilAtLeast);
+ }
+
+ public Entry(Range range, long startEpoch, long endEpoch, @Nonnull TxnId locallyAppliedOrInvalidatedBefore, @Nonnull TxnId locallyDecidedAndAppliedOrInvalidatedBefore, @Nonnull TxnId shardAppliedOrInvalidatedBefore, @Nonnull TxnId bootstrappedAt, @Nullable Timestamp staleUntilAtLeast)
{
this.range = range;
this.startEpoch = startEpoch;
this.endEpoch = endEpoch;
this.locallyAppliedOrInvalidatedBefore = locallyAppliedOrInvalidatedBefore;
+ this.locallyDecidedAndAppliedOrInvalidatedBefore = locallyDecidedAndAppliedOrInvalidatedBefore;
this.shardAppliedOrInvalidatedBefore = shardAppliedOrInvalidatedBefore;
this.bootstrappedAt = bootstrappedAt;
this.staleUntilAtLeast = staleUntilAtLeast;
Invariants.checkArgument(locallyAppliedOrInvalidatedBefore.equals(TxnId.NONE) || locallyAppliedOrInvalidatedBefore.domain().isRange());
+ Invariants.checkArgument(locallyDecidedAndAppliedOrInvalidatedBefore.equals(TxnId.NONE) || locallyDecidedAndAppliedOrInvalidatedBefore.domain().isRange());
Invariants.checkArgument(shardAppliedOrInvalidatedBefore.equals(TxnId.NONE) || shardAppliedOrInvalidatedBefore.domain().isRange());
}
@@ -118,42 +141,48 @@ public class RedundantBefore extends ReducingRangeMap<RedundantBefore.Entry>
return merge(a.range.slice(b.range), a, b);
}
- private static Entry merge(Range range, Entry a, Entry b)
+ private static Entry merge(Range range, Entry cur, Entry add)
{
- if (a.startEpoch > b.endEpoch)
- return a;
+ if (cur.startEpoch > add.endEpoch)
+ return cur;
- if (b.startEpoch > a.endEpoch)
- return b;
+ if (add.startEpoch > cur.endEpoch)
+ return add;
- long startEpoch = Long.max(a.startEpoch, b.startEpoch);
- long endEpoch = Long.min(a.endEpoch, b.endEpoch);
- int cl = a.locallyAppliedOrInvalidatedBefore.compareTo(b.locallyAppliedOrInvalidatedBefore);
- int cs = a.shardAppliedOrInvalidatedBefore.compareTo(b.shardAppliedOrInvalidatedBefore);
- int cb = a.bootstrappedAt.compareTo(b.bootstrappedAt);
- int csu = compareStaleUntilAtLeast(a.staleUntilAtLeast, b.staleUntilAtLeast);
+ long startEpoch = Long.max(cur.startEpoch, add.startEpoch);
+ long endEpoch = Long.min(cur.endEpoch, add.endEpoch);
+ int cl = cur.locallyAppliedOrInvalidatedBefore.compareTo(add.locallyAppliedOrInvalidatedBefore);
+ int cd = cur.locallyDecidedAndAppliedOrInvalidatedBefore.compareTo(add.locallyDecidedAndAppliedOrInvalidatedBefore);
+ int cs = cur.shardAppliedOrInvalidatedBefore.compareTo(add.shardAppliedOrInvalidatedBefore);
+ int cb = cur.bootstrappedAt.compareTo(add.bootstrappedAt);
+ int csu = compareStaleUntilAtLeast(cur.staleUntilAtLeast, add.staleUntilAtLeast);
- if (range.equals(a.range) && startEpoch == a.startEpoch && endEpoch == a.endEpoch && cl >= 0 && cb >= 0 && cs >= 0 && csu >= 0)
- return a;
- if (range.equals(b.range) && startEpoch == b.startEpoch && endEpoch == b.endEpoch && cl <= 0 && cb <= 0 && cs <= 0 && csu <= 0)
- return b;
+ if (range.equals(cur.range) && startEpoch == cur.startEpoch && endEpoch == cur.endEpoch && cl >= 0 && cb >= 0 && cs >= 0 && csu >= 0)
+ return cur;
+ if (range.equals(add.range) && startEpoch == add.startEpoch && endEpoch == add.endEpoch && cl <= 0 && cb <= 0 && cs <= 0 && csu <= 0)
+ return add;
- TxnId locallyAppliedOrInvalidatedBefore = cl >= 0 ? a.locallyAppliedOrInvalidatedBefore : b.locallyAppliedOrInvalidatedBefore;
- TxnId shardAppliedOrInvalidatedBefore = cs >= 0 ? a.shardAppliedOrInvalidatedBefore : b.shardAppliedOrInvalidatedBefore;
- TxnId bootstrappedAt = cb >= 0 ? a.bootstrappedAt : b.bootstrappedAt;
- Timestamp staleUntilAtLeast = csu >= 0 ? a.staleUntilAtLeast : b.staleUntilAtLeast;
+ TxnId locallyAppliedOrInvalidatedBefore = cl >= 0 ? cur.locallyAppliedOrInvalidatedBefore : add.locallyAppliedOrInvalidatedBefore;
+ TxnId locallyDecidedAndAppliedOrInvalidatedBefore = cd >= 0 ? cur.locallyDecidedAndAppliedOrInvalidatedBefore : add.locallyDecidedAndAppliedOrInvalidatedBefore;
+ TxnId shardAppliedOrInvalidatedBefore = cs >= 0 ? cur.shardAppliedOrInvalidatedBefore : add.shardAppliedOrInvalidatedBefore;
+ TxnId bootstrappedAt = cb >= 0 ? cur.bootstrappedAt : add.bootstrappedAt;
+ Timestamp staleUntilAtLeast = csu >= 0 ? cur.staleUntilAtLeast : add.staleUntilAtLeast;
- // if our redundantBefore predates bootstrappedAt, we should clear it to avoid erroneously treating
- // transactions prior as locally redundant when they may simply have not applied yet, since we may
+ // if a NEW redundantBefore predates our current bootstrappedAt, we should not update it to avoid erroneously
+ // treating transactions prior as locally redundant when they may simply have not applied yet, since we may
// permit the sync point that defines redundancy to apply locally without waiting for these earlier
- // transactions, since we now consider them to be bootstrapping
+ // transactions, since we now consider them to be bootstrapping.
+ // however, any locallyAppliedOrInvalidatedBefore that was set before bootstrap can be safely maintained,
+ // and should not ideally go backwards (as CommandsForKey utilises it for GC)
// TODO (desired): revisit later as semantics here evolve
if (bootstrappedAt.compareTo(locallyAppliedOrInvalidatedBefore) >= 0)
- locallyAppliedOrInvalidatedBefore = TxnId.NONE;
+ locallyAppliedOrInvalidatedBefore = cur.locallyAppliedOrInvalidatedBefore;
+ if (bootstrappedAt.compareTo(shardAppliedOrInvalidatedBefore) >= 0)
+ locallyDecidedAndAppliedOrInvalidatedBefore = cur.locallyDecidedAndAppliedOrInvalidatedBefore;
if (staleUntilAtLeast != null && bootstrappedAt.compareTo(staleUntilAtLeast) >= 0)
staleUntilAtLeast = null;
- return new Entry(range, startEpoch, endEpoch, locallyAppliedOrInvalidatedBefore, shardAppliedOrInvalidatedBefore, bootstrappedAt, staleUntilAtLeast);
+ return new Entry(range, startEpoch, endEpoch, locallyAppliedOrInvalidatedBefore, locallyDecidedAndAppliedOrInvalidatedBefore, shardAppliedOrInvalidatedBefore, bootstrappedAt, staleUntilAtLeast);
}
static @Nonnull RedundantStatus getAndMerge(Entry entry, @Nonnull RedundantStatus prev, TxnId txnId, EpochSupplier executeAt)
@@ -197,7 +226,7 @@ public class RedundantBefore extends ReducingRangeMap<RedundantBefore.Entry>
return safeToRead;
if (bootstrapAt.compareTo(entry.bootstrappedAt) < 0 || (entry.staleUntilAtLeast != null && bootstrapAt.compareTo(entry.staleUntilAtLeast) < 0))
- return safeToRead.subtract(Ranges.of(entry.range));
+ return safeToRead.without(Ranges.of(entry.range));
return safeToRead;
}
@@ -208,7 +237,7 @@ public class RedundantBefore extends ReducingRangeMap<RedundantBefore.Entry>
return executeRanges;
if (txnId.compareTo(entry.bootstrappedAt) < 0 || entry.staleUntilAtLeast != null)
- return executeRanges.subtract(Ranges.of(entry.range));
+ return executeRanges.without(Ranges.of(entry.range));
return executeRanges;
}
@@ -219,7 +248,7 @@ public class RedundantBefore extends ReducingRangeMap<RedundantBefore.Entry>
return notRedundant;
if (txnId.compareTo(entry.shardAppliedOrInvalidatedBefore) < 0)
- return notRedundant.subtract(Ranges.of(entry.range));
+ return notRedundant.without(Ranges.of(entry.range));
return notRedundant;
}
@@ -256,6 +285,11 @@ public class RedundantBefore extends ReducingRangeMap<RedundantBefore.Entry>
return locallyAppliedOrInvalidatedBefore;
}
+ public final TxnId locallyRedundantOrBootstrappedBefore()
+ {
+ return TxnId.max(locallyAppliedOrInvalidatedBefore, bootstrappedAt);
+ }
+
// TODO (required, consider): this admits the range of epochs that cross the two timestamps, which matches our
// behaviour elsewhere but we probably want to only interact with the two point epochs in which we participate,
// but note hasRedundantDependencies which really does want to scan a range
diff --git a/accord-core/src/main/java/accord/local/SafeCommand.java b/accord-core/src/main/java/accord/local/SafeCommand.java
index dbc4b54b..b89b8d94 100644
--- a/accord-core/src/main/java/accord/local/SafeCommand.java
+++ b/accord-core/src/main/java/accord/local/SafeCommand.java
@@ -19,7 +19,6 @@
package accord.local;
import accord.api.Result;
-import accord.local.Command.TransientListener;
import accord.local.Command.Truncated;
import accord.primitives.Ballot;
import accord.primitives.Seekables;
@@ -40,9 +39,6 @@ public abstract class SafeCommand
public abstract Command current();
public abstract void invalidate();
public abstract boolean invalidated();
- public abstract void addListener(Command.TransientListener listener);
- public abstract boolean removeListener(Command.TransientListener listener);
- public abstract Listeners<Command.TransientListener> transientListeners();
public boolean isUnset()
{
@@ -63,6 +59,7 @@ public abstract class SafeCommand
return update;
set(update);
+ safeStore.progressLog().update(safeStore, txnId, prev, update);
safeStore.update(prev, update);
return update;
}
@@ -76,33 +73,17 @@ public abstract class SafeCommand
return update;
}
- public Command addListener(Command.DurableAndIdempotentListener listener)
- {
- return incidentalUpdate(Command.addListener(current(), listener));
- }
-
- public void addAndInvokeListener(SafeCommandStore safeStore, TransientListener listener)
- {
- addListener(listener);
- listener.onChange(safeStore, this);
- }
-
- public Command removeListener(Command.DurableAndIdempotentListener listener)
- {
- Command current = current();
- if (!current.durableListeners().contains(listener))
- return current;
- return incidentalUpdate(Command.removeListener(current(), listener));
- }
-
public Command.Committed updateWaitingOn(Command.WaitingOn.Update waitingOn)
{
return incidentalUpdate(Command.updateWaitingOn(current().asCommitted(), waitingOn));
}
- public Command updateAttributes(CommonAttributes attrs)
+ public Command updateAttributes(SafeCommandStore safeStore, CommonAttributes attrs)
{
- return incidentalUpdate(current().updateAttributes(attrs));
+ Command prev = current();
+ Command update = incidentalUpdate(prev.updateAttributes(attrs));
+ safeStore.progressLog().update(safeStore, txnId, prev, update);
+ return update;
}
public Command.PreAccepted preaccept(SafeCommandStore safeStore, CommonAttributes attrs, Timestamp executeAt, Ballot ballot)
@@ -137,7 +118,7 @@ public abstract class SafeCommand
return attrs;
if (attrs.partialTxn() != null)
- keysOrRanges = ((Seekables)keysOrRanges).subtract(attrs.partialTxn().keys());
+ keysOrRanges = ((Seekables)keysOrRanges).without(attrs.partialTxn().keys());
if (attrs.additionalKeysOrRanges() != null)
keysOrRanges = ((Seekables)keysOrRanges).with(attrs.additionalKeysOrRanges());
@@ -148,7 +129,7 @@ public abstract class SafeCommand
return attrs;
}
- public Command.Accepted acceptInvalidated(SafeCommandStore safeStore, Ballot ballot)
+ public Command acceptInvalidated(SafeCommandStore safeStore, Ballot ballot)
{
return update(safeStore, Command.acceptInvalidated(current(), ballot));
}
@@ -194,9 +175,7 @@ public abstract class SafeCommand
public Command.Executed applied(SafeCommandStore safeStore)
{
- Command.Executed executed = update(safeStore, Command.applied(current().asExecuted()));
- safeStore.progressLog().clear(txnId);
- return executed;
+ return update(safeStore, Command.applied(current().asExecuted()));
}
public Command.NotDefined uninitialised()
diff --git a/accord-core/src/main/java/accord/local/SafeCommandStore.java b/accord-core/src/main/java/accord/local/SafeCommandStore.java
index 6d06959c..e74e29af 100644
--- a/accord-core/src/main/java/accord/local/SafeCommandStore.java
+++ b/accord-core/src/main/java/accord/local/SafeCommandStore.java
@@ -18,12 +18,12 @@
package accord.local;
-import java.util.Iterator;
import javax.annotation.Nullable;
import accord.api.Agent;
import accord.api.DataStore;
import accord.api.Key;
+import accord.api.LocalListeners;
import accord.api.ProgressLog;
import accord.api.RoutingKey;
import accord.impl.ErasedSafeCommand;
@@ -41,6 +41,7 @@ import accord.primitives.Timestamp;
import accord.primitives.Txn.Kind.Kinds;
import accord.primitives.TxnId;
import accord.primitives.Unseekables;
+import accord.utils.Invariants;
import static accord.local.Cleanup.NO;
import static accord.local.KeyHistory.COMMANDS;
@@ -324,68 +325,29 @@ public abstract class SafeCommandStore
return commandStore().redundantBefore().preBootstrapOrStale(command.txnId(), command.executeAtOrTxnId(), forKeys) == FULLY;
}
- public void notifyListeners(SafeCommand safeCommand)
+ public void registerListener(SafeCommand listeningTo, SaveStatus await, Status.KnownRoute route, TxnId waiting)
{
- Command command = safeCommand.current();
- notifyListeners(safeCommand, command, command.durableListeners(), safeCommand.transientListeners());
+ Invariants.checkState(listeningTo.current().saveStatus().compareTo(await) < 0);
+ Invariants.checkState(!CommandsForKey.managesExecution(listeningTo.txnId()));
+ commandStore().listeners.register(listeningTo.txnId(), await, waiting);
}
- public void notifyListeners(SafeCommand safeCommand, Command command, Listeners<Command.DurableAndIdempotentListener> durableListeners, Listeners<Command.TransientListener> transientListeners)
+ public LocalListeners.Registered registerAndInvoke(TxnId txnId, RoutingKey someKey, LocalListeners.ComplexListener listener)
{
- Iterator<Command.DurableAndIdempotentListener> durableIterator = durableListeners.reverseIterator();
- while (durableIterator.hasNext())
- {
- Command.DurableAndIdempotentListener listener = durableIterator.next();
- notifyListener(this, safeCommand, command, listener);
- }
-
- Iterator<Command.TransientListener> transientIterator = transientListeners.reverseIterator();
- while (transientIterator.hasNext())
- {
- Command.TransientListener listener = transientIterator.next();
- notifyListener(this, safeCommand, command, listener);
- }
+ LocalListeners.Registered registered = register(txnId, listener);
+ if (!listener.notify(this, get(txnId, someKey)))
+ registered.cancel();
+ return registered;
}
- public static void notifyListener(SafeCommandStore safeStore, SafeCommand safeCommand, Command command, Command.TransientListener listener)
+ public LocalListeners.Registered register(TxnId txnId, LocalListeners.ComplexListener listener)
{
- if (!safeCommand.transientListeners().contains(listener))
- return;
-
- PreLoadContext context = listener.listenerPreLoadContext(command.txnId());
- if (safeStore.canExecuteWith(context))
- {
- listener.onChange(safeStore, safeCommand);
- }
- else
- {
- TxnId txnId = command.txnId();
- safeStore.commandStore()
- .execute(context, safeStore2 -> {
- SafeCommand safeCommand2 = safeStore2.get(txnId);
- // listeners invocations may be triggered more than once asynchronously for different changes
- // so one pending invocation may unregister the listener prior to the second invocation running
- // so we check if the listener is still valid before running
- if (safeCommand2.transientListeners().contains(listener))
- listener.onChange(safeStore2, safeCommand2);
- })
- .begin(safeStore.agent());
- }
+ return commandStore().listeners.register(txnId, listener);
}
- public static void notifyListener(SafeCommandStore safeStore, SafeCommand safeCommand, Command command, Command.DurableAndIdempotentListener listener)
+ public void notifyListeners(SafeCommand safeCommand, Command prev)
{
- PreLoadContext context = listener.listenerPreLoadContext(command.txnId());
- if (safeStore.canExecuteWith(context))
- {
- listener.onChange(safeStore, safeCommand);
- }
- else
- {
- TxnId txnId = command.txnId();
- safeStore.commandStore()
- .execute(context, safeStore2 -> listener.onChange(safeStore2, safeStore2.get(txnId)))
- .begin(safeStore.agent());
- }
+ commandStore().listeners.notify(this, safeCommand, prev);
}
+
}
diff --git a/accord-core/src/main/java/accord/local/SaveStatus.java b/accord-core/src/main/java/accord/local/SaveStatus.java
index 5dfa13fb..3b38224f 100644
--- a/accord-core/src/main/java/accord/local/SaveStatus.java
+++ b/accord-core/src/main/java/accord/local/SaveStatus.java
@@ -18,23 +18,32 @@
package accord.local;
-import accord.local.Status.*;
+import java.util.List;
+import java.util.function.Function;
+import java.util.function.Predicate;
+
+import accord.local.Status.Definition;
+import accord.local.Status.Known;
+import accord.local.Status.KnownDeps;
+import accord.local.Status.KnownExecuteAt;
+import accord.local.Status.KnownRoute;
+import accord.local.Status.Outcome;
+import accord.local.Status.Phase;
import accord.primitives.Ballot;
-import accord.primitives.Timestamp;
-import accord.primitives.TxnId;
import static accord.local.SaveStatus.LocalExecution.CleaningUp;
import static accord.local.SaveStatus.LocalExecution.NotReady;
import static accord.local.Status.Definition.DefinitionErased;
import static accord.local.Status.Definition.DefinitionKnown;
import static accord.local.Status.Definition.DefinitionUnknown;
-import static accord.local.Status.Known.Apply;
-import static accord.local.Status.Known.Decision;
-import static accord.local.Status.Known.ExecuteAtOnly;
-import static accord.local.Status.Known.Nothing;
-import static accord.local.Status.KnownDeps.*;
-import static accord.local.Status.KnownExecuteAt.*;
-import static accord.local.Status.KnownRoute.Covering;
+import static accord.local.Status.KnownDeps.DepsErased;
+import static accord.local.Status.KnownDeps.DepsKnown;
+import static accord.local.Status.KnownDeps.DepsProposed;
+import static accord.local.Status.KnownDeps.DepsUnknown;
+import static accord.local.Status.KnownExecuteAt.ExecuteAtErased;
+import static accord.local.Status.KnownExecuteAt.ExecuteAtKnown;
+import static accord.local.Status.KnownExecuteAt.ExecuteAtProposed;
+import static accord.local.Status.KnownExecuteAt.ExecuteAtUnknown;
import static accord.local.Status.KnownRoute.Full;
import static accord.local.Status.KnownRoute.Maybe;
import static accord.local.Status.Outcome.Unknown;
@@ -64,7 +73,7 @@ public enum SaveStatus
Accepted (Status.Accepted),
AcceptedWithDefinition (Status.Accepted, Full, DefinitionKnown, ExecuteAtProposed, DepsProposed, Unknown),
PreCommitted (Status.PreCommitted, LocalExecution.ReadyToExclude),
- PreCommittedWithAcceptedDeps (Status.PreCommitted, Covering, DefinitionUnknown, ExecuteAtKnown, DepsProposed, Unknown, LocalExecution.ReadyToExclude),
+ PreCommittedWithAcceptedDeps (Status.PreCommitted, Full, DefinitionUnknown, ExecuteAtKnown, DepsProposed, Unknown, LocalExecution.ReadyToExclude),
PreCommittedWithDefinition (Status.PreCommitted, Full, DefinitionKnown, ExecuteAtKnown, DepsUnknown, Unknown, LocalExecution.ReadyToExclude),
PreCommittedWithDefinitionAndAcceptedDeps(Status.PreCommitted, Full, DefinitionKnown, ExecuteAtKnown, DepsProposed, Unknown, LocalExecution.ReadyToExclude),
Committed (Status.Committed, LocalExecution.ReadyToExclude),
@@ -91,52 +100,51 @@ public enum SaveStatus
*/
public enum LocalExecution
{
- NotReady(Nothing),
- ReadyToExclude(ExecuteAtOnly),
- WaitingToExecute(Decision),
- ReadyToExecute(Decision),
- // TODO (expected): we seem to be able to await this when we know there are still local execution dependencies
- // we should only request this when we know the transaction can execute locally
- WaitingToApply(Apply),
- Applying(Apply),
- Applied(Apply),
- CleaningUp(Nothing);
-
- public final Known requires;
-
- LocalExecution(Known requires)
- {
- this.requires = requires;
- }
-
- public boolean isSatisfiedBy(Known known)
- {
- return requires.isSatisfiedBy(known);
- }
-
- public long fetchEpoch(TxnId txnId, Timestamp timestamp)
- {
- if (timestamp == null || timestamp.equals(Timestamp.NONE))
- return txnId.epoch();
-
- switch (this)
- {
- default: throw new AssertionError("Unexpected status: " + this);
- case NotReady:
- case ReadyToExclude:
- case ReadyToExecute:
- case WaitingToExecute:
- return txnId.epoch();
-
- case WaitingToApply:
- case Applying:
- case Applied:
- case CleaningUp:
- return timestamp.epoch();
- }
- }
+ /**
+ * Still coordinating a decision
+ */
+ NotReady,
+
+ /**
+ * Ready to exclude based on the decided executeAt, but the dependencies are not known.
+ */
+ ReadyToExclude,
+
+ /**
+ * A complete execution decision has been made, but the dependencies have not executed
+ */
+ WaitingToExecute,
+
+ /**
+ * The command is ready to execute, and a coordinator should promptly compute and distribute the command's outcome
+ */
+ ReadyToExecute,
+
+ /**
+ * The command has been executed and its outcome is known, but we have not locally executed all of its dependencies
+ * TODO (expected): we should only await this when we have no local execution dependencies, or we know that
+ * the command is durable remotely
+ */
+ WaitingToApply,
+
+ /**
+ * The command is being asynchronously applied to the local data store
+ */
+ Applying,
+
+ /**
+ * The command has been applied to the local data store
+ */
+ Applied,
+
+ /**
+ * Some or all of the command's local state has been garbage collected
+ */
+ CleaningUp
}
+ private static final SaveStatus[] lookup = values();
+
public final Status status;
public final Phase phase;
public final Known known;
@@ -246,11 +254,17 @@ public enum SaveStatus
}
}
+ private static final Known DefinitionOnly = new Known(Full, DefinitionKnown, ExecuteAtUnknown, DepsUnknown, Unknown);
+ public static SaveStatus withDefinition(SaveStatus status)
+ {
+ return enrich(status, DefinitionOnly);
+ }
+
public static SaveStatus enrich(SaveStatus status, Known known)
{
+ // most statuses already know everything they can
switch (status.status)
{
- // most statuses already know everything they can
case NotDefined:
case PreAccepted:
case Accepted:
@@ -260,6 +274,7 @@ public enum SaveStatus
if (known.isSatisfiedBy(status.known))
return status;
return get(status.status, status.known.atLeast(known));
+
case Stable:
return status;
@@ -329,6 +344,34 @@ public enum SaveStatus
return a.compareTo(b) >= 0 ? av : bv;
}
+ public static <T> T max(List<T> list, Function<T, SaveStatus> getStatus, Function<T, Ballot> getAcceptedOrCommittedBallot, Predicate<T> filter, boolean preferKnowledge)
+ {
+ T max = null;
+ SaveStatus maxStatus = null;
+ Ballot maxBallot = null;
+ for (T item : list)
+ {
+ if (!filter.test(item))
+ continue;
+
+ SaveStatus status = getStatus.apply(item);
+ Ballot ballot = getAcceptedOrCommittedBallot.apply(item);
+ boolean update = max == null
+ || maxStatus.phase.compareTo(status.phase) < 0
+ || (maxStatus.phase.tieBreakWithBallot ? maxStatus.phase == status.phase && maxBallot.compareTo(ballot) < 0
+ : maxStatus.compareTo(status) < 0);
+
+ if (!update)
+ continue;
+
+ max = item;
+ maxStatus = status;
+ maxBallot = ballot;
+ }
+
+ return max;
+ }
+
// TODO (desired): this isn't a simple linear relationship - Committed has some more knowledge, but some less; PreAccepted has much less
public boolean lowerHasMoreKnowledge(SaveStatus than)
{
@@ -340,4 +383,11 @@ public enum SaveStatus
return false;
}
+
+ public static SaveStatus forOrdinal(int ordinal)
+ {
+ if (ordinal < 0 || ordinal > lookup.length)
+ throw new IndexOutOfBoundsException(ordinal);
+ return lookup[ordinal];
+ }
}
diff --git a/accord-core/src/main/java/accord/local/Status.java b/accord-core/src/main/java/accord/local/Status.java
index ccb5add4..a39c4829 100644
--- a/accord-core/src/main/java/accord/local/Status.java
+++ b/accord-core/src/main/java/accord/local/Status.java
@@ -76,7 +76,7 @@ public enum Status
* To solve this problem we simply permit the executeAt we discover for B to be propagated to A* without
* its dependencies. Though this does complicate the state machine a little.
*/
- PreCommitted (Accept, Maybe, DefinitionUnknown, ExecuteAtKnown, DepsUnknown, Unknown),
+ PreCommitted (Accept, Full, DefinitionUnknown, ExecuteAtKnown, DepsUnknown, Unknown),
Committed (Commit, Full, DefinitionKnown, ExecuteAtKnown, DepsCommitted,Unknown),
Stable (Execute, Full, DefinitionKnown, ExecuteAtKnown, DepsKnown, Unknown),
@@ -124,11 +124,7 @@ public enum Status
public static class Known
{
public static final Known Nothing = new Known(Maybe, DefinitionUnknown, ExecuteAtUnknown, DepsUnknown, Unknown);
- // TODO (expected): deprecate DefinitionOnly
- public static final Known DefinitionOnly = new Known(Maybe, DefinitionKnown, ExecuteAtUnknown, DepsUnknown, Unknown);
public static final Known DefinitionAndRoute = new Known(Full, DefinitionKnown, ExecuteAtUnknown, DepsUnknown, Unknown);
- public static final Known ExecuteAtOnly = new Known(Maybe, DefinitionUnknown, ExecuteAtKnown, DepsUnknown, Unknown);
- public static final Known Decision = new Known(Full, DefinitionKnown, ExecuteAtKnown, DepsKnown, Unknown);
public static final Known Apply = new Known(Full, DefinitionUnknown, ExecuteAtKnown, DepsKnown, Outcome.Apply);
public static final Known Invalidated = new Known(Maybe, DefinitionUnknown, ExecuteAtUnknown, DepsUnknown, Outcome.Invalidated);
@@ -277,23 +273,34 @@ public enum Status
* Convert this Known to one that represents the knowledge that can be propagated to another replica without
* further information available to that replica, e.g. we cannot apply without also knowing the definition.
*/
- // TODO (expected): merge propagates and propagatesStatus
public Known propagates()
+ {
+ return propagatesSaveStatus().known;
+ }
+
+ public SaveStatus propagatesSaveStatus()
{
if (isInvalidated())
- return Invalidated;
+ return SaveStatus.Invalidated;
+
+ if (route != Full)
+ return SaveStatus.NotDefined;
if (definition == DefinitionUnknown)
- return executeAt.isDecidedAndKnownToExecute() ? ExecuteAtOnly : Nothing;
+ {
+ if (executeAt.isDecidedAndKnownToExecute())
+ return SaveStatus.PreCommitted;
+ return SaveStatus.NotDefined;
+ }
KnownExecuteAt executeAt = this.executeAt;
if (!executeAt.isDecidedAndKnownToExecute())
- return DefinitionOnly;
+ return SaveStatus.PreAccepted;
// cannot propagate proposed deps; and cannot propagate known deps without executeAt
KnownDeps deps = this.deps;
if (!deps.hasDecidedDeps())
- return SaveStatus.PreCommittedWithDefinition.known;
+ return SaveStatus.PreCommittedWithDefinition;
switch (outcome)
{
... 11060 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org