You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "Cyrill (via GitHub)" <gi...@apache.org> on 2023/11/21 14:06:45 UTC

[PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Cyrill opened a new pull request, #2856:
URL: https://github.com/apache/ignite-3/pull/2856

   Thank you for submitting the pull request.
   
   https://issues.apache.org/jira/browse/IGNITE-20735
   
   To streamline the review process of the patch and ensure better code quality
   we ask both an author and a reviewer to verify the following:
   
   ### The Review Checklist
   - [ ] **Formal criteria:** TC status, codestyle, mandatory documentation. Also make sure to complete the following:  
   \- There is a single JIRA ticket related to the pull request.  
   \- The web-link to the pull request is attached to the JIRA ticket.  
   \- The JIRA ticket has the Patch Available state.  
   \- The description of the JIRA ticket explains WHAT was made, WHY and HOW.  
   \- The pull request title is treated as the final commit message. The following pattern must be used: IGNITE-XXXX Change summary where XXXX - number of JIRA issue.
   - [ ] **Design:** new code conforms with the design principles of the components it is added to.
   - [ ] **Patch quality:** patch cannot be split into smaller pieces, its size must be reasonable.
   - [ ] **Code quality:** code is clean and readable, necessary developer documentation is added if needed.
   - [ ] **Tests code quality:** test set covers positive/negative scenarios, happy/edge cases. Tests are effective in terms of execution time and resources.
   
   ### Notes
   - [Apache Ignite Coding Guidelines](https://cwiki.apache.org/confluence/display/IGNITE/Java+Code+Style+Guide)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1425101034


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -505,46 +506,74 @@ private CompletableFuture<?> processRequest(ReplicaRequest request, @Nullable Bo
      * @param request Tx recovery request.
      * @return The future is complete when the transaction state is finalized.
      */
-    private CompletableFuture<Void> processTxRecoveryMessage(TxRecoveryMessage request) {
+    private CompletableFuture<Void> processTxRecoveryMessage(TxRecoveryMessage request, String senderId) {
         UUID txId = request.txId();
 
         TxMeta txMeta = txStateStorage.get(txId);
 
         // Check whether a transaction has already been finished.
-        boolean transactionAlreadyFinished = txMeta != null && isFinalState(txMeta.txState());
+        if (txMeta != null && isFinalState(txMeta.txState())) {
+            return recoverFinishedTx(txId, txMeta)
+                    // If the sender has sent a recovery message, it failed to handle it on its own,
+                    // so sending cleanup to the sender for the transaction we know is finished.
+                    .whenComplete((v, ex) -> runCleanupOnNode(txId, senderId));
+        }
 
-        if (transactionAlreadyFinished) {
+        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+
+        return triggerTxRecovery(txId, senderId);
+    }
+
+    private CompletableFuture<Void> recoverFinishedTx(UUID txId, TxMeta txMeta) {
+        if (txMeta.locksReleased() || txMeta.enlistedPartitions().isEmpty()) {
+            // Nothing to do if the locks have been released already or there are no enlistedPartitions available.
             return nullCompletedFuture();
         }
 
-        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+        // Otherwise run a cleanup on the known set of partitions.
+        return (CompletableFuture<Void>) durableCleanup(txId, txMeta);
+    }
+
+    /**
+     * Run cleanup on a node.
+     *
+     * @param txId Transaction id.
+     * @param nodeId Node id (inconsistent).
+     */
+    private CompletableFuture<Void> runCleanupOnNode(UUID txId, String nodeId) {
+        // Get node id of the sender to send back cleanup requests.
+        String nodeConsistentId = clusterNodeResolver.getConsistentIdById(nodeId);
+
+        return nodeConsistentId == null ? nullCompletedFuture() : txManager.cleanup(nodeConsistentId, txId);
+    }
 
-        return triggerTxRecovery(txId)
-                .thenApply(v -> null);
+    private CompletableFuture<Void> triggerTxRecovery(UUID txId, String senderId) {
+        // If the transaction state is pending, then the transaction should be rolled back,
+        // meaning that the state is changed to aborted and a corresponding cleanup request
+        // is sent in a common durable manner to a partition that have initiated recovery.
+        return txManager.finish(
+                        new HybridTimestampTracker(),
+                        replicationGroupId,
+                        false,
+                        Map.of(replicationGroupId, 0L), // term is not required for the rollback.
+                        txId
+                )
+                .thenCompose(unused -> runCleanupOnNode(txId, senderId));
     }
 
     /**
      * Starts tx recovery process. Returns the future containing transaction meta with its final state that completes
      * when the recovery is completed.
      *
      * @param txId Transaction id.
+     * @param txStateMeta Transaction meta.
+     * @param senderId Sender inconsistent id.
      * @return Tx recovery future, or failed future if the tx recovery is not possible.
      */
-    private CompletableFuture<TransactionMeta> triggerTxRecovery(UUID txId) {
-        // TODO: IGNITE-20735 Implement initiate recovery handling logic. This part has to be fully rewritten.
-        TxStateMeta txStateMeta = txManager.stateMeta(txId);
-
+    private CompletableFuture<TransactionMeta> triggerTxRecoveryWithState(UUID txId, @Nullable TxStateMeta txStateMeta, String senderId) {
         if (txStateMeta == null || txStateMeta.txState() == ABANDONED) {
-            txStateStorage.put(txId, new TxMeta(ABORTED, List.of(), null));
-            return completedFuture(
-                    txManager.updateTxMeta(txId, old -> {
-                        if (old == null) {
-                            return new TxStateMeta(ABORTED, null, replicationGroupId, null);
-                        } else {
-                            return new TxStateMeta(ABORTED, old.txCoordinatorId(), replicationGroupId, null);
-                        }
-                    })
-            );
+            return triggerTxRecovery(txId, senderId)
+                    .thenApply(v -> txStateStorage.get(txId));

Review Comment:
   There are cases when we are on the commit partition primary which is not a leader, so if this state in tx state storage had been modified by the concurrent recovery process, the finish future from our recovery could fail but tx state could be not replicated yet:
   - recovery A (current): start txManager.finish()
   - recovery B (concurrent): start txManager.finish(), finishes successfully, but the final tx state is applied only on leader (which is some remote node)
   - recovery A: txManager.finish() is failed due to recovery B, but local instance of tx state storage still doesn't have an actual tx meta.
   
   In this case we can check the local tx state map (because the concurrent recovery happens on the same commit partition replica) and it will be either FINISHING (if the concurrent process still wasn't able to put the final state) or some final state, and in case of FINISHING we can return the finish future. What do you think?



##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -505,46 +506,74 @@ private CompletableFuture<?> processRequest(ReplicaRequest request, @Nullable Bo
      * @param request Tx recovery request.
      * @return The future is complete when the transaction state is finalized.
      */
-    private CompletableFuture<Void> processTxRecoveryMessage(TxRecoveryMessage request) {
+    private CompletableFuture<Void> processTxRecoveryMessage(TxRecoveryMessage request, String senderId) {
         UUID txId = request.txId();
 
         TxMeta txMeta = txStateStorage.get(txId);
 
         // Check whether a transaction has already been finished.
-        boolean transactionAlreadyFinished = txMeta != null && isFinalState(txMeta.txState());
+        if (txMeta != null && isFinalState(txMeta.txState())) {
+            return recoverFinishedTx(txId, txMeta)
+                    // If the sender has sent a recovery message, it failed to handle it on its own,
+                    // so sending cleanup to the sender for the transaction we know is finished.
+                    .whenComplete((v, ex) -> runCleanupOnNode(txId, senderId));
+        }
 
-        if (transactionAlreadyFinished) {
+        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+
+        return triggerTxRecovery(txId, senderId);
+    }
+
+    private CompletableFuture<Void> recoverFinishedTx(UUID txId, TxMeta txMeta) {
+        if (txMeta.locksReleased() || txMeta.enlistedPartitions().isEmpty()) {
+            // Nothing to do if the locks have been released already or there are no enlistedPartitions available.
             return nullCompletedFuture();
         }
 
-        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+        // Otherwise run a cleanup on the known set of partitions.
+        return (CompletableFuture<Void>) durableCleanup(txId, txMeta);
+    }
+
+    /**
+     * Run cleanup on a node.
+     *
+     * @param txId Transaction id.
+     * @param nodeId Node id (inconsistent).
+     */
+    private CompletableFuture<Void> runCleanupOnNode(UUID txId, String nodeId) {
+        // Get node id of the sender to send back cleanup requests.
+        String nodeConsistentId = clusterNodeResolver.getConsistentIdById(nodeId);
+
+        return nodeConsistentId == null ? nullCompletedFuture() : txManager.cleanup(nodeConsistentId, txId);
+    }
 
-        return triggerTxRecovery(txId)
-                .thenApply(v -> null);
+    private CompletableFuture<Void> triggerTxRecovery(UUID txId, String senderId) {
+        // If the transaction state is pending, then the transaction should be rolled back,
+        // meaning that the state is changed to aborted and a corresponding cleanup request
+        // is sent in a common durable manner to a partition that have initiated recovery.
+        return txManager.finish(
+                        new HybridTimestampTracker(),
+                        replicationGroupId,
+                        false,
+                        Map.of(replicationGroupId, 0L), // term is not required for the rollback.
+                        txId
+                )
+                .thenCompose(unused -> runCleanupOnNode(txId, senderId));
     }
 
     /**
      * Starts tx recovery process. Returns the future containing transaction meta with its final state that completes
      * when the recovery is completed.
      *
      * @param txId Transaction id.
+     * @param txStateMeta Transaction meta.
+     * @param senderId Sender inconsistent id.
      * @return Tx recovery future, or failed future if the tx recovery is not possible.
      */
-    private CompletableFuture<TransactionMeta> triggerTxRecovery(UUID txId) {
-        // TODO: IGNITE-20735 Implement initiate recovery handling logic. This part has to be fully rewritten.
-        TxStateMeta txStateMeta = txManager.stateMeta(txId);
-
+    private CompletableFuture<TransactionMeta> triggerTxRecoveryWithState(UUID txId, @Nullable TxStateMeta txStateMeta, String senderId) {
         if (txStateMeta == null || txStateMeta.txState() == ABANDONED) {

Review Comment:
   Why do you need this condition? this method is called when `txStateMeta == null || txStateMeta.txState() == ABANDONED || txCoordinatorIsDead()` so here you just eliminate the last condition and return the completed future when tx state is pending but coordinator is missing.



##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -507,22 +514,59 @@ private CompletableFuture<?> processRequest(ReplicaRequest request, @Nullable Bo
      * @param request Tx recovery request.
      * @return The future is complete when the transaction state is finalized.
      */
-    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request) {
+    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request, String senderId) {
         UUID txId = request.txId();
 
         TxMeta txMeta = txStateStorage.get(txId);
 
         // Check whether a transaction has already been finished.
-        boolean transactionAlreadyFinished = txMeta != null && isFinalState(txMeta.txState());
+        if (txMeta != null && isFinalState(txMeta.txState())) {
+            return recoverFinishedTx(txId, txMeta)
+                    // If the sender has sent a recovery message, it failed to handle it on its own,
+                    // so sending cleanup to the sender for the transaction we know is finished.
+                    .thenCompose(v -> runCleanupOnNode(txId, senderId));
+        }
 
-        if (transactionAlreadyFinished) {
+        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+
+        return triggerTxRecovery(txId, senderId);
+    }
+
+    private CompletableFuture<Void> recoverFinishedTx(UUID txId, TxMeta txMeta) {
+        if (txMeta.locksReleased() || txMeta.enlistedPartitions().isEmpty()) {
+            // Nothing to do if the locks have been released already or there are no enlistedPartitions available.
             return nullCompletedFuture();
         }
 
-        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+        // Otherwise run a cleanup on the known set of partitions.
+        return (CompletableFuture<Void>) durableCleanup(txId, txMeta);
+    }
 
-        // TODO: IGNITE-20735 Implement initiate recovery handling logic.
-        return nullCompletedFuture();
+    /**
+     * Run cleanup on a node.
+     *
+     * @param txId Transaction id.
+     * @param nodeId Node id (inconsistent).
+     */
+    private CompletableFuture<Void> runCleanupOnNode(UUID txId, String nodeId) {
+        // Get node id of the sender to send back cleanup requests.
+        String nodeConsistentId = clusterNodeResolver.getConsistentIdById(nodeId);
+
+        return nodeConsistentId == null ? nullCompletedFuture() : txManager.cleanup(nodeConsistentId, txId);
+    }
+
+    private CompletableFuture<Void> triggerTxRecovery(UUID txId, String senderId) {
+        // If the transaction state is pending, then the transaction should be rolled back,
+        // meaning that the state is changed to aborted and a corresponding cleanup request
+        // is sent in a common durable manner to a partition that have initiated recovery.
+        return txManager.finish(
+                        new HybridTimestampTracker(),
+                        replicationGroupId,
+                        false,
+                        Map.of(replicationGroupId, 0L), // term is not required for the rollback.
+                        txId
+                )
+                .thenCompose(unused -> runCleanupOnNode(txId, senderId));

Review Comment:
   maybe I don't get something but there is still thenCompose



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1423693247


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -507,22 +514,59 @@ private CompletableFuture<?> processRequest(ReplicaRequest request, @Nullable Bo
      * @param request Tx recovery request.
      * @return The future is complete when the transaction state is finalized.
      */
-    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request) {
+    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request, String senderId) {
         UUID txId = request.txId();
 
         TxMeta txMeta = txStateStorage.get(txId);
 
         // Check whether a transaction has already been finished.
-        boolean transactionAlreadyFinished = txMeta != null && isFinalState(txMeta.txState());
+        if (txMeta != null && isFinalState(txMeta.txState())) {
+            return recoverFinishedTx(txId, txMeta)
+                    // If the sender has sent a recovery message, it failed to handle it on its own,
+                    // so sending cleanup to the sender for the transaction we know is finished.
+                    .thenCompose(v -> runCleanupOnNode(txId, senderId));
+        }
 
-        if (transactionAlreadyFinished) {
+        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+
+        return triggerTxRecovery(txId, senderId);
+    }
+
+    private CompletableFuture<Void> recoverFinishedTx(UUID txId, TxMeta txMeta) {
+        if (txMeta.locksReleased() || txMeta.enlistedPartitions().isEmpty()) {
+            // Nothing to do if the locks have been released already or there are no enlistedPartitions available.
             return nullCompletedFuture();
         }
 
-        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+        // Otherwise run a cleanup on the known set of partitions.
+        return (CompletableFuture<Void>) durableCleanup(txId, txMeta);
+    }
 
-        // TODO: IGNITE-20735 Implement initiate recovery handling logic.
-        return nullCompletedFuture();
+    /**
+     * Run cleanup on a node.
+     *
+     * @param txId Transaction id.
+     * @param nodeId Node id (inconsistent).
+     */
+    private CompletableFuture<Void> runCleanupOnNode(UUID txId, String nodeId) {
+        // Get node id of the sender to send back cleanup requests.
+        String nodeConsistentId = clusterNodeResolver.getConsistentIdById(nodeId);
+
+        return nodeConsistentId == null ? nullCompletedFuture() : txManager.cleanup(nodeConsistentId, txId);
+    }
+
+    private CompletableFuture<Void> triggerTxRecovery(UUID txId, String senderId) {
+        // If the transaction state is pending, then the transaction should be rolled back,
+        // meaning that the state is changed to aborted and a corresponding cleanup request
+        // is sent in a common durable manner to a partition that have initiated recovery.
+        return txManager.finish(
+                        new HybridTimestampTracker(),
+                        replicationGroupId,
+                        false,
+                        Map.of(replicationGroupId, 0L), // term is not required for the rollback.
+                        txId
+                )
+                .thenCompose(unused -> runCleanupOnNode(txId, senderId));

Review Comment:
   I think, there is a possibility of returning a failed future from `txManager.finish()` due to a race with another recovery process for the same transaction. Maybe there should be `whenComplete` here.



##########
modules/sql-engine/src/test/java/org/apache/ignite/internal/sql/engine/exec/rel/TableScanNodeExecutionTest.java:
##########
@@ -207,7 +208,17 @@ private static class TestInternalTableImpl extends InternalTableImpl {
                     1,
                     Int2ObjectMaps.singleton(0, mock(RaftGroupService.class)),
                     PART_CNT,
-                    addr -> mock(ClusterNode.class),
+                    new ClusterNodeResolver() {
+                        @Override
+                        public ClusterNode getById(String id) {
+                            return mock(ClusterNode.class);
+                        }
+
+                        @Override
+                        public ClusterNode getByConsistentId(String consistentId) {
+                            return mock(ClusterNode.class);
+                        }
+                    },

Review Comment:
   I see too many of similar snippets, lets create a test implementation of ClusterNodeResolver.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1427187708


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -795,28 +813,50 @@ private CompletableFuture<TransactionMeta> processTxStateCommitPartitionRequest(
      * @param txStateMeta Transaction meta.
      * @return Tx recovery future, or completed future if the recovery isn't needed, or failed future if the recovery is not possible.
      */
-    private CompletableFuture<TransactionMeta> triggerTxRecoveryOnTxStateResolutionIfNeeded(UUID txId, @Nullable TxStateMeta txStateMeta) {
-        assert txStateMeta == null || !isFinalState(txStateMeta.txState()) : "Unexpected transaction state: " + txStateMeta;
+    private CompletableFuture<TransactionMeta> triggerTxRecoveryOnTxStateResolutionIfNeeded(
+            UUID txId,
+            @Nullable TxStateMeta txStateMeta
+    ) {
+        // The state is either null or PENDING or ABANDONED, other states have been filtered out previously.
+        assert txStateMeta == null || txStateMeta.txState() == PENDING || txStateMeta.txState() == ABANDONED
+                : "Unexpected transaction state: " + txStateMeta;
 
         TxMeta txMeta = txStateStorage.get(txId);
 
         if (txMeta == null) {
             // This means the transaction is pending and we should trigger the recovery if there is no tx coordinator in topology.
             if (txStateMeta == null
                     || txStateMeta.txState() == ABANDONED
+                    || txStateMeta.txCoordinatorId() == null
                     || clusterNodeResolver.getById(txStateMeta.txCoordinatorId()) == null) {
                 // This means that primary replica for commit partition has changed, since the local node doesn't have the volatile tx
                 // state; and there is no final tx state in txStateStorage, or the tx coordinator left the cluster. But we can assume
                 // that as the coordinator (or information about it) is missing, there is  no need to wait a finish request from
                 // tx coordinator, the transaction can't be committed at all.
-                return triggerTxRecovery(txId);
+                return triggerTxRecovery(txId, localNode.id())
+                        .handle((v, ex) -> {
+                            TransactionMeta transactionMeta = txManager.stateMeta(txId);
+
+                            assert transactionMeta != null;
+
+                            if (transactionMeta.txState() == FINISHING) {
+                                return ((TxStateMetaFinishing) transactionMeta).txFinishFuture();
+                            }
+
+                            assert transactionMeta.txState() == ABORTED;

Review Comment:
   removed all checks here - they are no longer required.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1430480760


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/raft/PartitionListener.java:
##########
@@ -576,7 +583,11 @@ private static void onTxStateStorageCasFail(UUID txId, TxMeta txMetaBeforeCas, T
                 txMetaToSet
         );
 
-        IgniteInternalException stateChangeException = new IgniteInternalException(TX_UNEXPECTED_STATE_ERR, errorMsg);
+        IgniteInternalException stateChangeException =
+                new UnexpectedTransactionStateException(
+                        errorMsg,
+                        new TransactionResult(txMetaBeforeCas.txState(), txMetaBeforeCas.commitTimestamp())

Review Comment:
   sorry, we are in FSM here. Should be ok.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "vldpyatkov (via GitHub)" <gi...@apache.org>.
vldpyatkov merged PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1422416579


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/TransactionStateResolver.java:
##########
@@ -129,6 +129,18 @@ public void start() {
         });
     }
 
+    /**
+     * Get consistent id by node id.
+     *
+     * @param id Node inconsistent id.
+     * @return Node consistent id.
+     */
+    public @Nullable String consistentId(String id) {

Review Comment:
   it was a temporary solution. replaced with a proper one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1427189908


##########
modules/api/src/main/java/org/apache/ignite/lang/ErrorGroups.java:
##########
@@ -354,6 +354,9 @@ public static class Transactions {
 
         /** Coordinator tries to commit a transaction that has already been aborted. */
         public static final int TX_WAS_ABORTED_ERR = TX_ERR_GROUP.registerErrorCode((short) 15);
+
+        /** Coordinator tries to abort a transaction that has already been committed. */
+        public static final int TX_WAS_COMMITTED_ERR = TX_ERR_GROUP.registerErrorCode((short) 16);

Review Comment:
   I started to doubt that we need both TX_WAS_ABORTED_ERR and TX_WAS_COMMITTED_ERR, isn't TX_UNEXPECTED_STATE_ERR enough for us?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1427196009


##########
modules/api/src/main/java/org/apache/ignite/lang/ErrorGroups.java:
##########
@@ -354,6 +354,9 @@ public static class Transactions {
 
         /** Coordinator tries to commit a transaction that has already been aborted. */
         public static final int TX_WAS_ABORTED_ERR = TX_ERR_GROUP.registerErrorCode((short) 15);
+
+        /** Coordinator tries to abort a transaction that has already been committed. */
+        public static final int TX_WAS_COMMITTED_ERR = TX_ERR_GROUP.registerErrorCode((short) 16);

Review Comment:
   I am not sure, maybe it still can be useful for some cases of parallel recovery to see the actual state in FSM



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1430480448


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/raft/PartitionListener.java:
##########
@@ -576,7 +583,11 @@ private static void onTxStateStorageCasFail(UUID txId, TxMeta txMetaBeforeCas, T
                 txMetaToSet
         );
 
-        IgniteInternalException stateChangeException = new IgniteInternalException(TX_UNEXPECTED_STATE_ERR, errorMsg);
+        IgniteInternalException stateChangeException =
+                new UnexpectedTransactionStateException(
+                        errorMsg,
+                        new TransactionResult(txMetaBeforeCas.txState(), txMetaBeforeCas.commitTimestamp())

Review Comment:
   Agreed, that it is impossible due to the fact that the whole method is actually executed in RAFT



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1423721383


##########
modules/sql-engine/src/test/java/org/apache/ignite/internal/sql/engine/exec/rel/TableScanNodeExecutionTest.java:
##########
@@ -207,7 +208,17 @@ private static class TestInternalTableImpl extends InternalTableImpl {
                     1,
                     Int2ObjectMaps.singleton(0, mock(RaftGroupService.class)),
                     PART_CNT,
-                    addr -> mock(ClusterNode.class),
+                    new ClusterNodeResolver() {
+                        @Override
+                        public ClusterNode getById(String id) {
+                            return mock(ClusterNode.class);
+                        }
+
+                        @Override
+                        public ClusterNode getByConsistentId(String consistentId) {
+                            return mock(ClusterNode.class);
+                        }
+                    },

Review Comment:
   done in IGNITE-21060



##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -507,22 +514,59 @@ private CompletableFuture<?> processRequest(ReplicaRequest request, @Nullable Bo
      * @param request Tx recovery request.
      * @return The future is complete when the transaction state is finalized.
      */
-    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request) {
+    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request, String senderId) {
         UUID txId = request.txId();
 
         TxMeta txMeta = txStateStorage.get(txId);
 
         // Check whether a transaction has already been finished.
-        boolean transactionAlreadyFinished = txMeta != null && isFinalState(txMeta.txState());
+        if (txMeta != null && isFinalState(txMeta.txState())) {
+            return recoverFinishedTx(txId, txMeta)
+                    // If the sender has sent a recovery message, it failed to handle it on its own,
+                    // so sending cleanup to the sender for the transaction we know is finished.
+                    .thenCompose(v -> runCleanupOnNode(txId, senderId));
+        }
 
-        if (transactionAlreadyFinished) {
+        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+
+        return triggerTxRecovery(txId, senderId);
+    }
+
+    private CompletableFuture<Void> recoverFinishedTx(UUID txId, TxMeta txMeta) {
+        if (txMeta.locksReleased() || txMeta.enlistedPartitions().isEmpty()) {
+            // Nothing to do if the locks have been released already or there are no enlistedPartitions available.
             return nullCompletedFuture();
         }
 
-        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+        // Otherwise run a cleanup on the known set of partitions.
+        return (CompletableFuture<Void>) durableCleanup(txId, txMeta);
+    }
 
-        // TODO: IGNITE-20735 Implement initiate recovery handling logic.
-        return nullCompletedFuture();
+    /**
+     * Run cleanup on a node.
+     *
+     * @param txId Transaction id.
+     * @param nodeId Node id (inconsistent).
+     */
+    private CompletableFuture<Void> runCleanupOnNode(UUID txId, String nodeId) {
+        // Get node id of the sender to send back cleanup requests.
+        String nodeConsistentId = clusterNodeResolver.getConsistentIdById(nodeId);
+
+        return nodeConsistentId == null ? nullCompletedFuture() : txManager.cleanup(nodeConsistentId, txId);
+    }
+
+    private CompletableFuture<Void> triggerTxRecovery(UUID txId, String senderId) {
+        // If the transaction state is pending, then the transaction should be rolled back,
+        // meaning that the state is changed to aborted and a corresponding cleanup request
+        // is sent in a common durable manner to a partition that have initiated recovery.
+        return txManager.finish(
+                        new HybridTimestampTracker(),
+                        replicationGroupId,
+                        false,
+                        Map.of(replicationGroupId, 0L), // term is not required for the rollback.
+                        txId
+                )
+                .thenCompose(unused -> runCleanupOnNode(txId, senderId));

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1422217280


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -507,7 +508,12 @@ private CompletableFuture<?> processRequest(ReplicaRequest request, @Nullable Bo
      * @param request Tx recovery request.
      * @return The future is complete when the transaction state is finalized.
      */
-    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request) {
+    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request, String senderId) {
+        // Get node id of the sender to send back cleanup requests.
+        String nodeConsistentId = transactionStateResolver.consistentId(senderId);
+
+        assert nodeConsistentId != null;

Review Comment:
   here we can have assertion errors if the node left topology



##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -507,7 +508,12 @@ private CompletableFuture<?> processRequest(ReplicaRequest request, @Nullable Bo
      * @param request Tx recovery request.
      * @return The future is complete when the transaction state is finalized.
      */
-    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request) {
+    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request, String senderId) {

Review Comment:
   could you please create a separate method like `private CompletableFuture<TransactionMeta> triggerTxRecovery(UUID txId)` for this, because it is needed to trigger the recovery from tx state resolution process.



##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/TransactionStateResolver.java:
##########
@@ -129,6 +129,18 @@ public void start() {
         });
     }
 
+    /**
+     * Get consistent id by node id.
+     *
+     * @param id Node inconsistent id.
+     * @return Node consistent id.
+     */
+    public @Nullable String consistentId(String id) {

Review Comment:
   its definitely not a task for tx state resolver, to resolve the node names by their ids



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1426916350


##########
modules/api/src/main/java/org/apache/ignite/lang/ErrorGroups.java:
##########
@@ -354,6 +354,9 @@ public static class Transactions {
 
         /** Coordinator tries to commit a transaction that has already been aborted. */
         public static final int TX_WAS_ABORTED_ERR = TX_ERR_GROUP.registerErrorCode((short) 15);
+
+        /** Coordinator tries to abort a transaction that has already been committed. */

Review Comment:
   ```suggestion
           /** An attempt to abort a transaction that has already been committed. */
   ```



##########
modules/transactions/src/main/java/org/apache/ignite/internal/tx/impl/TxManagerImpl.java:
##########
@@ -375,6 +398,18 @@ public CompletableFuture<Void> finish(
                 ));
     }
 
+    private static CompletableFuture<Void> checkTxOutcome(boolean commit, UUID txId, TransactionMeta stateMeta) {
+        if ((stateMeta.txState() == COMMITTED) == commit) {
+            return nullCompletedFuture();
+        }
+
+        int result = (stateMeta.txState() == COMMITTED) && !commit ? TX_WAS_COMMITTED_ERR : TX_WAS_ABORTED_ERR;
+
+        return CompletableFuture.failedFuture(new TransactionException(result,

Review Comment:
   ```suggestion
           int errorCode = (stateMeta.txState() == COMMITTED) && !commit ? TX_WAS_COMMITTED_ERR : TX_WAS_ABORTED_ERR;
   
           return CompletableFuture.failedFuture(new TransactionException(errorCode,
   ```



##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -795,28 +813,50 @@ private CompletableFuture<TransactionMeta> processTxStateCommitPartitionRequest(
      * @param txStateMeta Transaction meta.
      * @return Tx recovery future, or completed future if the recovery isn't needed, or failed future if the recovery is not possible.
      */
-    private CompletableFuture<TransactionMeta> triggerTxRecoveryOnTxStateResolutionIfNeeded(UUID txId, @Nullable TxStateMeta txStateMeta) {
-        assert txStateMeta == null || !isFinalState(txStateMeta.txState()) : "Unexpected transaction state: " + txStateMeta;
+    private CompletableFuture<TransactionMeta> triggerTxRecoveryOnTxStateResolutionIfNeeded(
+            UUID txId,
+            @Nullable TxStateMeta txStateMeta
+    ) {
+        // The state is either null or PENDING or ABANDONED, other states have been filtered out previously.
+        assert txStateMeta == null || txStateMeta.txState() == PENDING || txStateMeta.txState() == ABANDONED
+                : "Unexpected transaction state: " + txStateMeta;
 
         TxMeta txMeta = txStateStorage.get(txId);
 
         if (txMeta == null) {
             // This means the transaction is pending and we should trigger the recovery if there is no tx coordinator in topology.
             if (txStateMeta == null
                     || txStateMeta.txState() == ABANDONED
+                    || txStateMeta.txCoordinatorId() == null
                     || clusterNodeResolver.getById(txStateMeta.txCoordinatorId()) == null) {
                 // This means that primary replica for commit partition has changed, since the local node doesn't have the volatile tx
                 // state; and there is no final tx state in txStateStorage, or the tx coordinator left the cluster. But we can assume
                 // that as the coordinator (or information about it) is missing, there is  no need to wait a finish request from
                 // tx coordinator, the transaction can't be committed at all.
-                return triggerTxRecovery(txId);
+                return triggerTxRecovery(txId, localNode.id())
+                        .handle((v, ex) -> {
+                            TransactionMeta transactionMeta = txManager.stateMeta(txId);
+
+                            assert transactionMeta != null;
+
+                            if (transactionMeta.txState() == FINISHING) {
+                                return ((TxStateMetaFinishing) transactionMeta).txFinishFuture();
+                            }
+
+                            assert transactionMeta.txState() == ABORTED;

Review Comment:
   Why only aborted? There could be successful commit from coordinator.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1430457593


##########
modules/transactions/src/main/java/org/apache/ignite/internal/tx/TransactionResult.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.tx;
+
+import static org.apache.ignite.internal.hlc.HybridTimestamp.hybridTimestampToLong;
+import static org.apache.ignite.internal.hlc.HybridTimestamp.nullableHybridTimestamp;
+
+import java.io.Serializable;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Represents transaction result - the state and the commit timestamp.

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1430385186


##########
modules/transactions/src/main/java/org/apache/ignite/internal/tx/impl/TxManagerImpl.java:
##########
@@ -439,20 +472,26 @@ private CompletableFuture<Void> durableFinish(
                     if (ex != null) {
                         Throwable cause = ExceptionUtils.unwrapCause(ex);
 
-                        if (cause instanceof TransactionException) {
-                            TransactionException transactionException = (TransactionException) cause;
+                        if (cause instanceof TransactionAlreadyFinishedException) {
+                            TransactionAlreadyFinishedException transactionException = (TransactionAlreadyFinishedException) cause;
+
+                            TransactionResult result = transactionException.transactionResult();
 
-                            if (transactionException.code() == TX_WAS_ABORTED_ERR) {
-                                updateTxMeta(txId, old -> {
-                                    TxStateMeta txStateMeta = new TxStateMeta(ABORTED, old.txCoordinatorId(), commitPartition, null);
+                            updateTxMeta(txId, old -> {
+                                TxStateMeta txStateMeta =
+                                        new TxStateMeta(
+                                                result.transactionState(),
+                                                old.txCoordinatorId(),
+                                                commitPartition,
+                                                result.commitTimestamp()
+                                        );
 
-                                    txFinishFuture.complete(txStateMeta);
+                                txFinishFuture.complete(txStateMeta);

Review Comment:
   By the way, let's better move this future completion out of the meta updater (and hence from the map.compute closure)



##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/raft/PartitionListener.java:
##########
@@ -576,7 +583,11 @@ private static void onTxStateStorageCasFail(UUID txId, TxMeta txMetaBeforeCas, T
                 txMetaToSet
         );
 
-        IgniteInternalException stateChangeException = new IgniteInternalException(TX_UNEXPECTED_STATE_ERR, errorMsg);
+        IgniteInternalException stateChangeException =
+                new UnexpectedTransactionStateException(
+                        errorMsg,
+                        new TransactionResult(txMetaBeforeCas.txState(), txMetaBeforeCas.commitTimestamp())

Review Comment:
   txMetaBeforeCas can be null, still CAS can fail.



##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -1575,33 +1608,38 @@ private CompletableFuture<Void> finishAndCleanup(
             // The transaction is finished, but the locks are not released.
             // If we got here, it means we are retrying the finish request.
             // Let's make sure the desired state is valid.
-            // Tx logic does not allow to send a rollback over a finished transaction:
             // - The Coordinator calls use same tx state over retries, both abort and commit are possible.
-            // - Server side recovery (which is not implemented yet) may only change tx state to aborted.
+            // - Server side recovery may only change tx state to aborted.
             // - The Coordinator itself should prevent user calls with different proposed state to the one,
-            //   that was already triggered (e.g. the client side -> txCoordinator.commitAsync(); txCoordinator.rollbackAsync())
-            //
-            // To sum it up, the possible states that a 'commit' is allowed to see:
+            //   that was already triggered (e.g. the client side -> txCoordinator.commitAsync(); txCoordinator.rollbackAsync()).
+            // - A coordinator might send a commit, then die, but the commit message might still arrive at the commit partition primary.
+            //   If it arrived with a delay, another node might come across a write intent/lock from that tx
+            //   and realize that the coordinator is no longer available and start tx recovery.
+            //   The original commit message might arrive later than the recovery one,
+            //   hence a 'commit over rollback' case.
+            // The possible states that a 'commit' is allowed to see:
             // - null (if it's the first change state attempt)
             // - committed (if it was already updated in the previous attempt)
             // - aborted (if it was aborted by the initiate recovery logic,
             //   though this is a very unlikely case because initiate recovery will only roll back the tx if coordinator is dead).
             //
             // Within 'roll back' it's allowed to see:
-            // - null
-            // - aborted
-            // Other combinations of states are not possible.
-
-            // First, throw an exception if we are trying to abort an already committed tx.
-            assert !(txMeta.txState() == COMMITTED && !commit) : "Not allowed to abort an already committed transaction.";
+            // - null (if it's the first change state attempt)
+            // - aborted  (if it was already updated in the previous attempt or the result of a concurrent recovery)
+            // - commit (if initiate recovery has started, but a delayed message from the coordinator finally arrived and executed earlier).
 
-            // If a 'commit' sees a tx in the ABORTED state (valid as per the explanation above), let the client know with an exception.
-            if (commit && txMeta.txState() == ABORTED) {
-                LOG.error("Failed to commit a transaction that is already aborted [txId={}].", txId);
+            // Let the client know a transaction has finished with a different outcome.
+            if (commit != (txMeta.txState() == COMMITTED)) {
+                LOG.error("Failed to finish a transaction that is already finished [txId={}, expectedState={}, actualState={}].",
+                        txId,
+                        commit ? COMMITTED : ABORTED,
+                        txMeta.txState()
+                );
 
-                throw new TransactionException(TX_WAS_ABORTED_ERR,
-                        "Failed to change the outcome of a finished transaction"
-                                + " [txId=" + txId + ", txState=" + txMeta.txState() + "].");
+                throw new TransactionAlreadyFinishedException(
+                        "Failed to change the outcome of a finished transaction [txId=" + txId + "].",
+                        new TransactionResult(txMeta.txState(), txMeta.commitTimestamp())
+                );

Review Comment:
   Don't you think that the exception message should include the transaction result?



##########
modules/transactions/src/main/java/org/apache/ignite/internal/tx/impl/TxManagerImpl.java:
##########
@@ -375,6 +397,17 @@ public CompletableFuture<Void> finish(
                 ));
     }
 
+    private static CompletableFuture<Void> checkTxOutcome(boolean commit, UUID txId, TransactionMeta stateMeta) {
+        if ((stateMeta.txState() == COMMITTED) == commit) {
+            return nullCompletedFuture();
+        }
+
+        return CompletableFuture.failedFuture(new TransactionAlreadyFinishedException(
+                "Failed to change the outcome of a finished transaction [txId=" + txId + "].",

Review Comment:
   Don't you think that the exception message should include the transaction result?



##########
modules/transactions/src/main/java/org/apache/ignite/internal/tx/TransactionResult.java:
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.tx;
+
+import static org.apache.ignite.internal.hlc.HybridTimestamp.hybridTimestampToLong;
+import static org.apache.ignite.internal.hlc.HybridTimestamp.nullableHybridTimestamp;
+
+import java.io.Serializable;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.jetbrains.annotations.Nullable;
+
+/**
+ * Represents transaction result - the state and the commit timestamp.

Review Comment:
   ```suggestion
    * Represents transaction final result - the state and the commit timestamp.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1422406992


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -516,13 +522,32 @@ private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage reques
         boolean transactionAlreadyFinished = txMeta != null && isFinalState(txMeta.txState());
 
         if (transactionAlreadyFinished) {
-            return nullCompletedFuture();
+            if (txMeta.locksReleased()) {
+                // Locks released - nothing to do.
+                return nullCompletedFuture();
+            }
+            // Locks were not released - send cleanup.
+            if (txMeta.enlistedPartitions().isEmpty()) {

Review Comment:
   by the way, if you call `txManager.finish` like below, you will never get empty list here: even in case of recovery you will send a tx finish request to commit partition with one partition in `enlistedPartitions` collection - the commit partition itself.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1422922016


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -507,7 +508,12 @@ private CompletableFuture<?> processRequest(ReplicaRequest request, @Nullable Bo
      * @param request Tx recovery request.
      * @return The future is complete when the transaction state is finalized.
      */
-    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request) {
+    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request, String senderId) {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1425912261


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -505,46 +506,74 @@ private CompletableFuture<?> processRequest(ReplicaRequest request, @Nullable Bo
      * @param request Tx recovery request.
      * @return The future is complete when the transaction state is finalized.
      */
-    private CompletableFuture<Void> processTxRecoveryMessage(TxRecoveryMessage request) {
+    private CompletableFuture<Void> processTxRecoveryMessage(TxRecoveryMessage request, String senderId) {
         UUID txId = request.txId();
 
         TxMeta txMeta = txStateStorage.get(txId);
 
         // Check whether a transaction has already been finished.
-        boolean transactionAlreadyFinished = txMeta != null && isFinalState(txMeta.txState());
+        if (txMeta != null && isFinalState(txMeta.txState())) {
+            return recoverFinishedTx(txId, txMeta)
+                    // If the sender has sent a recovery message, it failed to handle it on its own,
+                    // so sending cleanup to the sender for the transaction we know is finished.
+                    .whenComplete((v, ex) -> runCleanupOnNode(txId, senderId));
+        }
 
-        if (transactionAlreadyFinished) {
+        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+
+        return triggerTxRecovery(txId, senderId);
+    }
+
+    private CompletableFuture<Void> recoverFinishedTx(UUID txId, TxMeta txMeta) {
+        if (txMeta.locksReleased() || txMeta.enlistedPartitions().isEmpty()) {
+            // Nothing to do if the locks have been released already or there are no enlistedPartitions available.
             return nullCompletedFuture();
         }
 
-        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+        // Otherwise run a cleanup on the known set of partitions.
+        return (CompletableFuture<Void>) durableCleanup(txId, txMeta);
+    }
+
+    /**
+     * Run cleanup on a node.
+     *
+     * @param txId Transaction id.
+     * @param nodeId Node id (inconsistent).
+     */
+    private CompletableFuture<Void> runCleanupOnNode(UUID txId, String nodeId) {
+        // Get node id of the sender to send back cleanup requests.
+        String nodeConsistentId = clusterNodeResolver.getConsistentIdById(nodeId);
+
+        return nodeConsistentId == null ? nullCompletedFuture() : txManager.cleanup(nodeConsistentId, txId);
+    }
 
-        return triggerTxRecovery(txId)
-                .thenApply(v -> null);
+    private CompletableFuture<Void> triggerTxRecovery(UUID txId, String senderId) {
+        // If the transaction state is pending, then the transaction should be rolled back,
+        // meaning that the state is changed to aborted and a corresponding cleanup request
+        // is sent in a common durable manner to a partition that have initiated recovery.
+        return txManager.finish(
+                        new HybridTimestampTracker(),
+                        replicationGroupId,
+                        false,
+                        Map.of(replicationGroupId, 0L), // term is not required for the rollback.
+                        txId
+                )
+                .thenCompose(unused -> runCleanupOnNode(txId, senderId));
     }
 
     /**
      * Starts tx recovery process. Returns the future containing transaction meta with its final state that completes
      * when the recovery is completed.
      *
      * @param txId Transaction id.
+     * @param txStateMeta Transaction meta.
+     * @param senderId Sender inconsistent id.
      * @return Tx recovery future, or failed future if the tx recovery is not possible.
      */
-    private CompletableFuture<TransactionMeta> triggerTxRecovery(UUID txId) {
-        // TODO: IGNITE-20735 Implement initiate recovery handling logic. This part has to be fully rewritten.
-        TxStateMeta txStateMeta = txManager.stateMeta(txId);
-
+    private CompletableFuture<TransactionMeta> triggerTxRecoveryWithState(UUID txId, @Nullable TxStateMeta txStateMeta, String senderId) {
         if (txStateMeta == null || txStateMeta.txState() == ABANDONED) {
-            txStateStorage.put(txId, new TxMeta(ABORTED, List.of(), null));
-            return completedFuture(
-                    txManager.updateTxMeta(txId, old -> {
-                        if (old == null) {
-                            return new TxStateMeta(ABORTED, null, replicationGroupId, null);
-                        } else {
-                            return new TxStateMeta(ABORTED, old.txCoordinatorId(), replicationGroupId, null);
-                        }
-                    })
-            );
+            return triggerTxRecovery(txId, senderId)
+                    .thenApply(v -> txStateStorage.get(txId));

Review Comment:
   Actually, seems that it happens in another way. Any concurrent recovery may fail with assertion in txManager.finish here:
   `assert !tuple0.isTxFinishing() : "Transaction is already finished [id=" + uuid + "].";`
   We should return, for example, failed future with IllegalStateException instead of throwing an assertion error.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1422900267


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -516,13 +522,32 @@ private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage reques
         boolean transactionAlreadyFinished = txMeta != null && isFinalState(txMeta.txState());
 
         if (transactionAlreadyFinished) {
-            return nullCompletedFuture();
+            if (txMeta.locksReleased()) {
+                // Locks released - nothing to do.
+                return nullCompletedFuture();
+            }
+            // Locks were not released - send cleanup.
+            if (txMeta.enlistedPartitions().isEmpty()) {
+                // No enlisted partitions, send cleanup only back to the initiator.
+                return txManager.cleanup(nodeConsistentId, txId);
+            } else {
+                // If enlisted partitions exist, we can do the cleanup.
+                return (CompletableFuture<Void>) durableCleanup(txId, txMeta);
+            }
         }
 
         LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
 
-        // TODO: IGNITE-20735 Implement initiate recovery handling logic.
-        return nullCompletedFuture();
+        // If the transaction state is pending, then the transaction should be rolled back,
+        // meaning that the state is changed to aborted and a corresponding cleanup request
+        // is sent in a common durable manner to a partition that have initiated recovery.
+        return txManager.finish(

Review Comment:
   As discussed, no extra action needed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1422960656


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -507,7 +508,12 @@ private CompletableFuture<?> processRequest(ReplicaRequest request, @Nullable Bo
      * @param request Tx recovery request.
      * @return The future is complete when the transaction state is finalized.
      */
-    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request) {
+    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request, String senderId) {

Review Comment:
   Extracted to a method returning CompletableFuture<Void>. It's easy to apply a `thenApply` above it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1430456622


##########
modules/transactions/src/main/java/org/apache/ignite/internal/tx/impl/TxManagerImpl.java:
##########
@@ -375,6 +397,17 @@ public CompletableFuture<Void> finish(
                 ));
     }
 
+    private static CompletableFuture<Void> checkTxOutcome(boolean commit, UUID txId, TransactionMeta stateMeta) {
+        if ((stateMeta.txState() == COMMITTED) == commit) {
+            return nullCompletedFuture();
+        }
+
+        return CompletableFuture.failedFuture(new TransactionAlreadyFinishedException(
+                "Failed to change the outcome of a finished transaction [txId=" + txId + "].",

Review Comment:
   added it back



##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -1575,33 +1608,38 @@ private CompletableFuture<Void> finishAndCleanup(
             // The transaction is finished, but the locks are not released.
             // If we got here, it means we are retrying the finish request.
             // Let's make sure the desired state is valid.
-            // Tx logic does not allow to send a rollback over a finished transaction:
             // - The Coordinator calls use same tx state over retries, both abort and commit are possible.
-            // - Server side recovery (which is not implemented yet) may only change tx state to aborted.
+            // - Server side recovery may only change tx state to aborted.
             // - The Coordinator itself should prevent user calls with different proposed state to the one,
-            //   that was already triggered (e.g. the client side -> txCoordinator.commitAsync(); txCoordinator.rollbackAsync())
-            //
-            // To sum it up, the possible states that a 'commit' is allowed to see:
+            //   that was already triggered (e.g. the client side -> txCoordinator.commitAsync(); txCoordinator.rollbackAsync()).
+            // - A coordinator might send a commit, then die, but the commit message might still arrive at the commit partition primary.
+            //   If it arrived with a delay, another node might come across a write intent/lock from that tx
+            //   and realize that the coordinator is no longer available and start tx recovery.
+            //   The original commit message might arrive later than the recovery one,
+            //   hence a 'commit over rollback' case.
+            // The possible states that a 'commit' is allowed to see:
             // - null (if it's the first change state attempt)
             // - committed (if it was already updated in the previous attempt)
             // - aborted (if it was aborted by the initiate recovery logic,
             //   though this is a very unlikely case because initiate recovery will only roll back the tx if coordinator is dead).
             //
             // Within 'roll back' it's allowed to see:
-            // - null
-            // - aborted
-            // Other combinations of states are not possible.
-
-            // First, throw an exception if we are trying to abort an already committed tx.
-            assert !(txMeta.txState() == COMMITTED && !commit) : "Not allowed to abort an already committed transaction.";
+            // - null (if it's the first change state attempt)
+            // - aborted  (if it was already updated in the previous attempt or the result of a concurrent recovery)
+            // - commit (if initiate recovery has started, but a delayed message from the coordinator finally arrived and executed earlier).
 
-            // If a 'commit' sees a tx in the ABORTED state (valid as per the explanation above), let the client know with an exception.
-            if (commit && txMeta.txState() == ABORTED) {
-                LOG.error("Failed to commit a transaction that is already aborted [txId={}].", txId);
+            // Let the client know a transaction has finished with a different outcome.
+            if (commit != (txMeta.txState() == COMMITTED)) {
+                LOG.error("Failed to finish a transaction that is already finished [txId={}, expectedState={}, actualState={}].",
+                        txId,
+                        commit ? COMMITTED : ABORTED,
+                        txMeta.txState()
+                );
 
-                throw new TransactionException(TX_WAS_ABORTED_ERR,
-                        "Failed to change the outcome of a finished transaction"
-                                + " [txId=" + txId + ", txState=" + txMeta.txState() + "].");
+                throw new TransactionAlreadyFinishedException(
+                        "Failed to change the outcome of a finished transaction [txId=" + txId + "].",
+                        new TransactionResult(txMeta.txState(), txMeta.commitTimestamp())
+                );

Review Comment:
   added it back



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1426965688


##########
modules/api/src/main/java/org/apache/ignite/lang/ErrorGroups.java:
##########
@@ -354,6 +354,9 @@ public static class Transactions {
 
         /** Coordinator tries to commit a transaction that has already been aborted. */
         public static final int TX_WAS_ABORTED_ERR = TX_ERR_GROUP.registerErrorCode((short) 15);
+
+        /** Coordinator tries to abort a transaction that has already been committed. */
+        public static final int TX_WAS_COMMITTED_ERR = TX_ERR_GROUP.registerErrorCode((short) 16);

Review Comment:
   You should also check this code in `TxManagerImpl#durableFinish` to stop durable finish. By the way, seems we dont return the correct error code from `PartitionListener#onTxStateStorageCasFail` - there is `TX_UNEXPECTED_STATE_ERR` in any case, should be either `TX_WAS_ABORTED_ERR` or `TX_WAS_COMMITTED_ERR`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1422406992


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -516,13 +522,32 @@ private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage reques
         boolean transactionAlreadyFinished = txMeta != null && isFinalState(txMeta.txState());
 
         if (transactionAlreadyFinished) {
-            return nullCompletedFuture();
+            if (txMeta.locksReleased()) {
+                // Locks released - nothing to do.
+                return nullCompletedFuture();
+            }
+            // Locks were not released - send cleanup.
+            if (txMeta.enlistedPartitions().isEmpty()) {

Review Comment:
   by the way, if you call `txManager.finish` like below, you will never get empty list here: even in case of recovery you will send a tx finish request to commit partition with one partition in `enlistedPartitions` collection: the commit partition itself.



##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -516,13 +522,32 @@ private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage reques
         boolean transactionAlreadyFinished = txMeta != null && isFinalState(txMeta.txState());
 
         if (transactionAlreadyFinished) {
-            return nullCompletedFuture();
+            if (txMeta.locksReleased()) {
+                // Locks released - nothing to do.
+                return nullCompletedFuture();
+            }
+            // Locks were not released - send cleanup.
+            if (txMeta.enlistedPartitions().isEmpty()) {
+                // No enlisted partitions, send cleanup only back to the initiator.
+                return txManager.cleanup(nodeConsistentId, txId);
+            } else {
+                // If enlisted partitions exist, we can do the cleanup.
+                return (CompletableFuture<Void>) durableCleanup(txId, txMeta);
+            }
         }
 
         LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
 
-        // TODO: IGNITE-20735 Implement initiate recovery handling logic.
-        return nullCompletedFuture();
+        // If the transaction state is pending, then the transaction should be rolled back,
+        // meaning that the state is changed to aborted and a corresponding cleanup request
+        // is sent in a common durable manner to a partition that have initiated recovery.
+        return txManager.finish(

Review Comment:
   now we must check on PartitionListener side that the state that is already in txStateStorage, is not final, while doing CAS. If there is already a final state, the operation should be aborted.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1425675741


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -507,22 +514,59 @@ private CompletableFuture<?> processRequest(ReplicaRequest request, @Nullable Bo
      * @param request Tx recovery request.
      * @return The future is complete when the transaction state is finalized.
      */
-    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request) {
+    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request, String senderId) {
         UUID txId = request.txId();
 
         TxMeta txMeta = txStateStorage.get(txId);
 
         // Check whether a transaction has already been finished.
-        boolean transactionAlreadyFinished = txMeta != null && isFinalState(txMeta.txState());
+        if (txMeta != null && isFinalState(txMeta.txState())) {
+            return recoverFinishedTx(txId, txMeta)
+                    // If the sender has sent a recovery message, it failed to handle it on its own,
+                    // so sending cleanup to the sender for the transaction we know is finished.
+                    .thenCompose(v -> runCleanupOnNode(txId, senderId));
+        }
 
-        if (transactionAlreadyFinished) {
+        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+
+        return triggerTxRecovery(txId, senderId);
+    }
+
+    private CompletableFuture<Void> recoverFinishedTx(UUID txId, TxMeta txMeta) {
+        if (txMeta.locksReleased() || txMeta.enlistedPartitions().isEmpty()) {
+            // Nothing to do if the locks have been released already or there are no enlistedPartitions available.
             return nullCompletedFuture();
         }
 
-        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+        // Otherwise run a cleanup on the known set of partitions.
+        return (CompletableFuture<Void>) durableCleanup(txId, txMeta);
+    }
 
-        // TODO: IGNITE-20735 Implement initiate recovery handling logic.
-        return nullCompletedFuture();
+    /**
+     * Run cleanup on a node.
+     *
+     * @param txId Transaction id.
+     * @param nodeId Node id (inconsistent).
+     */
+    private CompletableFuture<Void> runCleanupOnNode(UUID txId, String nodeId) {
+        // Get node id of the sender to send back cleanup requests.
+        String nodeConsistentId = clusterNodeResolver.getConsistentIdById(nodeId);
+
+        return nodeConsistentId == null ? nullCompletedFuture() : txManager.cleanup(nodeConsistentId, txId);
+    }
+
+    private CompletableFuture<Void> triggerTxRecovery(UUID txId, String senderId) {
+        // If the transaction state is pending, then the transaction should be rolled back,
+        // meaning that the state is changed to aborted and a corresponding cleanup request
+        // is sent in a common durable manner to a partition that have initiated recovery.
+        return txManager.finish(
+                        new HybridTimestampTracker(),
+                        replicationGroupId,
+                        false,
+                        Map.of(replicationGroupId, 0L), // term is not required for the rollback.
+                        txId
+                )
+                .thenCompose(unused -> runCleanupOnNode(txId, senderId));

Review Comment:
   Bad rebase. fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1431034817


##########
modules/api/src/main/java/org/apache/ignite/lang/ErrorGroups.java:
##########
@@ -354,6 +354,9 @@ public static class Transactions {
 
         /** Coordinator tries to commit a transaction that has already been aborted. */
         public static final int TX_WAS_ABORTED_ERR = TX_ERR_GROUP.registerErrorCode((short) 15);
+
+        /** Coordinator tries to abort a transaction that has already been committed. */
+        public static final int TX_WAS_COMMITTED_ERR = TX_ERR_GROUP.registerErrorCode((short) 16);

Review Comment:
   agreed to change this part



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1430471459


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/raft/PartitionListener.java:
##########
@@ -576,7 +583,11 @@ private static void onTxStateStorageCasFail(UUID txId, TxMeta txMetaBeforeCas, T
                 txMetaToSet
         );
 
-        IgniteInternalException stateChangeException = new IgniteInternalException(TX_UNEXPECTED_STATE_ERR, errorMsg);
+        IgniteInternalException stateChangeException =
+                new UnexpectedTransactionStateException(
+                        errorMsg,
+                        new TransactionResult(txMetaBeforeCas.txState(), txMetaBeforeCas.commitTimestamp())

Review Comment:
   Does it come from the contract of this method or did you have a particular implementation in mind?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1422415956


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -507,7 +508,12 @@ private CompletableFuture<?> processRequest(ReplicaRequest request, @Nullable Bo
      * @param request Tx recovery request.
      * @return The future is complete when the transaction state is finalized.
      */
-    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request) {
+    private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage request, String senderId) {
+        // Get node id of the sender to send back cleanup requests.
+        String nodeConsistentId = transactionStateResolver.consistentId(senderId);
+
+        assert nodeConsistentId != null;

Review Comment:
   replaced with a nullcheck



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1422900597


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/TransactionStateResolver.java:
##########
@@ -129,6 +129,18 @@ public void start() {
         });
     }
 
+    /**
+     * Get consistent id by node id.
+     *
+     * @param id Node inconsistent id.
+     * @return Node consistent id.
+     */
+    public @Nullable String consistentId(String id) {

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1423001246


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -516,13 +522,32 @@ private CompletableFuture<Void> processTxRecoveryAction(TxRecoveryMessage reques
         boolean transactionAlreadyFinished = txMeta != null && isFinalState(txMeta.txState());
 
         if (transactionAlreadyFinished) {
-            return nullCompletedFuture();
+            if (txMeta.locksReleased()) {
+                // Locks released - nothing to do.
+                return nullCompletedFuture();
+            }
+            // Locks were not released - send cleanup.
+            if (txMeta.enlistedPartitions().isEmpty()) {

Review Comment:
   changed this part a bit



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1425912261


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -505,46 +506,74 @@ private CompletableFuture<?> processRequest(ReplicaRequest request, @Nullable Bo
      * @param request Tx recovery request.
      * @return The future is complete when the transaction state is finalized.
      */
-    private CompletableFuture<Void> processTxRecoveryMessage(TxRecoveryMessage request) {
+    private CompletableFuture<Void> processTxRecoveryMessage(TxRecoveryMessage request, String senderId) {
         UUID txId = request.txId();
 
         TxMeta txMeta = txStateStorage.get(txId);
 
         // Check whether a transaction has already been finished.
-        boolean transactionAlreadyFinished = txMeta != null && isFinalState(txMeta.txState());
+        if (txMeta != null && isFinalState(txMeta.txState())) {
+            return recoverFinishedTx(txId, txMeta)
+                    // If the sender has sent a recovery message, it failed to handle it on its own,
+                    // so sending cleanup to the sender for the transaction we know is finished.
+                    .whenComplete((v, ex) -> runCleanupOnNode(txId, senderId));
+        }
 
-        if (transactionAlreadyFinished) {
+        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+
+        return triggerTxRecovery(txId, senderId);
+    }
+
+    private CompletableFuture<Void> recoverFinishedTx(UUID txId, TxMeta txMeta) {
+        if (txMeta.locksReleased() || txMeta.enlistedPartitions().isEmpty()) {
+            // Nothing to do if the locks have been released already or there are no enlistedPartitions available.
             return nullCompletedFuture();
         }
 
-        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+        // Otherwise run a cleanup on the known set of partitions.
+        return (CompletableFuture<Void>) durableCleanup(txId, txMeta);
+    }
+
+    /**
+     * Run cleanup on a node.
+     *
+     * @param txId Transaction id.
+     * @param nodeId Node id (inconsistent).
+     */
+    private CompletableFuture<Void> runCleanupOnNode(UUID txId, String nodeId) {
+        // Get node id of the sender to send back cleanup requests.
+        String nodeConsistentId = clusterNodeResolver.getConsistentIdById(nodeId);
+
+        return nodeConsistentId == null ? nullCompletedFuture() : txManager.cleanup(nodeConsistentId, txId);
+    }
 
-        return triggerTxRecovery(txId)
-                .thenApply(v -> null);
+    private CompletableFuture<Void> triggerTxRecovery(UUID txId, String senderId) {
+        // If the transaction state is pending, then the transaction should be rolled back,
+        // meaning that the state is changed to aborted and a corresponding cleanup request
+        // is sent in a common durable manner to a partition that have initiated recovery.
+        return txManager.finish(
+                        new HybridTimestampTracker(),
+                        replicationGroupId,
+                        false,
+                        Map.of(replicationGroupId, 0L), // term is not required for the rollback.
+                        txId
+                )
+                .thenCompose(unused -> runCleanupOnNode(txId, senderId));
     }
 
     /**
      * Starts tx recovery process. Returns the future containing transaction meta with its final state that completes
      * when the recovery is completed.
      *
      * @param txId Transaction id.
+     * @param txStateMeta Transaction meta.
+     * @param senderId Sender inconsistent id.
      * @return Tx recovery future, or failed future if the tx recovery is not possible.
      */
-    private CompletableFuture<TransactionMeta> triggerTxRecovery(UUID txId) {
-        // TODO: IGNITE-20735 Implement initiate recovery handling logic. This part has to be fully rewritten.
-        TxStateMeta txStateMeta = txManager.stateMeta(txId);
-
+    private CompletableFuture<TransactionMeta> triggerTxRecoveryWithState(UUID txId, @Nullable TxStateMeta txStateMeta, String senderId) {
         if (txStateMeta == null || txStateMeta.txState() == ABANDONED) {
-            txStateStorage.put(txId, new TxMeta(ABORTED, List.of(), null));
-            return completedFuture(
-                    txManager.updateTxMeta(txId, old -> {
-                        if (old == null) {
-                            return new TxStateMeta(ABORTED, null, replicationGroupId, null);
-                        } else {
-                            return new TxStateMeta(ABORTED, old.txCoordinatorId(), replicationGroupId, null);
-                        }
-                    })
-            );
+            return triggerTxRecovery(txId, senderId)
+                    .thenApply(v -> txStateStorage.get(txId));

Review Comment:
   Actually, seems that it happens in a little bit different way. Any concurrent recovery may fail with assertion in txManager.finish here:
   `assert !tuple0.isTxFinishing() : "Transaction is already finished [id=" + uuid + "].";`
   We should return, for example, failed future with IllegalStateException instead of throwing an assertion error.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1427188316


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -505,46 +506,74 @@ private CompletableFuture<?> processRequest(ReplicaRequest request, @Nullable Bo
      * @param request Tx recovery request.
      * @return The future is complete when the transaction state is finalized.
      */
-    private CompletableFuture<Void> processTxRecoveryMessage(TxRecoveryMessage request) {
+    private CompletableFuture<Void> processTxRecoveryMessage(TxRecoveryMessage request, String senderId) {
         UUID txId = request.txId();
 
         TxMeta txMeta = txStateStorage.get(txId);
 
         // Check whether a transaction has already been finished.
-        boolean transactionAlreadyFinished = txMeta != null && isFinalState(txMeta.txState());
+        if (txMeta != null && isFinalState(txMeta.txState())) {
+            return recoverFinishedTx(txId, txMeta)
+                    // If the sender has sent a recovery message, it failed to handle it on its own,
+                    // so sending cleanup to the sender for the transaction we know is finished.
+                    .whenComplete((v, ex) -> runCleanupOnNode(txId, senderId));
+        }
 
-        if (transactionAlreadyFinished) {
+        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+
+        return triggerTxRecovery(txId, senderId);
+    }
+
+    private CompletableFuture<Void> recoverFinishedTx(UUID txId, TxMeta txMeta) {
+        if (txMeta.locksReleased() || txMeta.enlistedPartitions().isEmpty()) {
+            // Nothing to do if the locks have been released already or there are no enlistedPartitions available.
             return nullCompletedFuture();
         }
 
-        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+        // Otherwise run a cleanup on the known set of partitions.
+        return (CompletableFuture<Void>) durableCleanup(txId, txMeta);
+    }
+
+    /**
+     * Run cleanup on a node.
+     *
+     * @param txId Transaction id.
+     * @param nodeId Node id (inconsistent).
+     */
+    private CompletableFuture<Void> runCleanupOnNode(UUID txId, String nodeId) {
+        // Get node id of the sender to send back cleanup requests.
+        String nodeConsistentId = clusterNodeResolver.getConsistentIdById(nodeId);
+
+        return nodeConsistentId == null ? nullCompletedFuture() : txManager.cleanup(nodeConsistentId, txId);
+    }
 
-        return triggerTxRecovery(txId)
-                .thenApply(v -> null);
+    private CompletableFuture<Void> triggerTxRecovery(UUID txId, String senderId) {
+        // If the transaction state is pending, then the transaction should be rolled back,
+        // meaning that the state is changed to aborted and a corresponding cleanup request
+        // is sent in a common durable manner to a partition that have initiated recovery.
+        return txManager.finish(
+                        new HybridTimestampTracker(),
+                        replicationGroupId,
+                        false,
+                        Map.of(replicationGroupId, 0L), // term is not required for the rollback.
+                        txId
+                )
+                .thenCompose(unused -> runCleanupOnNode(txId, senderId));
     }
 
     /**
      * Starts tx recovery process. Returns the future containing transaction meta with its final state that completes
      * when the recovery is completed.
      *
      * @param txId Transaction id.
+     * @param txStateMeta Transaction meta.
+     * @param senderId Sender inconsistent id.
      * @return Tx recovery future, or failed future if the tx recovery is not possible.
      */
-    private CompletableFuture<TransactionMeta> triggerTxRecovery(UUID txId) {
-        // TODO: IGNITE-20735 Implement initiate recovery handling logic. This part has to be fully rewritten.
-        TxStateMeta txStateMeta = txManager.stateMeta(txId);
-
+    private CompletableFuture<TransactionMeta> triggerTxRecoveryWithState(UUID txId, @Nullable TxStateMeta txStateMeta, String senderId) {
         if (txStateMeta == null || txStateMeta.txState() == ABANDONED) {
-            txStateStorage.put(txId, new TxMeta(ABORTED, List.of(), null));
-            return completedFuture(
-                    txManager.updateTxMeta(txId, old -> {
-                        if (old == null) {
-                            return new TxStateMeta(ABORTED, null, replicationGroupId, null);
-                        } else {
-                            return new TxStateMeta(ABORTED, old.txCoordinatorId(), replicationGroupId, null);
-                        }
-                    })
-            );
+            return triggerTxRecovery(txId, senderId)
+                    .thenApply(v -> txStateStorage.get(txId));

Review Comment:
   Changed the implementation of `TxManager.finish`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1425676184


##########
modules/table/src/main/java/org/apache/ignite/internal/table/distributed/replicator/PartitionReplicaListener.java:
##########
@@ -505,46 +506,74 @@ private CompletableFuture<?> processRequest(ReplicaRequest request, @Nullable Bo
      * @param request Tx recovery request.
      * @return The future is complete when the transaction state is finalized.
      */
-    private CompletableFuture<Void> processTxRecoveryMessage(TxRecoveryMessage request) {
+    private CompletableFuture<Void> processTxRecoveryMessage(TxRecoveryMessage request, String senderId) {
         UUID txId = request.txId();
 
         TxMeta txMeta = txStateStorage.get(txId);
 
         // Check whether a transaction has already been finished.
-        boolean transactionAlreadyFinished = txMeta != null && isFinalState(txMeta.txState());
+        if (txMeta != null && isFinalState(txMeta.txState())) {
+            return recoverFinishedTx(txId, txMeta)
+                    // If the sender has sent a recovery message, it failed to handle it on its own,
+                    // so sending cleanup to the sender for the transaction we know is finished.
+                    .whenComplete((v, ex) -> runCleanupOnNode(txId, senderId));
+        }
 
-        if (transactionAlreadyFinished) {
+        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+
+        return triggerTxRecovery(txId, senderId);
+    }
+
+    private CompletableFuture<Void> recoverFinishedTx(UUID txId, TxMeta txMeta) {
+        if (txMeta.locksReleased() || txMeta.enlistedPartitions().isEmpty()) {
+            // Nothing to do if the locks have been released already or there are no enlistedPartitions available.
             return nullCompletedFuture();
         }
 
-        LOG.info("Orphan transaction has to be aborted [tx={}].", txId);
+        // Otherwise run a cleanup on the known set of partitions.
+        return (CompletableFuture<Void>) durableCleanup(txId, txMeta);
+    }
+
+    /**
+     * Run cleanup on a node.
+     *
+     * @param txId Transaction id.
+     * @param nodeId Node id (inconsistent).
+     */
+    private CompletableFuture<Void> runCleanupOnNode(UUID txId, String nodeId) {
+        // Get node id of the sender to send back cleanup requests.
+        String nodeConsistentId = clusterNodeResolver.getConsistentIdById(nodeId);
+
+        return nodeConsistentId == null ? nullCompletedFuture() : txManager.cleanup(nodeConsistentId, txId);
+    }
 
-        return triggerTxRecovery(txId)
-                .thenApply(v -> null);
+    private CompletableFuture<Void> triggerTxRecovery(UUID txId, String senderId) {
+        // If the transaction state is pending, then the transaction should be rolled back,
+        // meaning that the state is changed to aborted and a corresponding cleanup request
+        // is sent in a common durable manner to a partition that have initiated recovery.
+        return txManager.finish(
+                        new HybridTimestampTracker(),
+                        replicationGroupId,
+                        false,
+                        Map.of(replicationGroupId, 0L), // term is not required for the rollback.
+                        txId
+                )
+                .thenCompose(unused -> runCleanupOnNode(txId, senderId));
     }
 
     /**
      * Starts tx recovery process. Returns the future containing transaction meta with its final state that completes
      * when the recovery is completed.
      *
      * @param txId Transaction id.
+     * @param txStateMeta Transaction meta.
+     * @param senderId Sender inconsistent id.
      * @return Tx recovery future, or failed future if the tx recovery is not possible.
      */
-    private CompletableFuture<TransactionMeta> triggerTxRecovery(UUID txId) {
-        // TODO: IGNITE-20735 Implement initiate recovery handling logic. This part has to be fully rewritten.
-        TxStateMeta txStateMeta = txManager.stateMeta(txId);
-
+    private CompletableFuture<TransactionMeta> triggerTxRecoveryWithState(UUID txId, @Nullable TxStateMeta txStateMeta, String senderId) {
         if (txStateMeta == null || txStateMeta.txState() == ABANDONED) {

Review Comment:
   changed it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-20735 Implement initiate recovery handling logic [ignite-3]

Posted by "Cyrill (via GitHub)" <gi...@apache.org>.
Cyrill commented on code in PR #2856:
URL: https://github.com/apache/ignite-3/pull/2856#discussion_r1430460769


##########
modules/transactions/src/main/java/org/apache/ignite/internal/tx/impl/TxManagerImpl.java:
##########
@@ -439,20 +472,26 @@ private CompletableFuture<Void> durableFinish(
                     if (ex != null) {
                         Throwable cause = ExceptionUtils.unwrapCause(ex);
 
-                        if (cause instanceof TransactionException) {
-                            TransactionException transactionException = (TransactionException) cause;
+                        if (cause instanceof TransactionAlreadyFinishedException) {
+                            TransactionAlreadyFinishedException transactionException = (TransactionAlreadyFinishedException) cause;
+
+                            TransactionResult result = transactionException.transactionResult();
 
-                            if (transactionException.code() == TX_WAS_ABORTED_ERR) {
-                                updateTxMeta(txId, old -> {
-                                    TxStateMeta txStateMeta = new TxStateMeta(ABORTED, old.txCoordinatorId(), commitPartition, null);
+                            updateTxMeta(txId, old -> {
+                                TxStateMeta txStateMeta =
+                                        new TxStateMeta(
+                                                result.transactionState(),
+                                                old.txCoordinatorId(),
+                                                commitPartition,
+                                                result.commitTimestamp()
+                                        );
 
-                                    txFinishFuture.complete(txStateMeta);
+                                txFinishFuture.complete(txStateMeta);

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org