You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "denis-chudov (via GitHub)" <gi...@apache.org> on 2024/05/08 19:14:42 UTC

[PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

denis-chudov opened a new pull request, #3725:
URL: https://github.com/apache/ignite-3/pull/3725

   https://issues.apache.org/jira/browse/IGNITE-21881


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599726821


##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/impl/CommandIdGenerator.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.impl;
+
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Supplier;
+import org.apache.ignite.internal.metastorage.CommandId;
+import org.apache.ignite.internal.metastorage.dsl.MetaStorageMessagesFactory;
+
+/**
+ * Generates the command ids.
+ */
+public class CommandIdGenerator {
+    private static final MetaStorageMessagesFactory MSG_FACTORY = new MetaStorageMessagesFactory();
+
+    /** Supplies nodeId for transactionId generation. */

Review Comment:
   fixed.
   Regarding supplier - I was not sure that node id is always ready on meta storage start, there are some async operations on cluster service start



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599741805


##########
modules/metastorage/src/integrationTest/java/org/apache/ignite/internal/metastorage/impl/ItMetaStorageServicePersistenceTest.java:
##########
@@ -76,7 +76,8 @@ public void beforeFollowerStop(RaftGroupService service, RaftServer server) thro
 
         var clusterTime = new ClusterTimeImpl(followerNode.name(), new IgniteSpinBusyLock(), new HybridClockImpl());
 
-        metaStorage = new MetaStorageServiceImpl(followerNode.name(), service, new IgniteSpinBusyLock(), clusterTime);
+        metaStorage = new MetaStorageServiceImpl(followerNode.name(), service, new IgniteSpinBusyLock(), clusterTime,

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599731099


##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -278,4 +309,50 @@ boolean beforeApply(Command command) {
 
         return false;
     }
+
+    private static class IdempotentCommandCache {
+        @Nullable
+        final Serializable result;
+
+        final HybridTimestamp commandStartTime;

Review Comment:
   Why current time? Command start time is hybrid time. And `long` type can create an ambiguity. `HybridTimestamp` is more clear here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1600272385


##########
modules/metastorage/src/integrationTest/java/org/apache/ignite/internal/metastorage/impl/ItIdempotentCommandCacheTest.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.impl;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.internal.metastorage.dsl.Conditions.notExists;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.noop;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.put;
+import static org.apache.ignite.internal.network.utils.ClusterServiceTestUtils.clusterService;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.IgniteUtils.closeAll;
+import static org.apache.ignite.internal.util.IgniteUtils.startAsync;
+import static org.apache.ignite.internal.util.IgniteUtils.stopAsync;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.BiPredicate;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalTopologyService;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.failure.NoOpFailureProcessor;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridClockImpl;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.command.SyncTimeCommand;
+import org.apache.ignite.internal.metastorage.configuration.MetaStorageConfiguration;
+import org.apache.ignite.internal.metastorage.server.KeyValueStorage;
+import org.apache.ignite.internal.metastorage.server.persistence.RocksDbKeyValueStorage;
+import org.apache.ignite.internal.metrics.NoOpMetricManager;
+import org.apache.ignite.internal.network.ClusterService;
+import org.apache.ignite.internal.network.DefaultMessagingService;
+import org.apache.ignite.internal.network.NetworkMessage;
+import org.apache.ignite.internal.network.StaticNodeFinder;
+import org.apache.ignite.internal.raft.Loza;
+import org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupServiceFactory;
+import org.apache.ignite.internal.raft.configuration.RaftConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.network.NetworkAddress;
+import org.apache.ignite.raft.jraft.rpc.ActionResponse;
+import org.apache.ignite.raft.jraft.rpc.WriteActionRequest;
+import org.apache.ignite.raft.jraft.rpc.impl.RaftGroupEventsClientListener;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for idempotency of {@link org.apache.ignite.internal.metastorage.command.IdempotentCommand}.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class ItIdempotentCommandCacheTest extends IgniteAbstractTest {
+    private ClusterService clusterService;
+
+    private Loza raftManager;
+
+    private KeyValueStorage storage;
+
+    private MetaStorageManagerImpl metaStorageManager;
+
+    @BeforeEach
+    void setUp(
+            TestInfo testInfo,
+            @InjectConfiguration("mock.responseTimeout = 100") RaftConfiguration raftConfiguration,
+            @InjectConfiguration("mock.idleSyncTimeInterval = 100") MetaStorageConfiguration metaStorageConfiguration
+    ) {
+        var addr = new NetworkAddress("localhost", 10_000);
+
+        clusterService = clusterService(testInfo, addr.port(), new StaticNodeFinder(List.of(addr)));
+
+        HybridClock clock = new HybridClockImpl();
+
+        var raftGroupEventsClientListener = new RaftGroupEventsClientListener();
+
+        raftManager = new Loza(clusterService, new NoOpMetricManager(), raftConfiguration, workDir.resolve("loza"), clock,
+                raftGroupEventsClientListener);
+
+        var logicalTopologyService = mock(LogicalTopologyService.class);
+
+        var topologyAwareRaftGroupServiceFactory = new TopologyAwareRaftGroupServiceFactory(
+                clusterService,
+                logicalTopologyService,
+                Loza.FACTORY,
+                raftGroupEventsClientListener
+        );
+
+        ClusterManagementGroupManager cmgManager = mock(ClusterManagementGroupManager.class);
+
+        when(cmgManager.metaStorageNodes()).thenReturn(completedFuture(Set.of(clusterService.nodeName())));
+
+        storage = new RocksDbKeyValueStorage(
+                clusterService.nodeName(),
+                workDir.resolve("metastorage"),
+                new NoOpFailureProcessor(clusterService.nodeName()));
+
+        metaStorageManager = new MetaStorageManagerImpl(
+                clusterService,
+                cmgManager,
+                logicalTopologyService,
+                raftManager,
+                storage,
+                clock,
+                topologyAwareRaftGroupServiceFactory,
+                metaStorageConfiguration
+        );
+
+        assertThat(
+                startAsync(clusterService, raftManager, metaStorageManager)
+                        .thenCompose(unused -> metaStorageManager.deployWatches()),
+                willCompleteSuccessfully()
+        );
+    }
+
+    @AfterEach
+    void tearDown() throws Exception {
+        List<IgniteComponent> components = List.of(metaStorageManager, raftManager, clusterService);
+
+        closeAll(Stream.concat(
+                components.stream().map(c -> c::beforeNodeStop),
+                Stream.of(() -> assertThat(stopAsync(components), willCompleteSuccessfully()))
+        ));
+    }
+
+    @Test
+    public void testIdempotentInvoke() throws InterruptedException {
+        ByteArray testKey = new ByteArray("key".getBytes(StandardCharsets.UTF_8));

Review Comment:
   added ItIdempotentCommandCacheTest#testIdempotentInvokeAfterLeaderChange



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599949165


##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -278,4 +309,50 @@ boolean beforeApply(Command command) {
 
         return false;
     }
+
+    private static class IdempotentCommandCache {
+        @Nullable
+        final Serializable result;
+
+        final HybridTimestamp commandStartTime;
+
+        IdempotentCommandCache(@Nullable Serializable result, HybridTimestamp commandStartTime) {
+            this.result = result;
+            this.commandStartTime = commandStartTime;
+        }
+    }
+
+    private class ResultCachingClosure implements CommandClosure<WriteCommand> {
+        CommandClosure<WriteCommand> closure;
+
+        ResultCachingClosure(CommandClosure<WriteCommand> closure) {
+            this.closure = closure;
+
+            assert closure.command() instanceof IdempotentCommand;
+        }
+
+        @Override
+        public long index() {
+            return closure.index();
+        }
+
+        @Override
+        public long term() {
+            return closure.term();
+        }
+
+        @Override
+        public WriteCommand command() {
+            return closure.command();
+        }
+
+        @Override
+        public void result(@Nullable Serializable res) {
+            IdempotentCommand command = (IdempotentCommand) closure.command();
+
+            idempotentCommandCache.put(command.id(), new IdempotentCommandCache(res, command.initiatorTime()));

Review Comment:
   `initiatorTime` is almost the same, but from initiator node. Makes no sense to call System.currentTimeMillies once more.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1598848481


##########
modules/metastorage/src/integrationTest/java/org/apache/ignite/internal/metastorage/impl/ItIdempotentCommandCacheTest.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.impl;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.internal.metastorage.dsl.Conditions.notExists;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.noop;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.put;
+import static org.apache.ignite.internal.network.utils.ClusterServiceTestUtils.clusterService;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.IgniteUtils.closeAll;
+import static org.apache.ignite.internal.util.IgniteUtils.startAsync;
+import static org.apache.ignite.internal.util.IgniteUtils.stopAsync;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.BiPredicate;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalTopologyService;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.failure.NoOpFailureProcessor;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridClockImpl;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.command.SyncTimeCommand;
+import org.apache.ignite.internal.metastorage.configuration.MetaStorageConfiguration;
+import org.apache.ignite.internal.metastorage.server.KeyValueStorage;
+import org.apache.ignite.internal.metastorage.server.persistence.RocksDbKeyValueStorage;
+import org.apache.ignite.internal.metrics.NoOpMetricManager;
+import org.apache.ignite.internal.network.ClusterService;
+import org.apache.ignite.internal.network.DefaultMessagingService;
+import org.apache.ignite.internal.network.NetworkMessage;
+import org.apache.ignite.internal.network.StaticNodeFinder;
+import org.apache.ignite.internal.raft.Loza;
+import org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupServiceFactory;
+import org.apache.ignite.internal.raft.configuration.RaftConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.network.NetworkAddress;
+import org.apache.ignite.raft.jraft.rpc.ActionResponse;
+import org.apache.ignite.raft.jraft.rpc.WriteActionRequest;
+import org.apache.ignite.raft.jraft.rpc.impl.RaftGroupEventsClientListener;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for idempotency of {@link org.apache.ignite.internal.metastorage.command.IdempotentCommand}.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class ItIdempotentCommandCacheTest extends IgniteAbstractTest {
+    private ClusterService clusterService;
+
+    private Loza raftManager;
+
+    private KeyValueStorage storage;
+
+    private MetaStorageManagerImpl metaStorageManager;
+
+    @BeforeEach
+    void setUp(
+            TestInfo testInfo,
+            @InjectConfiguration("mock.responseTimeout = 100") RaftConfiguration raftConfiguration,
+            @InjectConfiguration("mock.idleSyncTimeInterval = 100") MetaStorageConfiguration metaStorageConfiguration
+    ) {
+        var addr = new NetworkAddress("localhost", 10_000);
+
+        clusterService = clusterService(testInfo, addr.port(), new StaticNodeFinder(List.of(addr)));
+
+        HybridClock clock = new HybridClockImpl();
+
+        var raftGroupEventsClientListener = new RaftGroupEventsClientListener();
+
+        raftManager = new Loza(clusterService, new NoOpMetricManager(), raftConfiguration, workDir.resolve("loza"), clock,
+                raftGroupEventsClientListener);
+
+        var logicalTopologyService = mock(LogicalTopologyService.class);
+
+        var topologyAwareRaftGroupServiceFactory = new TopologyAwareRaftGroupServiceFactory(
+                clusterService,
+                logicalTopologyService,
+                Loza.FACTORY,
+                raftGroupEventsClientListener
+        );
+
+        ClusterManagementGroupManager cmgManager = mock(ClusterManagementGroupManager.class);
+
+        when(cmgManager.metaStorageNodes()).thenReturn(completedFuture(Set.of(clusterService.nodeName())));
+
+        storage = new RocksDbKeyValueStorage(
+                clusterService.nodeName(),
+                workDir.resolve("metastorage"),
+                new NoOpFailureProcessor(clusterService.nodeName()));
+
+        metaStorageManager = new MetaStorageManagerImpl(
+                clusterService,
+                cmgManager,
+                logicalTopologyService,
+                raftManager,
+                storage,
+                clock,
+                topologyAwareRaftGroupServiceFactory,
+                metaStorageConfiguration
+        );
+
+        assertThat(
+                startAsync(clusterService, raftManager, metaStorageManager)
+                        .thenCompose(unused -> metaStorageManager.deployWatches()),
+                willCompleteSuccessfully()
+        );
+    }
+
+    @AfterEach
+    void tearDown() throws Exception {
+        List<IgniteComponent> components = List.of(metaStorageManager, raftManager, clusterService);
+
+        closeAll(Stream.concat(
+                components.stream().map(c -> c::beforeNodeStop),
+                Stream.of(() -> assertThat(stopAsync(components), willCompleteSuccessfully()))
+        ));
+    }
+
+    @Test
+    public void testIdempotentInvoke() throws InterruptedException {

Review Comment:
   What about multiInvoke?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599973418


##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -278,4 +309,50 @@ boolean beforeApply(Command command) {
 
         return false;
     }
+
+    private static class IdempotentCommandCache {
+        @Nullable
+        final Serializable result;
+
+        final HybridTimestamp commandStartTime;
+
+        IdempotentCommandCache(@Nullable Serializable result, HybridTimestamp commandStartTime) {
+            this.result = result;
+            this.commandStartTime = commandStartTime;
+        }
+    }
+
+    private class ResultCachingClosure implements CommandClosure<WriteCommand> {
+        CommandClosure<WriteCommand> closure;
+
+        ResultCachingClosure(CommandClosure<WriteCommand> closure) {
+            this.closure = closure;
+
+            assert closure.command() instanceof IdempotentCommand;
+        }
+
+        @Override
+        public long index() {
+            return closure.index();
+        }
+
+        @Override
+        public long term() {
+            return closure.term();
+        }
+
+        @Override
+        public WriteCommand command() {
+            return closure.command();
+        }
+
+        @Override
+        public void result(@Nullable Serializable res) {
+            IdempotentCommand command = (IdempotentCommand) closure.command();
+
+            idempotentCommandCache.put(command.id(), new IdempotentCommandCache(res, command.initiatorTime()));

Review Comment:
   Up to you.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599890425


##########
modules/metastorage/src/integrationTest/java/org/apache/ignite/internal/metastorage/impl/ItIdempotentCommandCacheTest.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.impl;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.internal.metastorage.dsl.Conditions.notExists;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.noop;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.put;
+import static org.apache.ignite.internal.network.utils.ClusterServiceTestUtils.clusterService;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.IgniteUtils.closeAll;
+import static org.apache.ignite.internal.util.IgniteUtils.startAsync;
+import static org.apache.ignite.internal.util.IgniteUtils.stopAsync;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.BiPredicate;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalTopologyService;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.failure.NoOpFailureProcessor;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridClockImpl;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.command.SyncTimeCommand;
+import org.apache.ignite.internal.metastorage.configuration.MetaStorageConfiguration;
+import org.apache.ignite.internal.metastorage.server.KeyValueStorage;
+import org.apache.ignite.internal.metastorage.server.persistence.RocksDbKeyValueStorage;
+import org.apache.ignite.internal.metrics.NoOpMetricManager;
+import org.apache.ignite.internal.network.ClusterService;
+import org.apache.ignite.internal.network.DefaultMessagingService;
+import org.apache.ignite.internal.network.NetworkMessage;
+import org.apache.ignite.internal.network.StaticNodeFinder;
+import org.apache.ignite.internal.raft.Loza;
+import org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupServiceFactory;
+import org.apache.ignite.internal.raft.configuration.RaftConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.network.NetworkAddress;
+import org.apache.ignite.raft.jraft.rpc.ActionResponse;
+import org.apache.ignite.raft.jraft.rpc.WriteActionRequest;
+import org.apache.ignite.raft.jraft.rpc.impl.RaftGroupEventsClientListener;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for idempotency of {@link org.apache.ignite.internal.metastorage.command.IdempotentCommand}.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class ItIdempotentCommandCacheTest extends IgniteAbstractTest {
+    private ClusterService clusterService;
+
+    private Loza raftManager;
+
+    private KeyValueStorage storage;
+
+    private MetaStorageManagerImpl metaStorageManager;
+
+    @BeforeEach
+    void setUp(
+            TestInfo testInfo,
+            @InjectConfiguration("mock.responseTimeout = 100") RaftConfiguration raftConfiguration,
+            @InjectConfiguration("mock.idleSyncTimeInterval = 100") MetaStorageConfiguration metaStorageConfiguration
+    ) {
+        var addr = new NetworkAddress("localhost", 10_000);
+
+        clusterService = clusterService(testInfo, addr.port(), new StaticNodeFinder(List.of(addr)));
+
+        HybridClock clock = new HybridClockImpl();
+
+        var raftGroupEventsClientListener = new RaftGroupEventsClientListener();
+
+        raftManager = new Loza(clusterService, new NoOpMetricManager(), raftConfiguration, workDir.resolve("loza"), clock,
+                raftGroupEventsClientListener);
+
+        var logicalTopologyService = mock(LogicalTopologyService.class);
+
+        var topologyAwareRaftGroupServiceFactory = new TopologyAwareRaftGroupServiceFactory(
+                clusterService,
+                logicalTopologyService,
+                Loza.FACTORY,
+                raftGroupEventsClientListener
+        );
+
+        ClusterManagementGroupManager cmgManager = mock(ClusterManagementGroupManager.class);
+
+        when(cmgManager.metaStorageNodes()).thenReturn(completedFuture(Set.of(clusterService.nodeName())));
+
+        storage = new RocksDbKeyValueStorage(
+                clusterService.nodeName(),
+                workDir.resolve("metastorage"),
+                new NoOpFailureProcessor(clusterService.nodeName()));
+
+        metaStorageManager = new MetaStorageManagerImpl(
+                clusterService,
+                cmgManager,
+                logicalTopologyService,
+                raftManager,
+                storage,
+                clock,
+                topologyAwareRaftGroupServiceFactory,
+                metaStorageConfiguration
+        );
+
+        assertThat(
+                startAsync(clusterService, raftManager, metaStorageManager)
+                        .thenCompose(unused -> metaStorageManager.deployWatches()),
+                willCompleteSuccessfully()
+        );
+    }
+
+    @AfterEach
+    void tearDown() throws Exception {
+        List<IgniteComponent> components = List.of(metaStorageManager, raftManager, clusterService);
+
+        closeAll(Stream.concat(
+                components.stream().map(c -> c::beforeNodeStop),
+                Stream.of(() -> assertThat(stopAsync(components), willCompleteSuccessfully()))
+        ));
+    }
+
+    @Test
+    public void testIdempotentInvoke() throws InterruptedException {

Review Comment:
   it's tested in unit test, the point of integration test is to check any command within raft infrastructure



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1598782469


##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -278,4 +309,50 @@ boolean beforeApply(Command command) {
 
         return false;
     }
+
+    private static class IdempotentCommandCache {

Review Comment:
   I don't really like naming here. The map itself is a cache, but not the value. I'd rather rename to IdempotentCommandCacheEntry or similar.



##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/impl/CommandIdGenerator.java:
##########
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.impl;
+
+import java.util.UUID;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Supplier;
+import org.apache.ignite.internal.metastorage.CommandId;
+import org.apache.ignite.internal.metastorage.dsl.MetaStorageMessagesFactory;
+
+/**
+ * Generates the command ids.
+ */
+public class CommandIdGenerator {
+    private static final MetaStorageMessagesFactory MSG_FACTORY = new MetaStorageMessagesFactory();
+
+    /** Supplies nodeId for transactionId generation. */

Review Comment:
   TransactionId?
   Anyway, why it's a supplier? Do you mean that we don't know nodeId on generator creation?



##########
modules/metastorage/src/integrationTest/java/org/apache/ignite/internal/metastorage/impl/ItMetaStorageServicePersistenceTest.java:
##########
@@ -76,7 +76,8 @@ public void beforeFollowerStop(RaftGroupService service, RaftServer server) thro
 
         var clusterTime = new ClusterTimeImpl(followerNode.name(), new IgniteSpinBusyLock(), new HybridClockImpl());
 
-        metaStorage = new MetaStorageServiceImpl(followerNode.name(), service, new IgniteSpinBusyLock(), clusterTime);
+        metaStorage = new MetaStorageServiceImpl(followerNode.name(), service, new IgniteSpinBusyLock(), clusterTime,

Review Comment:
   I'd rather put one param at a line here. It's 4 of them already.



##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -278,4 +309,50 @@ boolean beforeApply(Command command) {
 
         return false;
     }
+
+    private static class IdempotentCommandCache {
+        @Nullable
+        final Serializable result;
+
+        final HybridTimestamp commandStartTime;
+
+        IdempotentCommandCache(@Nullable Serializable result, HybridTimestamp commandStartTime) {
+            this.result = result;
+            this.commandStartTime = commandStartTime;
+        }
+    }
+
+    private class ResultCachingClosure implements CommandClosure<WriteCommand> {
+        CommandClosure<WriteCommand> closure;
+
+        ResultCachingClosure(CommandClosure<WriteCommand> closure) {
+            this.closure = closure;
+
+            assert closure.command() instanceof IdempotentCommand;
+        }
+
+        @Override
+        public long index() {
+            return closure.index();
+        }
+
+        @Override
+        public long term() {
+            return closure.term();
+        }
+
+        @Override
+        public WriteCommand command() {
+            return closure.command();
+        }
+
+        @Override
+        public void result(@Nullable Serializable res) {
+            IdempotentCommand command = (IdempotentCommand) closure.command();
+
+            idempotentCommandCache.put(command.id(), new IdempotentCommandCache(res, command.initiatorTime()));

Review Comment:
   Instead of command.initiatorTime I'd rather use System.currentTimeMillies here.



##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -72,6 +80,29 @@ public class MetaStorageWriteHandler {
     void handleWriteCommand(CommandClosure<WriteCommand> clo) {
         WriteCommand command = clo.command();
 
+        CommandClosure<WriteCommand> resultClosure;
+
+        if (command instanceof IdempotentCommand) {
+            CommandId commandId = ((IdempotentCommand) command).id();
+            IdempotentCommandCache cache = idempotentCommandCache.get(commandId);

Review Comment:
   The var naming is confusing, please check another comment for more details.



##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -278,4 +309,50 @@ boolean beforeApply(Command command) {
 
         return false;
     }
+
+    private static class IdempotentCommandCache {
+        @Nullable
+        final Serializable result;
+
+        final HybridTimestamp commandStartTime;
+
+        IdempotentCommandCache(@Nullable Serializable result, HybridTimestamp commandStartTime) {
+            this.result = result;
+            this.commandStartTime = commandStartTime;
+        }
+    }
+
+    private class ResultCachingClosure implements CommandClosure<WriteCommand> {
+        CommandClosure<WriteCommand> closure;
+
+        ResultCachingClosure(CommandClosure<WriteCommand> closure) {
+            this.closure = closure;
+
+            assert closure.command() instanceof IdempotentCommand;
+        }
+
+        @Override
+        public long index() {
+            return closure.index();
+        }
+
+        @Override
+        public long term() {
+            return closure.term();
+        }
+
+        @Override
+        public WriteCommand command() {
+            return closure.command();
+        }
+
+        @Override
+        public void result(@Nullable Serializable res) {
+            IdempotentCommand command = (IdempotentCommand) closure.command();
+
+            idempotentCommandCache.put(command.id(), new IdempotentCommandCache(res, command.initiatorTime()));

Review Comment:
   So, we will put an exception as cached result. I don't think that it's correct. 



##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -278,4 +309,50 @@ boolean beforeApply(Command command) {
 
         return false;
     }
+
+    private static class IdempotentCommandCache {
+        @Nullable
+        final Serializable result;
+
+        final HybridTimestamp commandStartTime;

Review Comment:
   I don't think that we need HybridTimestamp here. Common long with current time should be enough. WDYT?



##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -278,4 +309,50 @@ boolean beforeApply(Command command) {
 
         return false;
     }
+
+    private static class IdempotentCommandCache {
+        @Nullable

Review Comment:
   Is it really nullable now?



##########
modules/metastorage-api/src/main/java/org/apache/ignite/internal/metastorage/CommandId.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage;
+
+import java.util.UUID;
+import org.apache.ignite.internal.metastorage.dsl.MetaStorageMessageGroup;
+import org.apache.ignite.internal.network.NetworkMessage;
+import org.apache.ignite.internal.network.annotations.Transferable;
+
+/**
+ * Command id.

Review Comment:
   Could you please add few more words about uniquness guarantess etc?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599740322


##########
modules/metastorage-api/src/main/java/org/apache/ignite/internal/metastorage/CommandId.java:
##########
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage;
+
+import java.util.UUID;
+import org.apache.ignite.internal.metastorage.dsl.MetaStorageMessageGroup;
+import org.apache.ignite.internal.network.NetworkMessage;
+import org.apache.ignite.internal.network.annotations.Transferable;
+
+/**
+ * Command id.

Review Comment:
   Added.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599505214


##########
modules/metastorage/src/integrationTest/java/org/apache/ignite/internal/metastorage/impl/ItIdempotentCommandCacheTest.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.impl;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.internal.metastorage.dsl.Conditions.notExists;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.noop;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.put;
+import static org.apache.ignite.internal.network.utils.ClusterServiceTestUtils.clusterService;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.IgniteUtils.closeAll;
+import static org.apache.ignite.internal.util.IgniteUtils.startAsync;
+import static org.apache.ignite.internal.util.IgniteUtils.stopAsync;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.BiPredicate;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalTopologyService;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.failure.NoOpFailureProcessor;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridClockImpl;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.command.SyncTimeCommand;
+import org.apache.ignite.internal.metastorage.configuration.MetaStorageConfiguration;
+import org.apache.ignite.internal.metastorage.server.KeyValueStorage;
+import org.apache.ignite.internal.metastorage.server.persistence.RocksDbKeyValueStorage;
+import org.apache.ignite.internal.metrics.NoOpMetricManager;
+import org.apache.ignite.internal.network.ClusterService;
+import org.apache.ignite.internal.network.DefaultMessagingService;
+import org.apache.ignite.internal.network.NetworkMessage;
+import org.apache.ignite.internal.network.StaticNodeFinder;
+import org.apache.ignite.internal.raft.Loza;
+import org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupServiceFactory;
+import org.apache.ignite.internal.raft.configuration.RaftConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.network.NetworkAddress;
+import org.apache.ignite.raft.jraft.rpc.ActionResponse;
+import org.apache.ignite.raft.jraft.rpc.WriteActionRequest;
+import org.apache.ignite.raft.jraft.rpc.impl.RaftGroupEventsClientListener;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for idempotency of {@link org.apache.ignite.internal.metastorage.command.IdempotentCommand}.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class ItIdempotentCommandCacheTest extends IgniteAbstractTest {
+    private ClusterService clusterService;
+
+    private Loza raftManager;
+
+    private KeyValueStorage storage;
+
+    private MetaStorageManagerImpl metaStorageManager;
+
+    @BeforeEach
+    void setUp(
+            TestInfo testInfo,
+            @InjectConfiguration("mock.responseTimeout = 100") RaftConfiguration raftConfiguration,
+            @InjectConfiguration("mock.idleSyncTimeInterval = 100") MetaStorageConfiguration metaStorageConfiguration
+    ) {
+        var addr = new NetworkAddress("localhost", 10_000);
+
+        clusterService = clusterService(testInfo, addr.port(), new StaticNodeFinder(List.of(addr)));
+
+        HybridClock clock = new HybridClockImpl();
+
+        var raftGroupEventsClientListener = new RaftGroupEventsClientListener();
+
+        raftManager = new Loza(clusterService, new NoOpMetricManager(), raftConfiguration, workDir.resolve("loza"), clock,
+                raftGroupEventsClientListener);
+
+        var logicalTopologyService = mock(LogicalTopologyService.class);
+
+        var topologyAwareRaftGroupServiceFactory = new TopologyAwareRaftGroupServiceFactory(
+                clusterService,
+                logicalTopologyService,
+                Loza.FACTORY,
+                raftGroupEventsClientListener
+        );
+
+        ClusterManagementGroupManager cmgManager = mock(ClusterManagementGroupManager.class);
+
+        when(cmgManager.metaStorageNodes()).thenReturn(completedFuture(Set.of(clusterService.nodeName())));
+
+        storage = new RocksDbKeyValueStorage(
+                clusterService.nodeName(),
+                workDir.resolve("metastorage"),
+                new NoOpFailureProcessor(clusterService.nodeName()));
+
+        metaStorageManager = new MetaStorageManagerImpl(
+                clusterService,
+                cmgManager,
+                logicalTopologyService,
+                raftManager,
+                storage,
+                clock,
+                topologyAwareRaftGroupServiceFactory,
+                metaStorageConfiguration
+        );
+
+        assertThat(
+                startAsync(clusterService, raftManager, metaStorageManager)
+                        .thenCompose(unused -> metaStorageManager.deployWatches()),
+                willCompleteSuccessfully()
+        );
+    }
+
+    @AfterEach
+    void tearDown() throws Exception {
+        List<IgniteComponent> components = List.of(metaStorageManager, raftManager, clusterService);
+
+        closeAll(Stream.concat(
+                components.stream().map(c -> c::beforeNodeStop),
+                Stream.of(() -> assertThat(stopAsync(components), willCompleteSuccessfully()))
+        ));
+    }
+
+    @Test
+    public void testIdempotentInvoke() throws InterruptedException {
+        ByteArray testKey = new ByteArray("key".getBytes(StandardCharsets.UTF_8));

Review Comment:
   Could you please also add a test that will verify that idempotent command cache is available (and thus proper command result is returned) after leader change? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599889410


##########
modules/metastorage/src/integrationTest/java/org/apache/ignite/internal/metastorage/impl/ItIdempotentCommandCacheTest.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.impl;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.internal.metastorage.dsl.Conditions.notExists;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.noop;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.put;
+import static org.apache.ignite.internal.network.utils.ClusterServiceTestUtils.clusterService;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.IgniteUtils.closeAll;
+import static org.apache.ignite.internal.util.IgniteUtils.startAsync;
+import static org.apache.ignite.internal.util.IgniteUtils.stopAsync;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.BiPredicate;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalTopologyService;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.failure.NoOpFailureProcessor;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridClockImpl;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.command.SyncTimeCommand;
+import org.apache.ignite.internal.metastorage.configuration.MetaStorageConfiguration;
+import org.apache.ignite.internal.metastorage.server.KeyValueStorage;
+import org.apache.ignite.internal.metastorage.server.persistence.RocksDbKeyValueStorage;
+import org.apache.ignite.internal.metrics.NoOpMetricManager;
+import org.apache.ignite.internal.network.ClusterService;
+import org.apache.ignite.internal.network.DefaultMessagingService;
+import org.apache.ignite.internal.network.NetworkMessage;
+import org.apache.ignite.internal.network.StaticNodeFinder;
+import org.apache.ignite.internal.raft.Loza;
+import org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupServiceFactory;
+import org.apache.ignite.internal.raft.configuration.RaftConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.network.NetworkAddress;
+import org.apache.ignite.raft.jraft.rpc.ActionResponse;
+import org.apache.ignite.raft.jraft.rpc.WriteActionRequest;
+import org.apache.ignite.raft.jraft.rpc.impl.RaftGroupEventsClientListener;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for idempotency of {@link org.apache.ignite.internal.metastorage.command.IdempotentCommand}.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class ItIdempotentCommandCacheTest extends IgniteAbstractTest {
+    private ClusterService clusterService;
+
+    private Loza raftManager;
+
+    private KeyValueStorage storage;
+
+    private MetaStorageManagerImpl metaStorageManager;
+
+    @BeforeEach
+    void setUp(
+            TestInfo testInfo,
+            @InjectConfiguration("mock.responseTimeout = 100") RaftConfiguration raftConfiguration,
+            @InjectConfiguration("mock.idleSyncTimeInterval = 100") MetaStorageConfiguration metaStorageConfiguration
+    ) {
+        var addr = new NetworkAddress("localhost", 10_000);
+
+        clusterService = clusterService(testInfo, addr.port(), new StaticNodeFinder(List.of(addr)));
+
+        HybridClock clock = new HybridClockImpl();
+
+        var raftGroupEventsClientListener = new RaftGroupEventsClientListener();
+
+        raftManager = new Loza(clusterService, new NoOpMetricManager(), raftConfiguration, workDir.resolve("loza"), clock,
+                raftGroupEventsClientListener);
+
+        var logicalTopologyService = mock(LogicalTopologyService.class);
+
+        var topologyAwareRaftGroupServiceFactory = new TopologyAwareRaftGroupServiceFactory(
+                clusterService,
+                logicalTopologyService,
+                Loza.FACTORY,
+                raftGroupEventsClientListener
+        );
+
+        ClusterManagementGroupManager cmgManager = mock(ClusterManagementGroupManager.class);
+
+        when(cmgManager.metaStorageNodes()).thenReturn(completedFuture(Set.of(clusterService.nodeName())));
+
+        storage = new RocksDbKeyValueStorage(
+                clusterService.nodeName(),
+                workDir.resolve("metastorage"),
+                new NoOpFailureProcessor(clusterService.nodeName()));
+
+        metaStorageManager = new MetaStorageManagerImpl(
+                clusterService,
+                cmgManager,
+                logicalTopologyService,
+                raftManager,
+                storage,
+                clock,
+                topologyAwareRaftGroupServiceFactory,
+                metaStorageConfiguration
+        );
+
+        assertThat(
+                startAsync(clusterService, raftManager, metaStorageManager)
+                        .thenCompose(unused -> metaStorageManager.deployWatches()),
+                willCompleteSuccessfully()
+        );
+    }
+
+    @AfterEach
+    void tearDown() throws Exception {
+        List<IgniteComponent> components = List.of(metaStorageManager, raftManager, clusterService);
+
+        closeAll(Stream.concat(
+                components.stream().map(c -> c::beforeNodeStop),
+                Stream.of(() -> assertThat(stopAsync(components), willCompleteSuccessfully()))
+        ));
+    }
+
+    @Test
+    public void testIdempotentInvoke() throws InterruptedException {
+        ByteArray testKey = new ByteArray("key".getBytes(StandardCharsets.UTF_8));
+        byte[] testValue = "value".getBytes(StandardCharsets.UTF_8);
+
+        AtomicInteger writeActionReqCount = new AtomicInteger();
+        CompletableFuture<Void> retryBlockingFuture = new CompletableFuture<>();
+
+        log.info("Test: blocking messages.");
+
+        dropMessages((n, msg) -> {
+            // Dropping the first response, this will cause timeout on first response, and then retry.
+            if (msg instanceof ActionResponse && ((ActionResponse) msg).result() != null && writeActionReqCount.get() == 1) {
+                log.info("Test: dropping ActionResponse: " + msg);
+
+                return true;
+            }
+
+            if (msg instanceof WriteActionRequest) {
+                WriteActionRequest request = (WriteActionRequest) msg;
+
+                if (!(request.deserializedCommand() instanceof SyncTimeCommand)) {
+                    writeActionReqCount.incrementAndGet();
+                    log.info("Test: WriteActionRequest intercepted, count=" + writeActionReqCount.get());
+
+                    // Second request: retry.
+                    if (writeActionReqCount.get() == 2) {
+                        log.info("Test: retry blocked.");
+
+                        retryBlockingFuture.orTimeout(10, TimeUnit.SECONDS).join();
+
+                        log.info("Test: retry unblocked.");
+                    }
+                }
+            }
+
+            return false;
+        });
+
+        CompletableFuture<Boolean> fut = metaStorageManager.invoke(notExists(testKey), put(testKey, testValue), noop());

Review Comment:
   Done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599974465


##########
modules/metastorage/src/integrationTest/java/org/apache/ignite/internal/metastorage/impl/ItIdempotentCommandCacheTest.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.impl;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.internal.metastorage.dsl.Conditions.notExists;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.noop;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.put;
+import static org.apache.ignite.internal.network.utils.ClusterServiceTestUtils.clusterService;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.IgniteUtils.closeAll;
+import static org.apache.ignite.internal.util.IgniteUtils.startAsync;
+import static org.apache.ignite.internal.util.IgniteUtils.stopAsync;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.BiPredicate;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalTopologyService;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.failure.NoOpFailureProcessor;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridClockImpl;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.command.SyncTimeCommand;
+import org.apache.ignite.internal.metastorage.configuration.MetaStorageConfiguration;
+import org.apache.ignite.internal.metastorage.server.KeyValueStorage;
+import org.apache.ignite.internal.metastorage.server.persistence.RocksDbKeyValueStorage;
+import org.apache.ignite.internal.metrics.NoOpMetricManager;
+import org.apache.ignite.internal.network.ClusterService;
+import org.apache.ignite.internal.network.DefaultMessagingService;
+import org.apache.ignite.internal.network.NetworkMessage;
+import org.apache.ignite.internal.network.StaticNodeFinder;
+import org.apache.ignite.internal.raft.Loza;
+import org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupServiceFactory;
+import org.apache.ignite.internal.raft.configuration.RaftConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.network.NetworkAddress;
+import org.apache.ignite.raft.jraft.rpc.ActionResponse;
+import org.apache.ignite.raft.jraft.rpc.WriteActionRequest;
+import org.apache.ignite.raft.jraft.rpc.impl.RaftGroupEventsClientListener;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for idempotency of {@link org.apache.ignite.internal.metastorage.command.IdempotentCommand}.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class ItIdempotentCommandCacheTest extends IgniteAbstractTest {
+    private ClusterService clusterService;
+
+    private Loza raftManager;
+
+    private KeyValueStorage storage;
+
+    private MetaStorageManagerImpl metaStorageManager;
+
+    @BeforeEach
+    void setUp(
+            TestInfo testInfo,
+            @InjectConfiguration("mock.responseTimeout = 100") RaftConfiguration raftConfiguration,
+            @InjectConfiguration("mock.idleSyncTimeInterval = 100") MetaStorageConfiguration metaStorageConfiguration
+    ) {
+        var addr = new NetworkAddress("localhost", 10_000);
+
+        clusterService = clusterService(testInfo, addr.port(), new StaticNodeFinder(List.of(addr)));
+
+        HybridClock clock = new HybridClockImpl();
+
+        var raftGroupEventsClientListener = new RaftGroupEventsClientListener();
+
+        raftManager = new Loza(clusterService, new NoOpMetricManager(), raftConfiguration, workDir.resolve("loza"), clock,
+                raftGroupEventsClientListener);
+
+        var logicalTopologyService = mock(LogicalTopologyService.class);
+
+        var topologyAwareRaftGroupServiceFactory = new TopologyAwareRaftGroupServiceFactory(
+                clusterService,
+                logicalTopologyService,
+                Loza.FACTORY,
+                raftGroupEventsClientListener
+        );
+
+        ClusterManagementGroupManager cmgManager = mock(ClusterManagementGroupManager.class);
+
+        when(cmgManager.metaStorageNodes()).thenReturn(completedFuture(Set.of(clusterService.nodeName())));
+
+        storage = new RocksDbKeyValueStorage(
+                clusterService.nodeName(),
+                workDir.resolve("metastorage"),
+                new NoOpFailureProcessor(clusterService.nodeName()));
+
+        metaStorageManager = new MetaStorageManagerImpl(
+                clusterService,
+                cmgManager,
+                logicalTopologyService,
+                raftManager,
+                storage,
+                clock,
+                topologyAwareRaftGroupServiceFactory,
+                metaStorageConfiguration
+        );
+
+        assertThat(
+                startAsync(clusterService, raftManager, metaStorageManager)
+                        .thenCompose(unused -> metaStorageManager.deployWatches()),
+                willCompleteSuccessfully()
+        );
+    }
+
+    @AfterEach
+    void tearDown() throws Exception {
+        List<IgniteComponent> components = List.of(metaStorageManager, raftManager, clusterService);
+
+        closeAll(Stream.concat(
+                components.stream().map(c -> c::beforeNodeStop),
+                Stream.of(() -> assertThat(stopAsync(components), willCompleteSuccessfully()))
+        ));
+    }
+
+    @Test
+    public void testIdempotentInvoke() throws InterruptedException {

Review Comment:
   Up to you.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1598847344


##########
modules/metastorage/src/integrationTest/java/org/apache/ignite/internal/metastorage/impl/ItIdempotentCommandCacheTest.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.impl;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.internal.metastorage.dsl.Conditions.notExists;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.noop;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.put;
+import static org.apache.ignite.internal.network.utils.ClusterServiceTestUtils.clusterService;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.IgniteUtils.closeAll;
+import static org.apache.ignite.internal.util.IgniteUtils.startAsync;
+import static org.apache.ignite.internal.util.IgniteUtils.stopAsync;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.BiPredicate;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalTopologyService;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.failure.NoOpFailureProcessor;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridClockImpl;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.command.SyncTimeCommand;
+import org.apache.ignite.internal.metastorage.configuration.MetaStorageConfiguration;
+import org.apache.ignite.internal.metastorage.server.KeyValueStorage;
+import org.apache.ignite.internal.metastorage.server.persistence.RocksDbKeyValueStorage;
+import org.apache.ignite.internal.metrics.NoOpMetricManager;
+import org.apache.ignite.internal.network.ClusterService;
+import org.apache.ignite.internal.network.DefaultMessagingService;
+import org.apache.ignite.internal.network.NetworkMessage;
+import org.apache.ignite.internal.network.StaticNodeFinder;
+import org.apache.ignite.internal.raft.Loza;
+import org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupServiceFactory;
+import org.apache.ignite.internal.raft.configuration.RaftConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.network.NetworkAddress;
+import org.apache.ignite.raft.jraft.rpc.ActionResponse;
+import org.apache.ignite.raft.jraft.rpc.WriteActionRequest;
+import org.apache.ignite.raft.jraft.rpc.impl.RaftGroupEventsClientListener;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for idempotency of {@link org.apache.ignite.internal.metastorage.command.IdempotentCommand}.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class ItIdempotentCommandCacheTest extends IgniteAbstractTest {
+    private ClusterService clusterService;
+
+    private Loza raftManager;
+
+    private KeyValueStorage storage;
+
+    private MetaStorageManagerImpl metaStorageManager;
+
+    @BeforeEach
+    void setUp(
+            TestInfo testInfo,
+            @InjectConfiguration("mock.responseTimeout = 100") RaftConfiguration raftConfiguration,
+            @InjectConfiguration("mock.idleSyncTimeInterval = 100") MetaStorageConfiguration metaStorageConfiguration
+    ) {
+        var addr = new NetworkAddress("localhost", 10_000);
+
+        clusterService = clusterService(testInfo, addr.port(), new StaticNodeFinder(List.of(addr)));
+
+        HybridClock clock = new HybridClockImpl();
+
+        var raftGroupEventsClientListener = new RaftGroupEventsClientListener();
+
+        raftManager = new Loza(clusterService, new NoOpMetricManager(), raftConfiguration, workDir.resolve("loza"), clock,
+                raftGroupEventsClientListener);
+
+        var logicalTopologyService = mock(LogicalTopologyService.class);
+
+        var topologyAwareRaftGroupServiceFactory = new TopologyAwareRaftGroupServiceFactory(
+                clusterService,
+                logicalTopologyService,
+                Loza.FACTORY,
+                raftGroupEventsClientListener
+        );
+
+        ClusterManagementGroupManager cmgManager = mock(ClusterManagementGroupManager.class);
+
+        when(cmgManager.metaStorageNodes()).thenReturn(completedFuture(Set.of(clusterService.nodeName())));
+
+        storage = new RocksDbKeyValueStorage(
+                clusterService.nodeName(),
+                workDir.resolve("metastorage"),
+                new NoOpFailureProcessor(clusterService.nodeName()));
+
+        metaStorageManager = new MetaStorageManagerImpl(
+                clusterService,
+                cmgManager,
+                logicalTopologyService,
+                raftManager,
+                storage,
+                clock,
+                topologyAwareRaftGroupServiceFactory,
+                metaStorageConfiguration
+        );
+
+        assertThat(
+                startAsync(clusterService, raftManager, metaStorageManager)
+                        .thenCompose(unused -> metaStorageManager.deployWatches()),
+                willCompleteSuccessfully()
+        );
+    }
+
+    @AfterEach
+    void tearDown() throws Exception {
+        List<IgniteComponent> components = List.of(metaStorageManager, raftManager, clusterService);
+
+        closeAll(Stream.concat(
+                components.stream().map(c -> c::beforeNodeStop),
+                Stream.of(() -> assertThat(stopAsync(components), willCompleteSuccessfully()))
+        ));
+    }
+
+    @Test
+    public void testIdempotentInvoke() throws InterruptedException {
+        ByteArray testKey = new ByteArray("key".getBytes(StandardCharsets.UTF_8));
+        byte[] testValue = "value".getBytes(StandardCharsets.UTF_8);
+
+        AtomicInteger writeActionReqCount = new AtomicInteger();
+        CompletableFuture<Void> retryBlockingFuture = new CompletableFuture<>();
+
+        log.info("Test: blocking messages.");
+
+        dropMessages((n, msg) -> {
+            // Dropping the first response, this will cause timeout on first response, and then retry.
+            if (msg instanceof ActionResponse && ((ActionResponse) msg).result() != null && writeActionReqCount.get() == 1) {
+                log.info("Test: dropping ActionResponse: " + msg);
+
+                return true;
+            }
+
+            if (msg instanceof WriteActionRequest) {
+                WriteActionRequest request = (WriteActionRequest) msg;
+
+                if (!(request.deserializedCommand() instanceof SyncTimeCommand)) {
+                    writeActionReqCount.incrementAndGet();
+                    log.info("Test: WriteActionRequest intercepted, count=" + writeActionReqCount.get());
+
+                    // Second request: retry.
+                    if (writeActionReqCount.get() == 2) {
+                        log.info("Test: retry blocked.");
+
+                        retryBlockingFuture.orTimeout(10, TimeUnit.SECONDS).join();
+
+                        log.info("Test: retry unblocked.");
+                    }
+                }
+            }
+
+            return false;
+        });
+
+        CompletableFuture<Boolean> fut = metaStorageManager.invoke(notExists(testKey), put(testKey, testValue), noop());

Review Comment:
   I'd also have put(anotherTestKey,...) instead of noop() and corresponding check that it won't be actually putted.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599735299


##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -72,6 +80,29 @@ public class MetaStorageWriteHandler {
     void handleWriteCommand(CommandClosure<WriteCommand> clo) {
         WriteCommand command = clo.command();
 
+        CommandClosure<WriteCommand> resultClosure;
+
+        if (command instanceof IdempotentCommand) {
+            CommandId commandId = ((IdempotentCommand) command).id();
+            IdempotentCommandCache cache = idempotentCommandCache.get(commandId);

Review Comment:
   if you mean `cache`, I renamed it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1598783865


##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -278,4 +309,50 @@ boolean beforeApply(Command command) {
 
         return false;
     }
+
+    private static class IdempotentCommandCache {
+        @Nullable

Review Comment:
   Is it really nullable now?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599731099


##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -278,4 +309,50 @@ boolean beforeApply(Command command) {
 
         return false;
     }
+
+    private static class IdempotentCommandCache {
+        @Nullable
+        final Serializable result;
+
+        final HybridTimestamp commandStartTime;

Review Comment:
   I can replace it with long, but why current time? Command start time is hybrid time



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599728941


##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -278,4 +309,50 @@ boolean beforeApply(Command command) {
 
         return false;
     }
+
+    private static class IdempotentCommandCache {

Review Comment:
   renamed to IdempotentCommandCachedResult



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc merged PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599731099


##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -278,4 +309,50 @@ boolean beforeApply(Command command) {
 
         return false;
     }
+
+    private static class IdempotentCommandCache {
+        @Nullable
+        final Serializable result;
+
+        final HybridTimestamp commandStartTime;

Review Comment:
   Why current time? Command start time is hybrid time. There is no sense to make a conversion. And `long` type can create an ambiguity. `HybridTimestamp` is more clear here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1598843955


##########
modules/metastorage/src/integrationTest/java/org/apache/ignite/internal/metastorage/impl/ItIdempotentCommandCacheTest.java:
##########
@@ -0,0 +1,224 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.impl;
+
+import static java.util.concurrent.CompletableFuture.completedFuture;
+import static org.apache.ignite.internal.metastorage.dsl.Conditions.notExists;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.noop;
+import static org.apache.ignite.internal.metastorage.dsl.Operations.put;
+import static org.apache.ignite.internal.network.utils.ClusterServiceTestUtils.clusterService;
+import static org.apache.ignite.internal.testframework.IgniteTestUtils.waitForCondition;
+import static org.apache.ignite.internal.testframework.matchers.CompletableFutureMatcher.willCompleteSuccessfully;
+import static org.apache.ignite.internal.util.IgniteUtils.closeAll;
+import static org.apache.ignite.internal.util.IgniteUtils.startAsync;
+import static org.apache.ignite.internal.util.IgniteUtils.stopAsync;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.BiPredicate;
+import java.util.stream.Stream;
+import org.apache.ignite.internal.cluster.management.ClusterManagementGroupManager;
+import org.apache.ignite.internal.cluster.management.topology.api.LogicalTopologyService;
+import org.apache.ignite.internal.configuration.testframework.ConfigurationExtension;
+import org.apache.ignite.internal.configuration.testframework.InjectConfiguration;
+import org.apache.ignite.internal.failure.NoOpFailureProcessor;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridClockImpl;
+import org.apache.ignite.internal.lang.ByteArray;
+import org.apache.ignite.internal.manager.IgniteComponent;
+import org.apache.ignite.internal.metastorage.Entry;
+import org.apache.ignite.internal.metastorage.command.SyncTimeCommand;
+import org.apache.ignite.internal.metastorage.configuration.MetaStorageConfiguration;
+import org.apache.ignite.internal.metastorage.server.KeyValueStorage;
+import org.apache.ignite.internal.metastorage.server.persistence.RocksDbKeyValueStorage;
+import org.apache.ignite.internal.metrics.NoOpMetricManager;
+import org.apache.ignite.internal.network.ClusterService;
+import org.apache.ignite.internal.network.DefaultMessagingService;
+import org.apache.ignite.internal.network.NetworkMessage;
+import org.apache.ignite.internal.network.StaticNodeFinder;
+import org.apache.ignite.internal.raft.Loza;
+import org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupServiceFactory;
+import org.apache.ignite.internal.raft.configuration.RaftConfiguration;
+import org.apache.ignite.internal.testframework.IgniteAbstractTest;
+import org.apache.ignite.network.NetworkAddress;
+import org.apache.ignite.raft.jraft.rpc.ActionResponse;
+import org.apache.ignite.raft.jraft.rpc.WriteActionRequest;
+import org.apache.ignite.raft.jraft.rpc.impl.RaftGroupEventsClientListener;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+import org.junit.jupiter.api.extension.ExtendWith;
+
+/**
+ * Integration tests for idempotency of {@link org.apache.ignite.internal.metastorage.command.IdempotentCommand}.
+ */
+@ExtendWith(ConfigurationExtension.class)
+public class ItIdempotentCommandCacheTest extends IgniteAbstractTest {
+    private ClusterService clusterService;
+
+    private Loza raftManager;
+
+    private KeyValueStorage storage;
+
+    private MetaStorageManagerImpl metaStorageManager;
+
+    @BeforeEach
+    void setUp(
+            TestInfo testInfo,
+            @InjectConfiguration("mock.responseTimeout = 100") RaftConfiguration raftConfiguration,
+            @InjectConfiguration("mock.idleSyncTimeInterval = 100") MetaStorageConfiguration metaStorageConfiguration
+    ) {
+        var addr = new NetworkAddress("localhost", 10_000);
+
+        clusterService = clusterService(testInfo, addr.port(), new StaticNodeFinder(List.of(addr)));
+
+        HybridClock clock = new HybridClockImpl();
+
+        var raftGroupEventsClientListener = new RaftGroupEventsClientListener();
+
+        raftManager = new Loza(clusterService, new NoOpMetricManager(), raftConfiguration, workDir.resolve("loza"), clock,
+                raftGroupEventsClientListener);
+
+        var logicalTopologyService = mock(LogicalTopologyService.class);
+
+        var topologyAwareRaftGroupServiceFactory = new TopologyAwareRaftGroupServiceFactory(
+                clusterService,
+                logicalTopologyService,
+                Loza.FACTORY,
+                raftGroupEventsClientListener
+        );
+
+        ClusterManagementGroupManager cmgManager = mock(ClusterManagementGroupManager.class);
+
+        when(cmgManager.metaStorageNodes()).thenReturn(completedFuture(Set.of(clusterService.nodeName())));
+
+        storage = new RocksDbKeyValueStorage(
+                clusterService.nodeName(),
+                workDir.resolve("metastorage"),
+                new NoOpFailureProcessor(clusterService.nodeName()));
+
+        metaStorageManager = new MetaStorageManagerImpl(
+                clusterService,
+                cmgManager,
+                logicalTopologyService,
+                raftManager,
+                storage,
+                clock,
+                topologyAwareRaftGroupServiceFactory,
+                metaStorageConfiguration
+        );
+
+        assertThat(
+                startAsync(clusterService, raftManager, metaStorageManager)
+                        .thenCompose(unused -> metaStorageManager.deployWatches()),
+                willCompleteSuccessfully()
+        );
+    }
+
+    @AfterEach
+    void tearDown() throws Exception {
+        List<IgniteComponent> components = List.of(metaStorageManager, raftManager, clusterService);
+
+        closeAll(Stream.concat(
+                components.stream().map(c -> c::beforeNodeStop),
+                Stream.of(() -> assertThat(stopAsync(components), willCompleteSuccessfully()))

Review Comment:
   Nice!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599975617


##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -278,4 +309,50 @@ boolean beforeApply(Command command) {
 
         return false;
     }
+
+    private static class IdempotentCommandCache {
+        @Nullable
+        final Serializable result;
+
+        final HybridTimestamp commandStartTime;

Review Comment:
   E.g. because we don't need HybridTimestampHere. Up to you. Not that important.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599972930


##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -72,6 +80,29 @@ public class MetaStorageWriteHandler {
     void handleWriteCommand(CommandClosure<WriteCommand> clo) {
         WriteCommand command = clo.command();
 
+        CommandClosure<WriteCommand> resultClosure;
+
+        if (command instanceof IdempotentCommand) {
+            CommandId commandId = ((IdempotentCommand) command).id();
+            IdempotentCommandCache cache = idempotentCommandCache.get(commandId);

Review Comment:
   Yep, cache.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-21881 Deal with retry send metastorage raft commands after a timeout [ignite-3]

Posted by "denis-chudov (via GitHub)" <gi...@apache.org>.
denis-chudov commented on code in PR #3725:
URL: https://github.com/apache/ignite-3/pull/3725#discussion_r1599947566


##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -278,4 +309,50 @@ boolean beforeApply(Command command) {
 
         return false;
     }
+
+    private static class IdempotentCommandCache {
+        @Nullable
+        final Serializable result;
+
+        final HybridTimestamp commandStartTime;
+
+        IdempotentCommandCache(@Nullable Serializable result, HybridTimestamp commandStartTime) {
+            this.result = result;
+            this.commandStartTime = commandStartTime;
+        }
+    }
+
+    private class ResultCachingClosure implements CommandClosure<WriteCommand> {
+        CommandClosure<WriteCommand> closure;
+
+        ResultCachingClosure(CommandClosure<WriteCommand> closure) {
+            this.closure = closure;
+
+            assert closure.command() instanceof IdempotentCommand;
+        }
+
+        @Override
+        public long index() {
+            return closure.index();
+        }
+
+        @Override
+        public long term() {
+            return closure.term();
+        }
+
+        @Override
+        public WriteCommand command() {
+            return closure.command();
+        }
+
+        @Override
+        public void result(@Nullable Serializable res) {
+            IdempotentCommand command = (IdempotentCommand) closure.command();
+
+            idempotentCommandCache.put(command.id(), new IdempotentCommandCache(res, command.initiatorTime()));

Review Comment:
   fixed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org