You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "SammyVimes (via GitHub)" <gi...@apache.org> on 2023/03/30 11:10:08 UTC
[GitHub] [ignite-3] SammyVimes opened a new pull request, #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
SammyVimes opened a new pull request, #1869:
URL: https://github.com/apache/ignite-3/pull/1869
https://issues.apache.org/jira/browse/IGNITE-19028
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1160879823
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/SyncTimeCommand.java:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.command;
+
+import org.apache.ignite.internal.raft.WriteCommand;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Command that initiates idle safe time synchronization.
+ */
+@Transferable(MetastorageCommandsMessageGroup.SYNC_TIME)
+public interface SyncTimeCommand extends WriteCommand {
+ /** New safe time. */
+ HybridTimestampMessage safeTime();
Review Comment:
It simply doesn't need metastorage write message's fields
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1159639824
##########
modules/core/src/main/java/org/apache/ignite/internal/hlc/HybridTimestamp.java:
##########
@@ -68,10 +67,9 @@ public HybridTimestamp(long physical, int logical) {
* @param times Times for comparing.
* @return The highest hybrid timestamp.
*/
- public static @Nullable HybridTimestamp max(HybridTimestamp... times) {
- if (times.length == 0) {
- return null;
- }
+ public static HybridTimestamp max(HybridTimestamp... times) {
+ assert times != null;
+ assert times.length > 0;
Review Comment:
Anyway, it's so weird to see a vararg max method, what's the point :)
You're not the author, I know
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1162381908
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/time/ClusterTimeImpl.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.server.time;
+
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.metastorage.impl.MetaStorageServiceImpl;
+import org.apache.ignite.internal.util.IgniteSpinBusyLock;
+import org.apache.ignite.internal.util.PendingComparableValuesTracker;
+import org.jetbrains.annotations.Nullable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Cluster time implementation with additional methods to adjust time and update safe time.
+ */
+public class ClusterTimeImpl implements ClusterTime {
+ private final IgniteSpinBusyLock busyLock;
+
+ private volatile @Nullable LeaderTimer leaderTimer;
+
+ private final HybridClock clock;
+
+ private final PendingComparableValuesTracker<HybridTimestamp> safeTime;
+
+ /**
+ * Constructor.
+ *
+ * @param busyLock Busy lock.
+ * @param clock Node's hybrid clock.
+ */
+ public ClusterTimeImpl(IgniteSpinBusyLock busyLock, HybridClock clock) {
+ this.busyLock = busyLock;
+ this.clock = clock;
+ this.safeTime = new PendingComparableValuesTracker<>(clock.now());
+ }
+
+ /**
+ * Starts sync time scheduler.
+ *
+ * @param service MetaStorage service that is used by scheduler to sync time.
+ */
+ public void startLeaderTimer(MetaStorageServiceImpl service) {
+ if (!busyLock.enterBusy()) {
+ return;
+ }
+
+ try {
+ assert leaderTimer == null;
+
+ LeaderTimer newTimer = new LeaderTimer(service);
+
+ leaderTimer = newTimer;
+
+ newTimer.start();
+ } finally {
+ busyLock.leaveBusy();
+ }
+ }
+
+ /**
+ * Stops sync time scheduler if it exists.
+ */
+ public void stopLeaderTimer() {
+ LeaderTimer timer = leaderTimer;
+
+ if (timer != null) {
+ timer.stop();
+
+ leaderTimer = null;
+ }
+ }
+
+ @Override
+ public HybridTimestamp now() {
+ return clock.now();
+ }
+
+ @Override
+ public CompletableFuture<Void> waitFor(HybridTimestamp time) {
+ return safeTime.waitFor(time);
+ }
+
+ /**
+ * Updates the internal safe time.
+ *
+ * @param newValue New safe time value.
+ */
+ public void updateSafeTime(HybridTimestamp newValue) {
Review Comment:
I thought they are always called together
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1161590847
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageListener.java:
##########
@@ -217,6 +219,11 @@ public void onWrite(Iterator<CommandClosure<WriteCommand>> iter) {
}
}
+ @Override
+ public void onBeforeApply(Command command) {
+ writeHandler.beforeApply(command);
Review Comment:
Same question here, why do we modify command in metastorage listener?
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/impl/MetaStorageServiceImpl.java:
##########
@@ -258,6 +275,21 @@ public Publisher<Entry> prefix(ByteArray prefix, long revUpperBound) {
return new CursorPublisher(context, createPrefixCommand);
}
+ /**
+ * Sends idle safe time sync message. Should be called only on the leader node.
+ *
+ * @param safeTime New safe time.
+ * @return Future that will be completed when message is sent.
+ */
+ public CompletableFuture<Void> syncTime(HybridTimestamp safeTime) {
+ // TODO: https://issues.apache.org/jira/browse/IGNITE-19199 Only propagate safe time when ms is idle
Review Comment:
So, the "idle" sync is not implemented yet, is that correct?
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/time/ClusterTimeImpl.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.server.time;
+
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.metastorage.impl.MetaStorageServiceImpl;
+import org.apache.ignite.internal.util.IgniteSpinBusyLock;
+import org.apache.ignite.internal.util.PendingComparableValuesTracker;
+import org.jetbrains.annotations.Nullable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Cluster time implementation with additional methods to adjust time and update safe time.
+ */
+public class ClusterTimeImpl implements ClusterTime {
+ private final IgniteSpinBusyLock busyLock;
+
+ private volatile @Nullable LeaderTimer leaderTimer;
+
+ private final HybridClock clock;
+
+ private final PendingComparableValuesTracker<HybridTimestamp> safeTime;
+
+ /**
+ * Constructor.
+ *
+ * @param busyLock Busy lock.
+ * @param clock Node's hybrid clock.
+ */
+ public ClusterTimeImpl(IgniteSpinBusyLock busyLock, HybridClock clock) {
+ this.busyLock = busyLock;
+ this.clock = clock;
+ this.safeTime = new PendingComparableValuesTracker<>(clock.now());
Review Comment:
I believe that we have a very nice bug here. Why is it safe to initialize safeTime as `now()` during the node start?
The answer is - it's not safe. Metastorage safe time should be far in the past during the start, to preserve the monotony of time in new entries (during rebalance, in particular)
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneManagerLogicalTopologyEventsTest.java:
##########
@@ -151,15 +155,22 @@ private DistributionZoneManager prepareDistributionZoneManager() {
keyValueStorage = spy(new SimpleInMemoryKeyValueStorage("test"));
- MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage);
+ MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage, mock(ClusterTimeImpl.class));
RaftGroupService metaStorageService = mock(RaftGroupService.class);
+ HybridTimestampMessage mockTsMessage = mock(HybridTimestampMessage.class);
+ when(mockTsMessage.asHybridTimestamp()).thenReturn(new HybridTimestamp(10, 10));
+
// Delegate directly to listener.
lenient().doAnswer(
invocationClose -> {
Review Comment:
As far as I'm aware, @sashapolo is removing the copy-paste. So one of you will have to fix conflicts
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageLearnerListener.java:
##########
@@ -56,6 +58,11 @@ public void onWrite(Iterator<CommandClosure<WriteCommand>> iter) {
}
}
+ @Override
+ public void onBeforeApply(Command command) {
+ writeHandler.beforeApply(command);
Review Comment:
Why do we modify the command on learners? Command should already be correct, otherwise I simply don't get it.
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/MetaStorageWriteCommand.java:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.command;
+
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.raft.Command;
+import org.apache.ignite.internal.raft.WriteCommand;
+import org.apache.ignite.network.annotations.WithSetter;
+
+/** Base meta storage write command. */
+public interface MetaStorageWriteCommand extends WriteCommand {
+ /**
+ * Returns time on the initiator node.
+ */
+ HybridTimestampMessage initiatorTime();
+
+ /**
+ * This is a dirty hack. This time is set by the leader node to disseminate new safe time across
+ * followers and learners. Leader of the ms group reads {@link #initiatorTime()}, adjusts its clock
+ * and sets safeTime as {@link HybridClock#now()} as safeTime here. This must be done before
+ * command is saved into the Raft log (see {@link org.apache.ignite.internal.raft.service.RaftGroupListener#onBeforeApply(Command)}.
+ */
+ @WithSetter
+ HybridTimestampMessage safeTime();
+
+ /**
+ * Setter for the safeTime field.
+ */
+ default void safeTime(HybridTimestampMessage safeTime) {
Review Comment:
Opps, I didn't yet read that one
##########
modules/metastorage/src/integrationTest/java/org/apache/ignite/internal/metastorage/impl/ItMetaStorageMultipleNodesTest.java:
##########
@@ -375,4 +382,40 @@ void testLearnerLeaveAndJoinBecauseOfNetworkPartition(TestInfo testInfo) throws
assertTrue(waitForCondition(() -> firstNode.getMetaStorageLearners().join().equals(Set.of(secondNode.name())), 10_000));
}
+
+ /**
+ * Tests that safe time is propagated from the leader to the follower/learner.
+ */
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testSafeTimePropagation(boolean useFollower, TestInfo testInfo) throws Exception {
Review Comment:
@tkalkirill could you please clarify what you meant?
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/time/ClusterTimeImpl.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.server.time;
+
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.metastorage.impl.MetaStorageServiceImpl;
+import org.apache.ignite.internal.util.IgniteSpinBusyLock;
+import org.apache.ignite.internal.util.PendingComparableValuesTracker;
+import org.jetbrains.annotations.Nullable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Cluster time implementation with additional methods to adjust time and update safe time.
+ */
+public class ClusterTimeImpl implements ClusterTime {
+ private final IgniteSpinBusyLock busyLock;
+
+ private volatile @Nullable LeaderTimer leaderTimer;
+
+ private final HybridClock clock;
+
+ private final PendingComparableValuesTracker<HybridTimestamp> safeTime;
+
+ /**
+ * Constructor.
+ *
+ * @param busyLock Busy lock.
+ * @param clock Node's hybrid clock.
+ */
+ public ClusterTimeImpl(IgniteSpinBusyLock busyLock, HybridClock clock) {
+ this.busyLock = busyLock;
+ this.clock = clock;
+ this.safeTime = new PendingComparableValuesTracker<>(clock.now());
+ }
+
+ /**
+ * Starts sync time scheduler.
+ *
+ * @param service MetaStorage service that is used by scheduler to sync time.
+ */
+ public void startLeaderTimer(MetaStorageServiceImpl service) {
+ if (!busyLock.enterBusy()) {
+ return;
+ }
+
+ try {
+ assert leaderTimer == null;
+
+ LeaderTimer newTimer = new LeaderTimer(service);
+
+ leaderTimer = newTimer;
+
+ newTimer.start();
+ } finally {
+ busyLock.leaveBusy();
+ }
+ }
+
+ /**
+ * Stops sync time scheduler if it exists.
+ */
+ public void stopLeaderTimer() {
+ LeaderTimer timer = leaderTimer;
+
+ if (timer != null) {
+ timer.stop();
+
+ leaderTimer = null;
+ }
+ }
+
+ @Override
+ public HybridTimestamp now() {
+ return clock.now();
+ }
+
+ @Override
+ public CompletableFuture<Void> waitFor(HybridTimestamp time) {
+ return safeTime.waitFor(time);
+ }
+
+ /**
+ * Updates the internal safe time.
+ *
+ * @param newValue New safe time value.
+ */
+ public void updateSafeTime(HybridTimestamp newValue) {
Review Comment:
Can this one and the next methods be combined into one?
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/time/ClusterTimeImpl.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.server.time;
+
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.metastorage.impl.MetaStorageServiceImpl;
+import org.apache.ignite.internal.util.IgniteSpinBusyLock;
+import org.apache.ignite.internal.util.PendingComparableValuesTracker;
+import org.jetbrains.annotations.Nullable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Cluster time implementation with additional methods to adjust time and update safe time.
+ */
+public class ClusterTimeImpl implements ClusterTime {
+ private final IgniteSpinBusyLock busyLock;
+
+ private volatile @Nullable LeaderTimer leaderTimer;
+
+ private final HybridClock clock;
+
+ private final PendingComparableValuesTracker<HybridTimestamp> safeTime;
+
+ /**
+ * Constructor.
+ *
+ * @param busyLock Busy lock.
+ * @param clock Node's hybrid clock.
+ */
+ public ClusterTimeImpl(IgniteSpinBusyLock busyLock, HybridClock clock) {
+ this.busyLock = busyLock;
+ this.clock = clock;
+ this.safeTime = new PendingComparableValuesTracker<>(clock.now());
+ }
+
+ /**
+ * Starts sync time scheduler.
+ *
+ * @param service MetaStorage service that is used by scheduler to sync time.
+ */
+ public void startLeaderTimer(MetaStorageServiceImpl service) {
+ if (!busyLock.enterBusy()) {
+ return;
+ }
+
+ try {
+ assert leaderTimer == null;
+
+ LeaderTimer newTimer = new LeaderTimer(service);
+
+ leaderTimer = newTimer;
+
+ newTimer.start();
+ } finally {
+ busyLock.leaveBusy();
+ }
+ }
+
+ /**
+ * Stops sync time scheduler if it exists.
+ */
+ public void stopLeaderTimer() {
+ LeaderTimer timer = leaderTimer;
Review Comment:
No busy lock here, why?
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -234,4 +255,18 @@ private static RevisionCondition.Type toRevisionConditionType(ConditionType type
throw new IllegalArgumentException("Unexpected revision condition type " + type);
}
}
+
+ void beforeApply(Command command) {
+ if (command instanceof MetaStorageWriteCommand) {
+ // Initiator sends us a timestamp to adjust to.
+ // Alter command by setting safe time based on the adjusted clock.
+ clusterTime.adjust(((MetaStorageWriteCommand) command).initiatorTime().asHybridTimestamp());
Review Comment:
I think you may extract `(MetaStorageWriteCommand) command` into a variable, but it's up to you
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -234,4 +253,18 @@ private static RevisionCondition.Type toRevisionConditionType(ConditionType type
throw new IllegalArgumentException("Unexpected revision condition type " + type);
}
}
+
+ void beforeApply(Command command) {
Review Comment:
Why do we call it in learner listener?
##########
modules/placement-driver/src/integrationTest/java/org/apache/ignite/internal/placementdriver/PlacementDriverManagerTest.java:
##########
@@ -142,11 +142,12 @@ private void startPlacementDriverManager() throws NodeStoppingException {
eventsClientListener
);
+ HybridClockImpl nodeClock = new HybridClockImpl();
Review Comment:
You did it again :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1160765908
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/MetaStorageWriteCommand.java:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.command;
+
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.raft.Command;
+import org.apache.ignite.internal.raft.WriteCommand;
+import org.apache.ignite.network.annotations.WithSetter;
+
+/** Base meta storage write command. */
+public interface MetaStorageWriteCommand extends WriteCommand {
+ /**
+ * Returns time on the initiator node.
+ */
+ HybridTimestampMessage initiatorTime();
+
+ /**
+ * This is a dirty hack. This time is set by the leader node to disseminate new safe time across
+ * followers and learners. Leader of the ms group reads {@link #initiatorTime()}, adjusts its clock
+ * and sets safeTime as {@link HybridClock#now()} as safeTime here. This must be done before
+ * command is saved into the Raft log (see {@link org.apache.ignite.internal.raft.service.RaftGroupListener#onBeforeApply(Command)}.
+ */
+ @WithSetter
+ HybridTimestampMessage safeTime();
+
+ /**
+ * Setter for the safeTime field.
+ */
+ default void safeTime(HybridTimestampMessage safeTime) {
Review Comment:
See comment to the "WithSetter" annotation. Without this default method there'd be no way to access the method on the interface
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "tkalkirill (via GitHub)" <gi...@apache.org>.
tkalkirill commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1160617287
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/impl/MetaStorageServiceImpl.java:
##########
@@ -258,6 +275,20 @@ public Publisher<Entry> prefix(ByteArray prefix, long revUpperBound) {
return new CursorPublisher(context, createPrefixCommand);
}
+ /**
+ * Sends safe time sync message. Should be called only on the leader node.
+ *
+ * @param safeTime New safe time.
+ * @return Future that will be completed when message is sent.
+ */
+ public CompletableFuture<Void> syncTime(HybridTimestamp safeTime) {
Review Comment:
Where todo ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1161583740
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/MetaStorageWriteCommand.java:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.command;
+
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.raft.Command;
+import org.apache.ignite.internal.raft.WriteCommand;
+import org.apache.ignite.network.annotations.WithSetter;
+
+/** Base meta storage write command. */
+public interface MetaStorageWriteCommand extends WriteCommand {
+ /**
+ * Returns time on the initiator node.
+ */
+ HybridTimestampMessage initiatorTime();
+
+ /**
+ * This is a dirty hack. This time is set by the leader node to disseminate new safe time across
+ * followers and learners. Leader of the ms group reads {@link #initiatorTime()}, adjusts its clock
+ * and sets safeTime as {@link HybridClock#now()} as safeTime here. This must be done before
+ * command is saved into the Raft log (see {@link org.apache.ignite.internal.raft.service.RaftGroupListener#onBeforeApply(Command)}.
+ */
+ @WithSetter
+ HybridTimestampMessage safeTime();
+
+ /**
+ * Setter for the safeTime field.
+ */
+ default void safeTime(HybridTimestampMessage safeTime) {
Review Comment:
But it is explained in the WithSetter annotation (with an example!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1159372644
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageLearnerListener.java:
##########
@@ -56,6 +58,11 @@ public void onWrite(Iterator<CommandClosure<WriteCommand>> iter) {
}
}
+ @Override
+ public void onBeforeApply(Command command) {
+ writeHandler.beforeApply(command);
Review Comment:
Because we need to change the command before it's saved to log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1159390853
##########
modules/core/src/main/java/org/apache/ignite/internal/hlc/HybridTimestamp.java:
##########
@@ -68,10 +67,9 @@ public HybridTimestamp(long physical, int logical) {
* @param times Times for comparing.
* @return The highest hybrid timestamp.
*/
- public static @Nullable HybridTimestamp max(HybridTimestamp... times) {
- if (times.length == 0) {
- return null;
- }
+ public static HybridTimestamp max(HybridTimestamp... times) {
+ assert times != null;
+ assert times.length > 0;
Review Comment:
Imho IllegalArgumentException is for public API. If someone passes an empty array here -- it's a bug in our own code, hence the assertion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1161583435
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/SyncTimeCommand.java:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.command;
+
+import org.apache.ignite.internal.raft.WriteCommand;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Command that initiates idle safe time synchronization.
+ */
+@Transferable(MetastorageCommandsMessageGroup.SYNC_TIME)
+public interface SyncTimeCommand extends WriteCommand {
+ /** New safe time. */
+ HybridTimestampMessage safeTime();
Review Comment:
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1160765251
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/HybridTimestampMessage.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.command;
+
+import java.io.Serializable;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.network.NetworkMessage;
+import org.apache.ignite.network.annotations.Transferable;
+
+/** Message with a {@link HybridTimestamp}. */
+@Transferable(MetastorageCommandsMessageGroup.HYBRID_TS)
+public interface HybridTimestampMessage extends NetworkMessage, Serializable {
Review Comment:
It is, but we don't have access to that module and I'm not sure if this should be in a common module
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes merged pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes merged PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1161483684
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/MetaStorageWriteCommand.java:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.command;
+
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.raft.Command;
+import org.apache.ignite.internal.raft.WriteCommand;
+import org.apache.ignite.network.annotations.WithSetter;
+
+/** Base meta storage write command. */
+public interface MetaStorageWriteCommand extends WriteCommand {
+ /**
+ * Returns time on the initiator node.
+ */
+ HybridTimestampMessage initiatorTime();
+
+ /**
+ * This is a dirty hack. This time is set by the leader node to disseminate new safe time across
+ * followers and learners. Leader of the ms group reads {@link #initiatorTime()}, adjusts its clock
+ * and sets safeTime as {@link HybridClock#now()} as safeTime here. This must be done before
+ * command is saved into the Raft log (see {@link org.apache.ignite.internal.raft.service.RaftGroupListener#onBeforeApply(Command)}.
+ */
+ @WithSetter
+ HybridTimestampMessage safeTime();
+
+ /**
+ * Setter for the safeTime field.
+ */
+ default void safeTime(HybridTimestampMessage safeTime) {
Review Comment:
I see it now. If it didn't have a default implementation, annotation processor would generate the property with the same name. That's why you need it.
It's not explained in the comment by the way, that's why I asked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1161702945
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/impl/MetaStorageServiceImpl.java:
##########
@@ -258,6 +275,21 @@ public Publisher<Entry> prefix(ByteArray prefix, long revUpperBound) {
return new CursorPublisher(context, createPrefixCommand);
}
+ /**
+ * Sends idle safe time sync message. Should be called only on the leader node.
+ *
+ * @param safeTime New safe time.
+ * @return Future that will be completed when message is sent.
+ */
+ public CompletableFuture<Void> syncTime(HybridTimestamp safeTime) {
+ // TODO: https://issues.apache.org/jira/browse/IGNITE-19199 Only propagate safe time when ms is idle
Review Comment:
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1159370015
##########
modules/network-api/src/main/java/org/apache/ignite/network/annotations/WithSetter.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.network.annotations;
+
+/**
+ * Annotation for the {@link Transferable} class methods. If a method is marked with this annotation,
+ * a setter for the field with the same name as method's will be generated.
+ * In order to have access to this setter via interface one can use default method:
+ * <pre>
+ * {@code @WithSetter
+ * HybridTimestampMessage safeTime();
+ *
+ * default void safeTime(HybridTimestampMessage safeTime) {
+ * // No-op.
+ * }
+ * }
+ * </pre>
+ * Note that fields with setters will not be final.
+ */
+public @interface WithSetter {
Review Comment:
This is needed only for current ticket and it's literally 5 extra lines in annotation processor generator
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1159369537
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/impl/MetaStorageRaftGroupEventsListener.java:
##########
@@ -156,11 +178,16 @@ private void executeIfLeader(OnLeaderAction action) {
}
private CompletableFuture<Void> executeIfLeaderImpl(OnLeaderAction action) {
+ return executeWithStatus((service, term, isLeader) -> action.apply(service, term));
Review Comment:
What do you mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1161483052
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneManagerScaleUpTest.java:
##########
@@ -198,7 +199,7 @@ public void setUp() {
keyValueStorage = spy(new SimpleInMemoryKeyValueStorage("test"));
- MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage);
+ MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage, mock(ClusterTimeImpl.class));
Review Comment:
Then why do you mock it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1161705693
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/time/ClusterTimeImpl.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.server.time;
+
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.metastorage.impl.MetaStorageServiceImpl;
+import org.apache.ignite.internal.util.IgniteSpinBusyLock;
+import org.apache.ignite.internal.util.PendingComparableValuesTracker;
+import org.jetbrains.annotations.Nullable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Cluster time implementation with additional methods to adjust time and update safe time.
+ */
+public class ClusterTimeImpl implements ClusterTime {
+ private final IgniteSpinBusyLock busyLock;
+
+ private volatile @Nullable LeaderTimer leaderTimer;
+
+ private final HybridClock clock;
+
+ private final PendingComparableValuesTracker<HybridTimestamp> safeTime;
+
+ /**
+ * Constructor.
+ *
+ * @param busyLock Busy lock.
+ * @param clock Node's hybrid clock.
+ */
+ public ClusterTimeImpl(IgniteSpinBusyLock busyLock, HybridClock clock) {
+ this.busyLock = busyLock;
+ this.clock = clock;
+ this.safeTime = new PendingComparableValuesTracker<>(clock.now());
Review Comment:
Right, I'll make it:
`this.safeTime = new PendingComparableValuesTracker<>(new HybridTimestamp(1, 0));`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1160764001
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneManagerScaleUpTest.java:
##########
@@ -198,7 +199,7 @@ public void setUp() {
keyValueStorage = spy(new SimpleInMemoryKeyValueStorage("test"));
- MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage);
+ MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage, mock(ClusterTimeImpl.class));
Review Comment:
It shouldn't
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1160952744
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -234,4 +253,18 @@ private static RevisionCondition.Type toRevisionConditionType(ConditionType type
throw new IllegalArgumentException("Unexpected revision condition type " + type);
}
}
+
+ void beforeApply(Command command) {
Review Comment:
This method is only executed on the leader, I thought it's clear from the javadoc, but probably not enough as you're not the first who's asking it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1161483930
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/SyncTimeCommand.java:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.command;
+
+import org.apache.ignite.internal.raft.WriteCommand;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Command that initiates idle safe time synchronization.
+ */
+@Transferable(MetastorageCommandsMessageGroup.SYNC_TIME)
+public interface SyncTimeCommand extends WriteCommand {
+ /** New safe time. */
+ HybridTimestampMessage safeTime();
Review Comment:
Like the "initial time"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1161584285
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneManagerScaleUpTest.java:
##########
@@ -198,7 +199,7 @@ public void setUp() {
keyValueStorage = spy(new SimpleInMemoryKeyValueStorage("test"));
- MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage);
+ MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage, mock(ClusterTimeImpl.class));
Review Comment:
I don't know, mocking seems easier to me. Shows that we don't care about this object
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1161703409
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageListener.java:
##########
@@ -217,6 +219,11 @@ public void onWrite(Iterator<CommandClosure<WriteCommand>> iter) {
}
}
+ @Override
+ public void onBeforeApply(Command command) {
+ writeHandler.beforeApply(command);
Review Comment:
MetaStorage listener's `onBeforeApply` is the one who is called on leader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "tkalkirill (via GitHub)" <gi...@apache.org>.
tkalkirill commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1159259377
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/MetastorageCommandsMessageGroup.java:
##########
@@ -84,4 +84,8 @@ public interface MetastorageCommandsMessageGroup {
/** Message type for {@link CloseAllCursorsCommand}. */
short CLOSE_ALL_CURSORS = 64;
+
+ short HYBRID_TS = 65;
+
+ short SYNC_TIME = 66;
Review Comment:
Missing javadoc
##########
modules/core/src/main/java/org/apache/ignite/internal/hlc/HybridTimestamp.java:
##########
@@ -68,10 +67,9 @@ public HybridTimestamp(long physical, int logical) {
* @param times Times for comparing.
* @return The highest hybrid timestamp.
*/
- public static @Nullable HybridTimestamp max(HybridTimestamp... times) {
- if (times.length == 0) {
- return null;
- }
+ public static HybridTimestamp max(HybridTimestamp... times) {
+ assert times != null;
+ assert times.length > 0;
Review Comment:
Please indicate it (`times.length > 0`) in the documentation and also correct the description of the method itself.
Or maybe `throw new IllegalArgumentException` ?
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/MetastorageCommandsMessageGroup.java:
##########
@@ -84,4 +84,8 @@ public interface MetastorageCommandsMessageGroup {
/** Message type for {@link CloseAllCursorsCommand}. */
short CLOSE_ALL_CURSORS = 64;
+
+ short HYBRID_TS = 65;
Review Comment:
Missing javadoc
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/impl/MetaStorageRaftGroupEventsListener.java:
##########
@@ -156,11 +178,16 @@ private void executeIfLeader(OnLeaderAction action) {
}
private CompletableFuture<Void> executeIfLeaderImpl(OnLeaderAction action) {
+ return executeWithStatus((service, term, isLeader) -> action.apply(service, term));
Review Comment:
It seems that here it is necessary to return `completedFuture(null)` if not the leader.
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/impl/MetaStorageManagerImpl.java:
##########
@@ -109,6 +112,8 @@ public class MetaStorageManagerImpl implements MetaStorageManager {
/** Prevents double stopping of the component. */
private final AtomicBoolean isStopped = new AtomicBoolean();
+ private final ClusterTimeImpl clusterTime;
Review Comment:
I think we should also stop at `MetaStorageManagerImpl#stop`
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/impl/MetaStorageRaftGroupEventsListener.java:
##########
@@ -128,6 +146,10 @@ private interface OnLeaderAction {
CompletableFuture<Void> apply(MetaStorageServiceImpl service, long term);
}
+ private interface OnStatusAction {
Review Comment:
Missing `FunctionalInterface`.
Can get rid of the `OnLeaderAction`?
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/time/ClusterTime.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.server.time;
+
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+
+/**
+ * Cluster time with a hybrid clock instance and access to safe time.
+ */
+public interface ClusterTime {
+ /**
+ * Returns current cluster time.
+ *
+ * @return Current cluster time.
Review Comment:
```suggestion
```
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/impl/MetaStorageServiceImpl.java:
##########
@@ -258,6 +275,20 @@ public Publisher<Entry> prefix(ByteArray prefix, long revUpperBound) {
return new CursorPublisher(context, createPrefixCommand);
}
+ /**
+ * Sends safe time sync message. Should be called only on the leader node.
+ *
+ * @param safeTime New safe time.
+ * @return Future that will be completed when message is sent.
+ */
+ public CompletableFuture<Void> syncTime(HybridTimestamp safeTime) {
Review Comment:
I noticed that you do not use the return value, would it not be a mistake to always send a new safeTime even if we did not process the previous one.
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageLearnerListener.java:
##########
@@ -56,6 +58,11 @@ public void onWrite(Iterator<CommandClosure<WriteCommand>> iter) {
}
}
+ @Override
+ public void onBeforeApply(Command command) {
+ writeHandler.beforeApply(command);
Review Comment:
Why before and not on command execution?
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/time/ClusterTimeImpl.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.server.time;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.impl.MetaStorageServiceImpl;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteSpinBusyLock;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.PendingComparableValuesTracker;
+import org.jetbrains.annotations.Nullable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Cluster time implementation with additional methods to adjust time and update safe time.
+ */
+public class ClusterTimeImpl implements ClusterTime {
+ /** The logger. */
+ private static final IgniteLogger LOG = Loggers.forClass(ClusterTimeImpl.class);
+
+ private final IgniteSpinBusyLock busyLock;
+
+ private volatile @Nullable LeaderTimer leaderTimer;
+
+ private final HybridClock clock;
+
+ private final PendingComparableValuesTracker<HybridTimestamp> safeTime;
+
+ /**
+ * Constructor.
+ *
+ * @param busyLock Busy lock.
+ * @param clock Node's hybrid clock.
+ */
+ public ClusterTimeImpl(IgniteSpinBusyLock busyLock, HybridClock clock) {
+ this.busyLock = busyLock;
+ this.clock = clock;
+ this.safeTime = new PendingComparableValuesTracker<>(clock.now());
+ }
+
+ /**
+ * Starts sync time scheduler.
+ *
+ * @param service MetaStorage service that is used by scheduler to sync time.
+ */
+ public void startLeaderTimer(MetaStorageServiceImpl service) {
+ if (!busyLock.enterBusy()) {
+ return;
+ }
+
+ try {
+ assert leaderTimer == null;
+
+ LeaderTimer newTimer = new LeaderTimer(service);
+
+ leaderTimer = newTimer;
+
+ newTimer.start();
+ } finally {
+ busyLock.leaveBusy();
+ }
+ }
+
+ /**
+ * Stops sync time scheduler.
+ */
+ public void stopLeaderTimer() {
+ LeaderTimer timer = leaderTimer;
+
+ assert timer != null;
+
+ timer.stop();
+
+ leaderTimer = null;
+ }
+
+ @Override
+ public HybridTimestamp now() {
+ return clock.now();
+ }
+
+ @Override
+ public CompletableFuture<Void> waitFor(HybridTimestamp time) {
+ return safeTime.waitFor(time);
+ }
+
+ public void updateSafeTime(HybridTimestamp ts) {
+ this.safeTime.update(ts);
+ }
+
+ public void adjust(HybridTimestamp ts) {
Review Comment:
Little confuse name
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/time/ClusterTimeImpl.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.server.time;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.impl.MetaStorageServiceImpl;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteSpinBusyLock;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.PendingComparableValuesTracker;
+import org.jetbrains.annotations.Nullable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Cluster time implementation with additional methods to adjust time and update safe time.
+ */
+public class ClusterTimeImpl implements ClusterTime {
+ /** The logger. */
+ private static final IgniteLogger LOG = Loggers.forClass(ClusterTimeImpl.class);
+
+ private final IgniteSpinBusyLock busyLock;
+
+ private volatile @Nullable LeaderTimer leaderTimer;
+
+ private final HybridClock clock;
+
+ private final PendingComparableValuesTracker<HybridTimestamp> safeTime;
+
+ /**
+ * Constructor.
+ *
+ * @param busyLock Busy lock.
+ * @param clock Node's hybrid clock.
+ */
+ public ClusterTimeImpl(IgniteSpinBusyLock busyLock, HybridClock clock) {
+ this.busyLock = busyLock;
+ this.clock = clock;
+ this.safeTime = new PendingComparableValuesTracker<>(clock.now());
+ }
+
+ /**
+ * Starts sync time scheduler.
+ *
+ * @param service MetaStorage service that is used by scheduler to sync time.
+ */
+ public void startLeaderTimer(MetaStorageServiceImpl service) {
+ if (!busyLock.enterBusy()) {
+ return;
+ }
+
+ try {
+ assert leaderTimer == null;
+
+ LeaderTimer newTimer = new LeaderTimer(service);
+
+ leaderTimer = newTimer;
+
+ newTimer.start();
+ } finally {
+ busyLock.leaveBusy();
+ }
+ }
+
+ /**
+ * Stops sync time scheduler.
+ */
+ public void stopLeaderTimer() {
+ LeaderTimer timer = leaderTimer;
+
+ assert timer != null;
+
+ timer.stop();
+
+ leaderTimer = null;
+ }
+
+ @Override
+ public HybridTimestamp now() {
+ return clock.now();
+ }
+
+ @Override
+ public CompletableFuture<Void> waitFor(HybridTimestamp time) {
+ return safeTime.waitFor(time);
+ }
+
+ public void updateSafeTime(HybridTimestamp ts) {
+ this.safeTime.update(ts);
+ }
+
+ public void adjust(HybridTimestamp ts) {
+ this.clock.update(ts);
+ }
+
+ private class LeaderTimer {
+
+ private final MetaStorageServiceImpl service;
+
+ /** Scheduled executor for cluster time sync. */
+ private final ScheduledExecutorService scheduledClusterTimeSyncExecutor =
+ Executors.newScheduledThreadPool(1, new NamedThreadFactory("scheduled-cluster-time-sync-thread", LOG));
Review Comment:
Please add nodeName
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/time/ClusterTimeImpl.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.server.time;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.impl.MetaStorageServiceImpl;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteSpinBusyLock;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.PendingComparableValuesTracker;
+import org.jetbrains.annotations.Nullable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Cluster time implementation with additional methods to adjust time and update safe time.
+ */
+public class ClusterTimeImpl implements ClusterTime {
+ /** The logger. */
+ private static final IgniteLogger LOG = Loggers.forClass(ClusterTimeImpl.class);
+
+ private final IgniteSpinBusyLock busyLock;
+
+ private volatile @Nullable LeaderTimer leaderTimer;
+
+ private final HybridClock clock;
+
+ private final PendingComparableValuesTracker<HybridTimestamp> safeTime;
+
+ /**
+ * Constructor.
+ *
+ * @param busyLock Busy lock.
+ * @param clock Node's hybrid clock.
+ */
+ public ClusterTimeImpl(IgniteSpinBusyLock busyLock, HybridClock clock) {
+ this.busyLock = busyLock;
+ this.clock = clock;
+ this.safeTime = new PendingComparableValuesTracker<>(clock.now());
+ }
+
+ /**
+ * Starts sync time scheduler.
+ *
+ * @param service MetaStorage service that is used by scheduler to sync time.
+ */
+ public void startLeaderTimer(MetaStorageServiceImpl service) {
+ if (!busyLock.enterBusy()) {
+ return;
+ }
+
+ try {
+ assert leaderTimer == null;
+
+ LeaderTimer newTimer = new LeaderTimer(service);
+
+ leaderTimer = newTimer;
+
+ newTimer.start();
+ } finally {
+ busyLock.leaveBusy();
+ }
+ }
+
+ /**
+ * Stops sync time scheduler.
+ */
+ public void stopLeaderTimer() {
+ LeaderTimer timer = leaderTimer;
+
+ assert timer != null;
+
+ timer.stop();
+
+ leaderTimer = null;
+ }
+
+ @Override
+ public HybridTimestamp now() {
+ return clock.now();
+ }
+
+ @Override
+ public CompletableFuture<Void> waitFor(HybridTimestamp time) {
+ return safeTime.waitFor(time);
+ }
+
+ public void updateSafeTime(HybridTimestamp ts) {
+ this.safeTime.update(ts);
+ }
+
+ public void adjust(HybridTimestamp ts) {
+ this.clock.update(ts);
+ }
+
+ private class LeaderTimer {
Review Comment:
Why not separate class ?
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -66,6 +76,10 @@ class MetaStorageWriteHandler {
boolean handleWriteCommand(CommandClosure<WriteCommand> clo) {
WriteCommand command = clo.command();
+ if (command instanceof MetaStorageWriteCommand) {
+ clusterTime.updateSafeTime(((MetaStorageWriteCommand) command).safeTime().asHybridTimestamp());
Review Comment:
I don't understand why this code.
##########
modules/network-api/src/main/java/org/apache/ignite/network/annotations/WithSetter.java:
##########
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.network.annotations;
+
+/**
+ * Annotation for the {@link Transferable} class methods. If a method is marked with this annotation,
+ * a setter for the field with the same name as method's will be generated.
+ * In order to have access to this setter via interface one can use default method:
+ * <pre>
+ * {@code @WithSetter
+ * HybridTimestampMessage safeTime();
+ *
+ * default void safeTime(HybridTimestampMessage safeTime) {
+ * // No-op.
+ * }
+ * }
+ * </pre>
+ * Note that fields with setters will not be final.
+ */
+public @interface WithSetter {
Review Comment:
Perhaps it would be more correct to add it in a separate ticket and test it more thoroughly.
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -234,4 +252,16 @@ private static RevisionCondition.Type toRevisionConditionType(ConditionType type
throw new IllegalArgumentException("Unexpected revision condition type " + type);
}
}
+
+ void beforeApply(Command command) {
Review Comment:
At the moment, the code is very strange for me and not clear.
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/impl/MetaStorageRaftGroupEventsListener.java:
##########
@@ -84,7 +89,20 @@ public void onLeaderElected(long term) {
registerTopologyEventListeners();
// Update learner configuration (in case we missed some topology updates) and initialize the serialization future.
- serializationFuture = executeIfLeaderImpl(this::resetLearners);
+ serializationFuture = executeWithStatus((service, term1, isLeader) -> {
+ CompletableFuture<Void> fut;
+ if (isLeader) {
+ fut = this.resetLearners(service, term1);
+
+ clusterTime.startLeaderTimer(service);
+ } else {
+ fut = CompletableFuture.completedFuture(null);
Review Comment:
```suggestion
fut = completedFuture(null);
```
##########
modules/metastorage/src/integrationTest/java/org/apache/ignite/internal/metastorage/impl/ItMetaStorageMultipleNodesTest.java:
##########
@@ -375,4 +382,40 @@ void testLearnerLeaveAndJoinBecauseOfNetworkPartition(TestInfo testInfo) throws
assertTrue(waitForCondition(() -> firstNode.getMetaStorageLearners().join().equals(Set.of(secondNode.name())), 10_000));
}
+
+ /**
+ * Tests that safe time is propagated from the leader to the follower/learner.
+ */
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testSafeTimePropagation(boolean useFollower, TestInfo testInfo) throws Exception {
Review Comment:
There is not enough test to change the leader.
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/MetaStorageWriteCommand.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.command;
+
+import org.apache.ignite.internal.raft.WriteCommand;
+import org.apache.ignite.network.annotations.WithSetter;
+
+/** Base meta storage write command. */
+public interface MetaStorageWriteCommand extends WriteCommand {
+ /**
+ * Returns time on the initiator node.
+ */
+ HybridTimestampMessage initiatorTime();
+
+ /**
+ * This is a dirty hack. This time is set by the leader node to disseminate new safe time across
+ * followers and learners.
+ */
+ @WithSetter
+ HybridTimestampMessage safeTime();
Review Comment:
While I don’t understand this dirty hack, we have a separate command for time synchronization.
We need a very detailed description of this hack.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1161704636
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/time/ClusterTimeImpl.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.server.time;
+
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.metastorage.impl.MetaStorageServiceImpl;
+import org.apache.ignite.internal.util.IgniteSpinBusyLock;
+import org.apache.ignite.internal.util.PendingComparableValuesTracker;
+import org.jetbrains.annotations.Nullable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Cluster time implementation with additional methods to adjust time and update safe time.
+ */
+public class ClusterTimeImpl implements ClusterTime {
+ private final IgniteSpinBusyLock busyLock;
+
+ private volatile @Nullable LeaderTimer leaderTimer;
+
+ private final HybridClock clock;
+
+ private final PendingComparableValuesTracker<HybridTimestamp> safeTime;
+
+ /**
+ * Constructor.
+ *
+ * @param busyLock Busy lock.
+ * @param clock Node's hybrid clock.
+ */
+ public ClusterTimeImpl(IgniteSpinBusyLock busyLock, HybridClock clock) {
+ this.busyLock = busyLock;
+ this.clock = clock;
+ this.safeTime = new PendingComparableValuesTracker<>(clock.now());
+ }
+
+ /**
+ * Starts sync time scheduler.
+ *
+ * @param service MetaStorage service that is used by scheduler to sync time.
+ */
+ public void startLeaderTimer(MetaStorageServiceImpl service) {
+ if (!busyLock.enterBusy()) {
+ return;
+ }
+
+ try {
+ assert leaderTimer == null;
+
+ LeaderTimer newTimer = new LeaderTimer(service);
+
+ leaderTimer = newTimer;
+
+ newTimer.start();
+ } finally {
+ busyLock.leaveBusy();
+ }
+ }
+
+ /**
+ * Stops sync time scheduler if it exists.
+ */
+ public void stopLeaderTimer() {
+ LeaderTimer timer = leaderTimer;
+
+ if (timer != null) {
+ timer.stop();
+
+ leaderTimer = null;
+ }
+ }
+
+ @Override
+ public HybridTimestamp now() {
+ return clock.now();
+ }
+
+ @Override
+ public CompletableFuture<Void> waitFor(HybridTimestamp time) {
+ return safeTime.waitFor(time);
+ }
+
+ /**
+ * Updates the internal safe time.
+ *
+ * @param newValue New safe time value.
+ */
+ public void updateSafeTime(HybridTimestamp newValue) {
Review Comment:
No, how do you mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1159373252
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/time/ClusterTimeImpl.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.server.time;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.impl.MetaStorageServiceImpl;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteSpinBusyLock;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.PendingComparableValuesTracker;
+import org.jetbrains.annotations.Nullable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Cluster time implementation with additional methods to adjust time and update safe time.
+ */
+public class ClusterTimeImpl implements ClusterTime {
+ /** The logger. */
+ private static final IgniteLogger LOG = Loggers.forClass(ClusterTimeImpl.class);
+
+ private final IgniteSpinBusyLock busyLock;
+
+ private volatile @Nullable LeaderTimer leaderTimer;
+
+ private final HybridClock clock;
+
+ private final PendingComparableValuesTracker<HybridTimestamp> safeTime;
+
+ /**
+ * Constructor.
+ *
+ * @param busyLock Busy lock.
+ * @param clock Node's hybrid clock.
+ */
+ public ClusterTimeImpl(IgniteSpinBusyLock busyLock, HybridClock clock) {
+ this.busyLock = busyLock;
+ this.clock = clock;
+ this.safeTime = new PendingComparableValuesTracker<>(clock.now());
+ }
+
+ /**
+ * Starts sync time scheduler.
+ *
+ * @param service MetaStorage service that is used by scheduler to sync time.
+ */
+ public void startLeaderTimer(MetaStorageServiceImpl service) {
+ if (!busyLock.enterBusy()) {
+ return;
+ }
+
+ try {
+ assert leaderTimer == null;
+
+ LeaderTimer newTimer = new LeaderTimer(service);
+
+ leaderTimer = newTimer;
+
+ newTimer.start();
+ } finally {
+ busyLock.leaveBusy();
+ }
+ }
+
+ /**
+ * Stops sync time scheduler.
+ */
+ public void stopLeaderTimer() {
+ LeaderTimer timer = leaderTimer;
+
+ assert timer != null;
+
+ timer.stop();
+
+ leaderTimer = null;
+ }
+
+ @Override
+ public HybridTimestamp now() {
+ return clock.now();
+ }
+
+ @Override
+ public CompletableFuture<Void> waitFor(HybridTimestamp time) {
+ return safeTime.waitFor(time);
+ }
+
+ public void updateSafeTime(HybridTimestamp ts) {
+ this.safeTime.update(ts);
+ }
+
+ public void adjust(HybridTimestamp ts) {
+ this.clock.update(ts);
+ }
+
+ private class LeaderTimer {
Review Comment:
Should it be separate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1159372193
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/impl/MetaStorageServiceImpl.java:
##########
@@ -258,6 +275,20 @@ public Publisher<Entry> prefix(ByteArray prefix, long revUpperBound) {
return new CursorPublisher(context, createPrefixCommand);
}
+ /**
+ * Sends safe time sync message. Should be called only on the leader node.
+ *
+ * @param safeTime New safe time.
+ * @return Future that will be completed when message is sent.
+ */
+ public CompletableFuture<Void> syncTime(HybridTimestamp safeTime) {
Review Comment:
See todo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] tkalkirill commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "tkalkirill (via GitHub)" <gi...@apache.org>.
tkalkirill commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1159979792
##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/rpc/impl/ActionRequestProcessor.java:
##########
@@ -19,8 +19,8 @@
import java.io.Serializable;
import java.nio.ByteBuffer;
import java.util.List;
-import java.util.concurrent.Executor;
-import org.apache.ignite.internal.logger.IgniteLogger;
+import java.util.concurrent.CompletableFuture;import java.util.concurrent.Executor;
+import java.util.function.BiFunction;import org.apache.ignite.internal.logger.IgniteLogger;
Review Comment:
```suggestion
import java.util.function.BiFunction;
import org.apache.ignite.internal.logger.IgniteLogger;
```
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/time/ClusterTimeImpl.java:
##########
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.server.time;
+
+import java.util.concurrent.CompletableFuture;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.metastorage.impl.MetaStorageServiceImpl;
+import org.apache.ignite.internal.util.IgniteSpinBusyLock;
+import org.apache.ignite.internal.util.PendingComparableValuesTracker;
+import org.jetbrains.annotations.Nullable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Cluster time implementation with additional methods to adjust time and update safe time.
+ */
+public class ClusterTimeImpl implements ClusterTime {
+ private final IgniteSpinBusyLock busyLock;
+
+ private volatile @Nullable LeaderTimer leaderTimer;
+
+ private final HybridClock clock;
+
+ private final PendingComparableValuesTracker<HybridTimestamp> safeTime;
+
+ /**
+ * Constructor.
+ *
+ * @param busyLock Busy lock.
+ * @param clock Node's hybrid clock.
+ */
+ public ClusterTimeImpl(IgniteSpinBusyLock busyLock, HybridClock clock) {
+ this.busyLock = busyLock;
+ this.clock = clock;
+ this.safeTime = new PendingComparableValuesTracker<>(clock.now());
+ }
+
+ /**
+ * Starts sync time scheduler.
+ *
+ * @param service MetaStorage service that is used by scheduler to sync time.
+ */
+ public void startLeaderTimer(MetaStorageServiceImpl service) {
+ if (!busyLock.enterBusy()) {
+ return;
+ }
+
+ try {
+ assert leaderTimer == null;
+
+ LeaderTimer newTimer = new LeaderTimer(service);
+
+ leaderTimer = newTimer;
+
+ newTimer.start();
+ } finally {
+ busyLock.leaveBusy();
+ }
+ }
+
+ /**
+ * Stops sync time scheduler if it exists.
+ */
+ public void stopLeaderTimer() {
+ LeaderTimer timer = leaderTimer;
+
+ if (timer != null) {
+ timer.stop();
+
+ leaderTimer = null;
+ }
+ }
+
+ @Override
+ public HybridTimestamp now() {
+ return clock.now();
+ }
+
+ @Override
+ public CompletableFuture<Void> waitFor(HybridTimestamp time) {
+ return safeTime.waitFor(time);
+ }
+
+ public void updateSafeTime(HybridTimestamp ts) {
Review Comment:
Please add in the description of the method why it is, how it differs from `adjust`.
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/time/ClusterTimeImpl.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.server.time;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.impl.MetaStorageServiceImpl;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteSpinBusyLock;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.PendingComparableValuesTracker;
+import org.jetbrains.annotations.Nullable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Cluster time implementation with additional methods to adjust time and update safe time.
+ */
+public class ClusterTimeImpl implements ClusterTime {
+ /** The logger. */
+ private static final IgniteLogger LOG = Loggers.forClass(ClusterTimeImpl.class);
+
+ private final IgniteSpinBusyLock busyLock;
+
+ private volatile @Nullable LeaderTimer leaderTimer;
+
+ private final HybridClock clock;
+
+ private final PendingComparableValuesTracker<HybridTimestamp> safeTime;
+
+ /**
+ * Constructor.
+ *
+ * @param busyLock Busy lock.
+ * @param clock Node's hybrid clock.
+ */
+ public ClusterTimeImpl(IgniteSpinBusyLock busyLock, HybridClock clock) {
+ this.busyLock = busyLock;
+ this.clock = clock;
+ this.safeTime = new PendingComparableValuesTracker<>(clock.now());
+ }
+
+ /**
+ * Starts sync time scheduler.
+ *
+ * @param service MetaStorage service that is used by scheduler to sync time.
+ */
+ public void startLeaderTimer(MetaStorageServiceImpl service) {
+ if (!busyLock.enterBusy()) {
+ return;
+ }
+
+ try {
+ assert leaderTimer == null;
+
+ LeaderTimer newTimer = new LeaderTimer(service);
+
+ leaderTimer = newTimer;
+
+ newTimer.start();
+ } finally {
+ busyLock.leaveBusy();
+ }
+ }
+
+ /**
+ * Stops sync time scheduler.
+ */
+ public void stopLeaderTimer() {
+ LeaderTimer timer = leaderTimer;
+
+ assert timer != null;
+
+ timer.stop();
+
+ leaderTimer = null;
+ }
+
+ @Override
+ public HybridTimestamp now() {
+ return clock.now();
+ }
+
+ @Override
+ public CompletableFuture<Void> waitFor(HybridTimestamp time) {
+ return safeTime.waitFor(time);
+ }
+
+ public void updateSafeTime(HybridTimestamp ts) {
+ this.safeTime.update(ts);
+ }
+
+ public void adjust(HybridTimestamp ts) {
Review Comment:
Please add in the description of the method why it is.
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/time/ClusterTimeImpl.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.server.time;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.impl.MetaStorageServiceImpl;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteSpinBusyLock;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.PendingComparableValuesTracker;
+import org.jetbrains.annotations.Nullable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Cluster time implementation with additional methods to adjust time and update safe time.
+ */
+public class ClusterTimeImpl implements ClusterTime {
+ /** The logger. */
+ private static final IgniteLogger LOG = Loggers.forClass(ClusterTimeImpl.class);
+
+ private final IgniteSpinBusyLock busyLock;
+
+ private volatile @Nullable LeaderTimer leaderTimer;
+
+ private final HybridClock clock;
+
+ private final PendingComparableValuesTracker<HybridTimestamp> safeTime;
+
+ /**
+ * Constructor.
+ *
+ * @param busyLock Busy lock.
+ * @param clock Node's hybrid clock.
+ */
+ public ClusterTimeImpl(IgniteSpinBusyLock busyLock, HybridClock clock) {
+ this.busyLock = busyLock;
+ this.clock = clock;
+ this.safeTime = new PendingComparableValuesTracker<>(clock.now());
+ }
+
+ /**
+ * Starts sync time scheduler.
+ *
+ * @param service MetaStorage service that is used by scheduler to sync time.
+ */
+ public void startLeaderTimer(MetaStorageServiceImpl service) {
+ if (!busyLock.enterBusy()) {
+ return;
+ }
+
+ try {
+ assert leaderTimer == null;
+
+ LeaderTimer newTimer = new LeaderTimer(service);
+
+ leaderTimer = newTimer;
+
+ newTimer.start();
+ } finally {
+ busyLock.leaveBusy();
+ }
+ }
+
+ /**
+ * Stops sync time scheduler.
+ */
+ public void stopLeaderTimer() {
+ LeaderTimer timer = leaderTimer;
+
+ assert timer != null;
+
+ timer.stop();
+
+ leaderTimer = null;
+ }
+
+ @Override
+ public HybridTimestamp now() {
+ return clock.now();
+ }
+
+ @Override
+ public CompletableFuture<Void> waitFor(HybridTimestamp time) {
+ return safeTime.waitFor(time);
+ }
+
+ public void updateSafeTime(HybridTimestamp ts) {
+ this.safeTime.update(ts);
+ }
+
+ public void adjust(HybridTimestamp ts) {
+ this.clock.update(ts);
+ }
+
+ private class LeaderTimer {
Review Comment:
It would be nice, I think.
##########
modules/raft-api/src/main/java/org/apache/ignite/internal/raft/service/RaftGroupListener.java:
##########
@@ -81,4 +82,13 @@ default void onConfigurationCommitted(CommittedConfiguration config) {
* Invoked once after a raft node has been shut down.
*/
void onShutdown();
+
+ /**
+ * Invoked before submitting a command to a raft group.
Review Comment:
It seems that you reminded what is performed before writing to the raft log or not?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1159371299
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/MetaStorageWriteCommand.java:
##########
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.command;
+
+import org.apache.ignite.internal.raft.WriteCommand;
+import org.apache.ignite.network.annotations.WithSetter;
+
+/** Base meta storage write command. */
+public interface MetaStorageWriteCommand extends WriteCommand {
+ /**
+ * Returns time on the initiator node.
+ */
+ HybridTimestampMessage initiatorTime();
+
+ /**
+ * This is a dirty hack. This time is set by the leader node to disseminate new safe time across
+ * followers and learners.
+ */
+ @WithSetter
+ HybridTimestampMessage safeTime();
Review Comment:
The hack is setter, not the field itself. We do need to synchronise time both with every write command and with idle sync command
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1159369537
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/impl/MetaStorageRaftGroupEventsListener.java:
##########
@@ -156,11 +178,16 @@ private void executeIfLeader(OnLeaderAction action) {
}
private CompletableFuture<Void> executeIfLeaderImpl(OnLeaderAction action) {
+ return executeWithStatus((service, term, isLeader) -> action.apply(service, term));
Review Comment:
What do you mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1159437503
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/impl/MetaStorageRaftGroupEventsListener.java:
##########
@@ -128,6 +146,10 @@ private interface OnLeaderAction {
CompletableFuture<Void> apply(MetaStorageServiceImpl service, long term);
}
+ private interface OnStatusAction {
Review Comment:
Leader action is useful. And anyway, this will all change with replica based ms
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1159372882
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/time/ClusterTimeImpl.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.server.time;
+
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.Executors;
+import java.util.concurrent.RejectedExecutionException;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.internal.logger.IgniteLogger;
+import org.apache.ignite.internal.logger.Loggers;
+import org.apache.ignite.internal.metastorage.impl.MetaStorageServiceImpl;
+import org.apache.ignite.internal.thread.NamedThreadFactory;
+import org.apache.ignite.internal.util.IgniteSpinBusyLock;
+import org.apache.ignite.internal.util.IgniteUtils;
+import org.apache.ignite.internal.util.PendingComparableValuesTracker;
+import org.jetbrains.annotations.Nullable;
+import org.jetbrains.annotations.TestOnly;
+
+/**
+ * Cluster time implementation with additional methods to adjust time and update safe time.
+ */
+public class ClusterTimeImpl implements ClusterTime {
+ /** The logger. */
+ private static final IgniteLogger LOG = Loggers.forClass(ClusterTimeImpl.class);
+
+ private final IgniteSpinBusyLock busyLock;
+
+ private volatile @Nullable LeaderTimer leaderTimer;
+
+ private final HybridClock clock;
+
+ private final PendingComparableValuesTracker<HybridTimestamp> safeTime;
+
+ /**
+ * Constructor.
+ *
+ * @param busyLock Busy lock.
+ * @param clock Node's hybrid clock.
+ */
+ public ClusterTimeImpl(IgniteSpinBusyLock busyLock, HybridClock clock) {
+ this.busyLock = busyLock;
+ this.clock = clock;
+ this.safeTime = new PendingComparableValuesTracker<>(clock.now());
+ }
+
+ /**
+ * Starts sync time scheduler.
+ *
+ * @param service MetaStorage service that is used by scheduler to sync time.
+ */
+ public void startLeaderTimer(MetaStorageServiceImpl service) {
+ if (!busyLock.enterBusy()) {
+ return;
+ }
+
+ try {
+ assert leaderTimer == null;
+
+ LeaderTimer newTimer = new LeaderTimer(service);
+
+ leaderTimer = newTimer;
+
+ newTimer.start();
+ } finally {
+ busyLock.leaveBusy();
+ }
+ }
+
+ /**
+ * Stops sync time scheduler.
+ */
+ public void stopLeaderTimer() {
+ LeaderTimer timer = leaderTimer;
+
+ assert timer != null;
+
+ timer.stop();
+
+ leaderTimer = null;
+ }
+
+ @Override
+ public HybridTimestamp now() {
+ return clock.now();
+ }
+
+ @Override
+ public CompletableFuture<Void> waitFor(HybridTimestamp time) {
+ return safeTime.waitFor(time);
+ }
+
+ public void updateSafeTime(HybridTimestamp ts) {
+ this.safeTime.update(ts);
+ }
+
+ public void adjust(HybridTimestamp ts) {
Review Comment:
Why? We call adjusting the clock "adjust"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1160763781
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneManagerLogicalTopologyEventsTest.java:
##########
@@ -151,15 +155,22 @@ private DistributionZoneManager prepareDistributionZoneManager() {
keyValueStorage = spy(new SimpleInMemoryKeyValueStorage("test"));
- MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage);
+ MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage, mock(ClusterTimeImpl.class));
RaftGroupService metaStorageService = mock(RaftGroupService.class);
+ HybridTimestampMessage mockTsMessage = mock(HybridTimestampMessage.class);
+ when(mockTsMessage.asHybridTimestamp()).thenReturn(new HybridTimestamp(10, 10));
+
// Delegate directly to listener.
lenient().doAnswer(
invocationClose -> {
Review Comment:
It is copy-pasted, yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] ibessonov commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "ibessonov (via GitHub)" <gi...@apache.org>.
ibessonov commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1159645506
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneManagerLogicalTopologyEventsTest.java:
##########
@@ -151,15 +155,22 @@ private DistributionZoneManager prepareDistributionZoneManager() {
keyValueStorage = spy(new SimpleInMemoryKeyValueStorage("test"));
- MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage);
+ MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage, mock(ClusterTimeImpl.class));
RaftGroupService metaStorageService = mock(RaftGroupService.class);
+ HybridTimestampMessage mockTsMessage = mock(HybridTimestampMessage.class);
Review Comment:
Can't you just build it?
Man, I wish massage factories were static, it's so annoying
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneManagerLogicalTopologyEventsTest.java:
##########
@@ -151,15 +155,22 @@ private DistributionZoneManager prepareDistributionZoneManager() {
keyValueStorage = spy(new SimpleInMemoryKeyValueStorage("test"));
- MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage);
+ MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage, mock(ClusterTimeImpl.class));
RaftGroupService metaStorageService = mock(RaftGroupService.class);
+ HybridTimestampMessage mockTsMessage = mock(HybridTimestampMessage.class);
+ when(mockTsMessage.asHybridTimestamp()).thenReturn(new HybridTimestamp(10, 10));
+
// Delegate directly to listener.
lenient().doAnswer(
invocationClose -> {
Review Comment:
I wonder, what's the "close" in this `invocationClose`. Closure?
Anyway, I have a strong feeling of deja vu, almost as if this code is copy-pasted multiple times. Can you please check that?
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneManagerScaleUpTest.java:
##########
@@ -198,7 +199,7 @@ public void setUp() {
keyValueStorage = spy(new SimpleInMemoryKeyValueStorage("test"));
- MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage);
+ MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage, mock(ClusterTimeImpl.class));
Review Comment:
Will passing a real instance make a difference?
##########
modules/metastorage/src/integrationTest/java/org/apache/ignite/internal/metastorage/impl/ItMetaStorageMultipleNodesTest.java:
##########
@@ -118,11 +123,12 @@ private static class Node {
Path basePath = dataPath.resolve(name());
+ HybridClockImpl clock = new HybridClockImpl();
Review Comment:
Same here
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/MetaStorageWriteCommand.java:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.command;
+
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.raft.Command;
+import org.apache.ignite.internal.raft.WriteCommand;
+import org.apache.ignite.network.annotations.WithSetter;
+
+/** Base meta storage write command. */
+public interface MetaStorageWriteCommand extends WriteCommand {
+ /**
+ * Returns time on the initiator node.
+ */
+ HybridTimestampMessage initiatorTime();
+
+ /**
+ * This is a dirty hack. This time is set by the leader node to disseminate new safe time across
+ * followers and learners. Leader of the ms group reads {@link #initiatorTime()}, adjusts its clock
+ * and sets safeTime as {@link HybridClock#now()} as safeTime here. This must be done before
+ * command is saved into the Raft log (see {@link org.apache.ignite.internal.raft.service.RaftGroupListener#onBeforeApply(Command)}.
+ */
+ @WithSetter
+ HybridTimestampMessage safeTime();
+
+ /**
+ * Setter for the safeTime field.
+ */
+ default void safeTime(HybridTimestampMessage safeTime) {
Review Comment:
What's the reason to have a default implementation here?
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/MetastorageCommandsMessageGroup.java:
##########
@@ -84,4 +84,10 @@ public interface MetastorageCommandsMessageGroup {
/** Message type for {@link CloseAllCursorsCommand}. */
short CLOSE_ALL_CURSORS = 64;
+
+ /** Message type for {@link HybridTimestampMessage}. */
+ short HYBRID_TS = 65;
Review Comment:
Maybe it would make sense to move these ids into a `7x` numeration.
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -66,6 +76,11 @@ class MetaStorageWriteHandler {
boolean handleWriteCommand(CommandClosure<WriteCommand> clo) {
WriteCommand command = clo.command();
+ if (command instanceof MetaStorageWriteCommand) {
Review Comment:
No-no-no, please update timestamp _after_ you finished processing :)
Otherwise you'll have data races
##########
modules/metastorage/src/integrationTest/java/org/apache/ignite/internal/metastorage/impl/ItMetaStorageManagerImplTest.java:
##########
@@ -91,7 +91,8 @@ void setUp(TestInfo testInfo, @InjectConfiguration RaftConfiguration raftConfigu
clusterService = clusterService(testInfo, addr.port(), new StaticNodeFinder(List.of(addr)));
- raftManager = new Loza(clusterService, raftConfiguration, workDir.resolve("loza"), new HybridClockImpl());
+ HybridClockImpl clock = new HybridClockImpl();
Review Comment:
Please keep types clean. Don't commit this suggestion blindly, there's probably no import for this class.
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneManagerLogicalTopologyEventsTest.java:
##########
@@ -151,15 +155,22 @@ private DistributionZoneManager prepareDistributionZoneManager() {
keyValueStorage = spy(new SimpleInMemoryKeyValueStorage("test"));
- MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage);
+ MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage, mock(ClusterTimeImpl.class));
RaftGroupService metaStorageService = mock(RaftGroupService.class);
+ HybridTimestampMessage mockTsMessage = mock(HybridTimestampMessage.class);
+ when(mockTsMessage.asHybridTimestamp()).thenReturn(new HybridTimestamp(10, 10));
+
// Delegate directly to listener.
lenient().doAnswer(
invocationClose -> {
Command cmd = invocationClose.getArgument(0);
+ if (cmd instanceof MetaStorageWriteCommand) {
Review Comment:
I'm sure you had a reason to do it this way. What happens when someone forgets to make this propagation in test? NPE?
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/MetaStorageWriteCommand.java:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.command;
+
+import org.apache.ignite.internal.hlc.HybridClock;
+import org.apache.ignite.internal.raft.Command;
+import org.apache.ignite.internal.raft.WriteCommand;
+import org.apache.ignite.network.annotations.WithSetter;
+
+/** Base meta storage write command. */
+public interface MetaStorageWriteCommand extends WriteCommand {
+ /**
+ * Returns time on the initiator node.
+ */
+ HybridTimestampMessage initiatorTime();
Review Comment:
We don't need to have this property in the future, because it's the attribute of action request, right? I would prefer you to address this in the comment. Maybe even deprecating it straight away
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/SyncTimeCommand.java:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.command;
+
+import org.apache.ignite.internal.raft.WriteCommand;
+import org.apache.ignite.network.annotations.Transferable;
+
+/**
+ * Command that initiates idle safe time synchronization.
+ */
+@Transferable(MetastorageCommandsMessageGroup.SYNC_TIME)
+public interface SyncTimeCommand extends WriteCommand {
+ /** New safe time. */
+ HybridTimestampMessage safeTime();
Review Comment:
Does not-inheriting it from metastorage write message make code simpler?
##########
modules/metastorage/src/integrationTest/java/org/apache/ignite/internal/metastorage/impl/ItMetaStorageManagerImplTest.java:
##########
@@ -91,7 +91,8 @@ void setUp(TestInfo testInfo, @InjectConfiguration RaftConfiguration raftConfigu
clusterService = clusterService(testInfo, addr.port(), new StaticNodeFinder(List.of(addr)));
- raftManager = new Loza(clusterService, raftConfiguration, workDir.resolve("loza"), new HybridClockImpl());
+ HybridClockImpl clock = new HybridClockImpl();
Review Comment:
```suggestion
HybridClock clock = new HybridClockImpl();
```
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/command/HybridTimestampMessage.java:
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.metastorage.command;
+
+import java.io.Serializable;
+import org.apache.ignite.internal.hlc.HybridTimestamp;
+import org.apache.ignite.network.NetworkMessage;
+import org.apache.ignite.network.annotations.Transferable;
+
+/** Message with a {@link HybridTimestamp}. */
+@Transferable(MetastorageCommandsMessageGroup.HYBRID_TS)
+public interface HybridTimestampMessage extends NetworkMessage, Serializable {
Review Comment:
Is this a copy-paste of `org.apache.ignite.internal.replicator.command.HybridTimestampMessage`?
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -234,4 +253,18 @@ private static RevisionCondition.Type toRevisionConditionType(ConditionType type
throw new IllegalArgumentException("Unexpected revision condition type " + type);
}
}
+
+ void beforeApply(Command command) {
Review Comment:
I'm trying to wrap my head around this method.
First of all, it is executed on each node in the cluster, correct? If it is, then it feels very dangerous, because it may result in different command times at different nodes.
Please explain it thoroughly, I feel like there's a bug here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1160750846
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneManagerLogicalTopologyEventsTest.java:
##########
@@ -151,15 +155,22 @@ private DistributionZoneManager prepareDistributionZoneManager() {
keyValueStorage = spy(new SimpleInMemoryKeyValueStorage("test"));
- MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage);
+ MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage, mock(ClusterTimeImpl.class));
RaftGroupService metaStorageService = mock(RaftGroupService.class);
+ HybridTimestampMessage mockTsMessage = mock(HybridTimestampMessage.class);
+ when(mockTsMessage.asHybridTimestamp()).thenReturn(new HybridTimestamp(10, 10));
+
// Delegate directly to listener.
lenient().doAnswer(
invocationClose -> {
Command cmd = invocationClose.getArgument(0);
+ if (cmd instanceof MetaStorageWriteCommand) {
Review Comment:
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1160751025
##########
modules/distribution-zones/src/test/java/org/apache/ignite/internal/distributionzones/DistributionZoneManagerLogicalTopologyEventsTest.java:
##########
@@ -151,15 +155,22 @@ private DistributionZoneManager prepareDistributionZoneManager() {
keyValueStorage = spy(new SimpleInMemoryKeyValueStorage("test"));
- MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage);
+ MetaStorageListener metaStorageListener = new MetaStorageListener(keyValueStorage, mock(ClusterTimeImpl.class));
RaftGroupService metaStorageService = mock(RaftGroupService.class);
+ HybridTimestampMessage mockTsMessage = mock(HybridTimestampMessage.class);
Review Comment:
I can, but there's no need
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite-3] SammyVimes commented on a diff in pull request #1869: IGNITE-19028 Implement safe-time propagation for meta-storage raft-group
Posted by "SammyVimes (via GitHub)" <gi...@apache.org>.
SammyVimes commented on code in PR #1869:
URL: https://github.com/apache/ignite-3/pull/1869#discussion_r1161702743
##########
modules/metastorage/src/main/java/org/apache/ignite/internal/metastorage/server/raft/MetaStorageWriteHandler.java:
##########
@@ -234,4 +253,18 @@ private static RevisionCondition.Type toRevisionConditionType(ConditionType type
throw new IllegalArgumentException("Unexpected revision condition type " + type);
}
}
+
+ void beforeApply(Command command) {
Review Comment:
Now that's a good question
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org