You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "EgorKuts (via GitHub)" <gi...@apache.org> on 2025/10/24 05:16:16 UTC

[PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

EgorKuts opened a new pull request, #6843:
URL: https://github.com/apache/ignite-3/pull/6843

   https://issues.apache.org/jira/browse/IGNITE-26638


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2480903517


##########
modules/runner/src/testFixtures/java/org/apache/ignite/internal/ClusterPerClassIntegrationTest.java:
##########
@@ -184,6 +194,44 @@ protected String getNodeBootstrapConfigTemplate() {
         return NODE_BOOTSTRAP_CFG_TEMPLATE;
     }
 
+    /**
+     * Waits for all partitions in the specified zone to reach the HEALTHY state across all cluster nodes.
+     *
+     * @param zone The name of the distribution zone to check.
+     * @param tableName The name of the table (currently unused by the implementation, reserved for future use).
+     * @param partitionsCount The number of partitions to check (currently unused by the implementation, reserved for future use).
+     * @throws InterruptedException If the thread is interrupted while waiting.
+     * @throws AssertionError If partitions do not become healthy within the timeout period.
+     */
+    protected void awaitPartitionToBeHealthy(String zone, String tableName, int partitionsCount) throws InterruptedException {
+        assertTrue(waitForCondition(() -> CLUSTER.runningNodes().count() == initialNodes(), 10_000));
+        IgniteImpl node = unwrapIgniteImpl(CLUSTER.aliveNode());
+
+        assertTrue(waitForCondition(() -> {
+                    CompletableFuture<Map<ZonePartitionId, LocalPartitionStateByNode>> localStateTableFuture =
+                            node.disasterRecoveryManager().localPartitionStates(Set.of(zone), emptySet(), emptySet());

Review Comment:
   Why do we check localPartitionStates and not global ones?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2480906606


##########
modules/runner/src/testFixtures/java/org/apache/ignite/internal/ClusterPerClassIntegrationTest.java:
##########
@@ -184,6 +194,44 @@ protected String getNodeBootstrapConfigTemplate() {
         return NODE_BOOTSTRAP_CFG_TEMPLATE;
     }
 
+    /**
+     * Waits for all partitions in the specified zone to reach the HEALTHY state across all cluster nodes.
+     *
+     * @param zone The name of the distribution zone to check.
+     * @param tableName The name of the table (currently unused by the implementation, reserved for future use).
+     * @param partitionsCount The number of partitions to check (currently unused by the implementation, reserved for future use).
+     * @throws InterruptedException If the thread is interrupted while waiting.
+     * @throws AssertionError If partitions do not become healthy within the timeout period.
+     */
+    protected void awaitPartitionToBeHealthy(String zone, String tableName, int partitionsCount) throws InterruptedException {

Review Comment:
   - awaitPartition**s**ToBeHealthy
   -`partitionsCount` naming is misleading, it's rather `Set<Integer> partitionIds`



##########
modules/runner/src/testFixtures/java/org/apache/ignite/internal/ClusterPerClassIntegrationTest.java:
##########
@@ -184,6 +194,44 @@ protected String getNodeBootstrapConfigTemplate() {
         return NODE_BOOTSTRAP_CFG_TEMPLATE;
     }
 
+    /**
+     * Waits for all partitions in the specified zone to reach the HEALTHY state across all cluster nodes.
+     *
+     * @param zone The name of the distribution zone to check.
+     * @param tableName The name of the table (currently unused by the implementation, reserved for future use).
+     * @param partitionsCount The number of partitions to check (currently unused by the implementation, reserved for future use).
+     * @throws InterruptedException If the thread is interrupted while waiting.
+     * @throws AssertionError If partitions do not become healthy within the timeout period.
+     */
+    protected void awaitPartitionToBeHealthy(String zone, String tableName, int partitionsCount) throws InterruptedException {

Review Comment:
   - awaitPartition**s**ToBeHealthy
   - `partitionsCount` naming is misleading, it's rather `Set<Integer> partitionIds`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "EgorKuts (via GitHub)" <gi...@apache.org>.
EgorKuts commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2485426517


##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   there's a return statement, so fooBar() is called at most once



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "EgorKuts (via GitHub)" <gi...@apache.org>.
EgorKuts commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2478666502


##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   but i can tell that everything in partition/../meta is being destroyed before "fresh" node rejoins the group



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "EgorKuts (via GitHub)" <gi...@apache.org>.
EgorKuts commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2480257001


##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   1. There are 3 nodes in the group. 3rd becomes a leader, term=2. 
   2. 3rd creates Replicators for each follower in becomeLeader()
   3. 1st node deletes its state and joins the group.
   4.  The leader sees no configuration difference here: newConf.diff(oldConf, adding, removing); No new Replicator is created, because 1st rejoins the group with same consistentId(see [equals](https://github.com/EgorKuts/ignite-3/blob/52ba3bfe8c21e4785474477a0ff3a40b4a3247f4/modules/raft/src/main/java/org/apache/ignite/raft/jraft/entity/PeerId.java#L245)) 
   5. The old Replicator on the 3d node keeps thinking that node 1 has prevLogIndex=52 from step 1. Though the actual state of node1 is prevLogIndex=0 term=0. The leader keeps sending heartbeats to 1 with prevLogIndex=52. They don't trigger probeCheck. Everything is stuck until new election or real AppendEntry from the leader.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2486555117


##########
modules/runner/src/testFixtures/java/org/apache/ignite/internal/ClusterPerClassIntegrationTest.java:
##########
@@ -209,6 +218,43 @@ protected static void dropAllTables() {
         }
     }
 
+    /**
+     * Waits for the specified partitionIds in the specified zone to reach the HEALTHY state across all cluster nodes.
+     *
+     * @param zone The name of the distribution zone to check.
+     * @param  partitionIds The specified set of partitions.
+     * @throws InterruptedException If the thread is interrupted while waiting.
+     * @throws AssertionError If partitionIds do not become healthy within the timeout period.
+     */
+    protected static void awaitPartitionsToBeHealthy(
+            String zone,
+            Set<Integer> partitionIds
+    ) throws InterruptedException {
+        IgniteImpl node = unwrapIgniteImpl(CLUSTER.aliveNode());
+
+        assertTrue(waitForCondition(() -> {
+                    CompletableFuture<Map<ZonePartitionId, GlobalPartitionState>> globalTablePartitionStates =

Review Comment:
   Could you please also rename `globalTablePartitionStates`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "EgorKuts (via GitHub)" <gi...@apache.org>.
EgorKuts commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2480257001


##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   1. There are 3 nodes in the cluster. 3rd becomes a leader, term=2. 
   2. 3rd creates Replicators for each follower in becomeLeader()
   3. 1st node deletes its state and joins the group.
   4.  The leader sees no configuration difference here: newConf.diff(oldConf, adding, removing); No new Replicator is created, because 1st rejoins the group with same consistentId(see [equals](https://github.com/EgorKuts/ignite-3/blob/52ba3bfe8c21e4785474477a0ff3a40b4a3247f4/modules/raft/src/main/java/org/apache/ignite/raft/jraft/entity/PeerId.java#L245)) 
   5. The old Replicator on the 3d node keeps thinking that node 1 has prevLogIndex=52 from step 1. Though actual of node1 is prevLogIndex=0 term=0. The leader keeps sending heartbeats to 1 with prevLogIndex=52. They don't trigger probeCheck. Everything is stuck until new election or real AppendEntry from the leader.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "EgorKuts (via GitHub)" <gi...@apache.org>.
EgorKuts commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2480257001


##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   1. There are 3 nodes in the group. 3rd becomes a leader, term=2. 
   2. 3rd creates Replicators for each follower in becomeLeader()
   3. 1st node deletes its state and joins the group.
   4.  The leader sees no configuration difference here: newConf.diff(oldConf, adding, removing); No new Replicator is created, because 1st rejoins the group with same consistentId(see [equals](https://github.com/EgorKuts/ignite-3/blob/52ba3bfe8c21e4785474477a0ff3a40b4a3247f4/modules/raft/src/main/java/org/apache/ignite/raft/jraft/entity/PeerId.java#L245)) 
   5. The old Replicator on the 3d node keeps thinking that node 1 has prevLogIndex=52 from step 1. Though actual of node1 is prevLogIndex=0 term=0. The leader keeps sending heartbeats to 1 with prevLogIndex=52. They don't trigger probeCheck. Everything is stuck until new election or real AppendEntry from the leader.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2480702053


##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {
+                 long term = this.options.getLogManager().getTerm(response.lastLogIndex());
+                 if (term < r.options.getTerm()) {
+                      LOG.info("Heartbeat to nodeId {}, groupId {} failure with outdated term, try to send a probe request.", r.options.getNode().getNodeId(), r.options.getGroupId());
+                      r.sendProbeRequest();

Review Comment:
   Should we also call `doUnlock = false;` like we do in `if (!response.success() && response.lastLogIndex() != 0) {`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "EgorKuts (via GitHub)" <gi...@apache.org>.
EgorKuts commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2478666502


##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   but i can tell that everything in partition/../meta is being destroyed before "fresh" node rejoins the cluster



##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   but i can tell that everything in partition/../meta is being destroyed before "fresh" node rejoins the group



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "alievmirza (via GitHub)" <gi...@apache.org>.
alievmirza commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2478076758


##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   Why didn't this problem appear before when new raft node joined? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2486151990


##########
modules/runner/src/testFixtures/java/org/apache/ignite/internal/ClusterPerClassIntegrationTest.java:
##########
@@ -184,6 +194,44 @@ protected String getNodeBootstrapConfigTemplate() {
         return NODE_BOOTSTRAP_CFG_TEMPLATE;
     }
 
+    /**
+     * Waits for all partitions in the specified zone to reach the HEALTHY state across all cluster nodes.
+     *
+     * @param zone The name of the distribution zone to check.
+     * @param tableName The name of the table (currently unused by the implementation, reserved for future use).
+     * @param partitionsCount The number of partitions to check (currently unused by the implementation, reserved for future use).
+     * @throws InterruptedException If the thread is interrupted while waiting.
+     * @throws AssertionError If partitions do not become healthy within the timeout period.
+     */
+    protected void awaitPartitionToBeHealthy(String zone, String tableName, int partitionsCount) throws InterruptedException {
+        assertTrue(waitForCondition(() -> CLUSTER.runningNodes().count() == initialNodes(), 10_000));

Review Comment:
   I don't think so.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "EgorKuts (via GitHub)" <gi...@apache.org>.
EgorKuts commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2477964438


##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   because the follower/learner just joined the cluster



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2480899788


##########
modules/runner/src/testFixtures/java/org/apache/ignite/internal/ClusterPerClassIntegrationTest.java:
##########
@@ -184,6 +194,44 @@ protected String getNodeBootstrapConfigTemplate() {
         return NODE_BOOTSTRAP_CFG_TEMPLATE;
     }
 
+    /**
+     * Waits for all partitions in the specified zone to reach the HEALTHY state across all cluster nodes.
+     *
+     * @param zone The name of the distribution zone to check.
+     * @param tableName The name of the table (currently unused by the implementation, reserved for future use).
+     * @param partitionsCount The number of partitions to check (currently unused by the implementation, reserved for future use).
+     * @throws InterruptedException If the thread is interrupted while waiting.
+     * @throws AssertionError If partitions do not become healthy within the timeout period.
+     */
+    protected void awaitPartitionToBeHealthy(String zone, String tableName, int partitionsCount) throws InterruptedException {
+        assertTrue(waitForCondition(() -> CLUSTER.runningNodes().count() == initialNodes(), 10_000));

Review Comment:
   It's not generic. I mean that `awaitPartitionToBeHealthy()` might be called at any time during the life of the cluster. In that case `CLUSTER.runningNodes().count() == initialNodes()` might not match because:
   
   1. We may add more nodes in addition to initialNodes().
   2. We may remove nodes, without touching partition ones. I mean that there are 3 nodes in cluster [A,B,C], partition assignments is [A,B] though. In that case we do not really care about node C, right?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc merged PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "alievmirza (via GitHub)" <gi...@apache.org>.
alievmirza commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2477912070


##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   Why in this case response.lastLogIndex() == 0? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "EgorKuts (via GitHub)" <gi...@apache.org>.
EgorKuts commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2478065459


##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   and we observe the following in the logs:
   Node <21_part_0/idrcrpwct_n_3344> reject term_unmatched AppendEntriesRequest from idrcrpwct_n_3345, term=2, prevLogIndex=58, prevLogTerm=2, localPrevLogTerm=0, lastLogIndex=0, entriesSize=0.



##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   and we observe the following in the logs:
   Node <21_part_0/idrcrpwct_n_3344> reject term_unmatched AppendEntriesRequest from idrcrpwct_n_3345, term=2, prevLogIndex=58, prevLogTerm=2, localPrevLogTerm=0, lastLogIndex=0, entriesSize=0.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "EgorKuts (via GitHub)" <gi...@apache.org>.
EgorKuts commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2478662707


##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   I can't tell if this problem appeared before. I don't see such a testcase



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2480700277


##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   if (!response.success() && response.lastLogIndex() != 0) {
     fooBar();
   }
   if (!response.success()) {
     fooBar();
   }
   
   We will call fooBar() twice if (!response.success() && response.lastLogIndex() != 0) which is not intended. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2486217562


##########
modules/runner/src/testFixtures/java/org/apache/ignite/internal/ClusterPerClassIntegrationTest.java:
##########
@@ -209,6 +218,45 @@ protected static void dropAllTables() {
         }
     }
 
+    /**
+     * Waits for the specified partitionIds in the specified zone to reach the HEALTHY state across all cluster nodes.
+     *
+     * @param zone The name of the distribution zone to check.
+     * @param tableName The name of the table (currently unused by the implementation, reserved for future use).
+     * @param  partitionIds The specified set of partitions.
+     * @throws InterruptedException If the thread is interrupted while waiting.
+     * @throws AssertionError If partitionIds do not become healthy within the timeout period.
+     */
+    protected static void awaitPartitionsToBeHealthy(
+            String zone,
+            String tableName,
+            Set<Integer> partitionIds) throws InterruptedException {
+        assertTrue(waitForCondition(() -> CLUSTER.runningNodes().count() == CLUSTER.nodes().size(), 10_000));
+        IgniteImpl node = unwrapIgniteImpl(CLUSTER.aliveNode());
+
+        assertTrue(waitForCondition(() -> {
+                    CompletableFuture<Map<TablePartitionId, GlobalTablePartitionState>> globalTablePartitionStates =
+                            node.disasterRecoveryManager().globalTablePartitionStates(Set.of(zone), partitionIds);

Review Comment:
   BTW, I believe that it's better to use globalPartitionStates() (which is actually globalZonePartitionStates()) instead of global**Table**PartitionStates.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "EgorKuts (via GitHub)" <gi...@apache.org>.
EgorKuts commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2485427765


##########
modules/runner/src/testFixtures/java/org/apache/ignite/internal/ClusterPerClassIntegrationTest.java:
##########
@@ -184,6 +194,44 @@ protected String getNodeBootstrapConfigTemplate() {
         return NODE_BOOTSTRAP_CFG_TEMPLATE;
     }
 
+    /**
+     * Waits for all partitions in the specified zone to reach the HEALTHY state across all cluster nodes.
+     *
+     * @param zone The name of the distribution zone to check.
+     * @param tableName The name of the table (currently unused by the implementation, reserved for future use).
+     * @param partitionsCount The number of partitions to check (currently unused by the implementation, reserved for future use).
+     * @throws InterruptedException If the thread is interrupted while waiting.
+     * @throws AssertionError If partitions do not become healthy within the timeout period.
+     */
+    protected void awaitPartitionToBeHealthy(String zone, String tableName, int partitionsCount) throws InterruptedException {
+        assertTrue(waitForCondition(() -> CLUSTER.runningNodes().count() == initialNodes(), 10_000));

Review Comment:
   fixed it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "sanpwc (via GitHub)" <gi...@apache.org>.
sanpwc commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2486148696


##########
modules/runner/src/testFixtures/java/org/apache/ignite/internal/ClusterPerClassIntegrationTest.java:
##########
@@ -209,6 +218,45 @@ protected static void dropAllTables() {
         }
     }
 
+    /**
+     * Waits for the specified partitionIds in the specified zone to reach the HEALTHY state across all cluster nodes.
+     *
+     * @param zone The name of the distribution zone to check.
+     * @param tableName The name of the table (currently unused by the implementation, reserved for future use).
+     * @param  partitionIds The specified set of partitions.
+     * @throws InterruptedException If the thread is interrupted while waiting.
+     * @throws AssertionError If partitionIds do not become healthy within the timeout period.
+     */
+    protected static void awaitPartitionsToBeHealthy(
+            String zone,
+            String tableName,
+            Set<Integer> partitionIds) throws InterruptedException {

Review Comment:
   It's better to but ` ) throws InterruptedException {` on a new line.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "EgorKuts (via GitHub)" <gi...@apache.org>.
EgorKuts commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2477964438


##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   because the follower/learner just rejoined the group



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] IGNITE-26638 wait for partitions to become healthy [ignite-3]

Posted by "alievmirza (via GitHub)" <gi...@apache.org>.
alievmirza commented on code in PR #6843:
URL: https://github.com/apache/ignite-3/pull/6843#discussion_r2480856367


##########
modules/raft/src/main/java/org/apache/ignite/raft/jraft/core/Replicator.java:
##########
@@ -1239,6 +1239,15 @@ void onHeartbeatReturned(final ThreadId id, final Status status, final AppendEnt
                 r.startHeartbeatTimer(startTimeMs);
                 return;
             }
+            if (!response.success()) {

Review Comment:
   thanks! 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org