You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ignite.apache.org by "Mikhail Petrov (Jira)" <ji...@apache.org> on 2026/05/02 06:16:00 UTC
[jira] [Updated] (IGNITE-28381) Node failure with Assertion Error during Atomic Cache operation topology remapping
[ https://issues.apache.org/jira/browse/IGNITE-28381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Petrov updated IGNITE-28381:
------------------------------------
Description:
The problem was reproduced during org.apache.ignite.client.ReliabilityTest#testFailover test on TC. See attachments for the full log.
Exception:
{code:java}
java.lang.AssertionError
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.onPrimaryResponse(GridNearAtomicUpdateFuture.java:400)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateResponse(GridDhtAtomicCache.java:3236)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:311)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:306)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1084)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:594)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:396)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:306)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1862)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1483)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1387)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.thread.pool.IgniteStripedExecutor$Stripe.body(IgniteStripedExecutor.java:638)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
[04:31:10] [org.apache.ignite:ignite-indexing] at java.base/java.lang.Thread.run(Thread.java:829)
{code}
The root cause of the assertion error:
Consider a cluster consisting of 3 nodes - node0, node1, node2
1. node2 is gracefully stopped
2. PME is started on node0 and node 1
3. PME is completed on node 0. So its topology version is now V+1.
4. Ignite#close method is called to gracefully stop the node 1
5. PME future on node1 is cancelled with the following exception
{code:java}
org.apache.ignite.internal.NodeStoppingException: Node is stopping: e2c0e324-70b6-45b8-925d-fe9a9fa5d877
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.onKernalStop0(GridCachePartitionExchangeManager.java:842)
at org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.onKernalStop(GridCacheSharedManagerAdapter.java:131)
at org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStop(GridCacheProcessor.java:819)
at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:1782)
at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:1731)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2297)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2117)
at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:308)
at org.apache.ignite.Ignition.stop(Ignition.java:202)
at org.apache.ignite.internal.IgniteKernal.close(IgniteKernal.java:2686)
at org.apache.ignite.client.LocalIgniteCluster.failNode(LocalIgniteCluster.java:110)
at org.apache.ignite.client.ReliabilityTest.lambda$assertOnUnstableCluster$17(ReliabilityTest.java:714)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
{code}
6. GridCachePartitionExchangeManager#onExchangeDone and GridDhtPartitionTopology#onExchangeDone are not called on node 1 because the PME future was completed with the exception.
7.Even when stopped, node 1 can receive and process requests from other nodes for a short period of time. Node 1 (primary) receives GridDhtAtomicAbstractUpdateRequest from node 0 (near). Which was mapped by node 0 to V+1 topology version.
8. PME future was completed on node 1 so nothing prevents it from handling the request (see GridMessageListener#onMessage)
9. The cache request is handled by GridDhtAtomicCache#updateAllAsyncInternal0
10. GridDhtCacheAdapter#needRemap check returns true because cache request was mapped on V+1 topology version but node1 local version is V (see `cctx.topology().readyTopologyVersion()` call in GridCacheAffinityManager#isCompatibleWithCurrentTopologyVersion method and clause 6).
11. node1 sends response to node0 with remap topology version == V+1. See `res.remapTopologyVersion(top.lastTopologyChangeVersion());` call. `top.lastTopologyChangeVersion()` returns V+1 because even PME future failed, it was started and related to V+1 topology version.
12. node0 receives node1 response and fails with the mentioned above stack trace
Actually it may differ whether single cache put is called or putAll.
Reproducer:
{code:java}
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.ignite;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.failure.StopNodeOrHaltFailureHandler;
import org.apache.ignite.internal.IgniteEx;
import org.apache.ignite.internal.IgniteInternalFuture;
import org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture;
import org.apache.ignite.internal.processors.cache.distributed.dht.preloader.PartitionsExchangeAware;
import org.apache.ignite.testframework.GridTestUtils;
import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
import org.junit.Test;
import static java.util.concurrent.TimeUnit.MILLISECONDS;
/** */
public class AtomicCacheOperationRemappingOnNodeStopTest extends GridCommonAbstractTest {
/** {@inheritDoc} */
@Override protected IgniteConfiguration getConfiguration() throws Exception {
return super.getConfiguration().setFailureHandler(new StopNodeOrHaltFailureHandler());
}
/** */
@Test
public void test() throws Exception {
startGrids(3);
grid(0).createCache(DEFAULT_CACHE_NAME);
IgniteEx firstNode = grid(1);
CountDownLatch zeroNodePmeFinishedLatch = new CountDownLatch(1);
CountDownLatch firstNodePmeStartedLatch = new CountDownLatch(1);
CountDownLatch firstNodePmeUnblockedLatch = new CountDownLatch(1);
CountDownLatch firstNodePmeFinishedLatch = new CountDownLatch(1);
CountDownLatch firstNodeStopUnblockedLatch = new CountDownLatch(1);
grid(0).context().cache().context().exchange().registerExchangeAwareComponent(new PartitionsExchangeAware() {
@Override public void onDoneAfterTopologyUnlock(GridDhtPartitionsExchangeFuture fut) {
zeroNodePmeFinishedLatch.countDown();
}
});
firstNode.context().cache().context().exchange().registerExchangeAwareComponent(new PartitionsExchangeAware() {
@Override public void onInitBeforeTopologyLock(GridDhtPartitionsExchangeFuture fut) {
try {
firstNodePmeStartedLatch.countDown();
firstNodePmeUnblockedLatch.await(getTestTimeout(), MILLISECONDS);
}
catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
});
stopGrid(2);
firstNodePmeStartedLatch.await(getTestTimeout(), MILLISECONDS);
firstNode.context().cache().context().exchange().lastTopologyFuture().listen(() -> {
try {
firstNodePmeFinishedLatch.countDown();
firstNodeStopUnblockedLatch.await(getTestTimeout(), MILLISECONDS);
}
catch (InterruptedException e) {
throw new RuntimeException(e);
}
});
IgniteInternalFuture<Object> firstNodeStopFut = GridTestUtils.runAsync(() -> stopGrid(1));
firstNodePmeUnblockedLatch.countDown();
firstNodePmeFinishedLatch.await(getTestTimeout(), MILLISECONDS);
zeroNodePmeFinishedLatch.await(getTestTimeout(), MILLISECONDS);
grid(0).cache(DEFAULT_CACHE_NAME).put(keyForNode(grid(0).affinity(DEFAULT_CACHE_NAME), new AtomicInteger(), firstNode.localNode()), "test-val");
firstNodeStopUnblockedLatch.countDown();
firstNodeStopFut.get(getTestTimeout(), MILLISECONDS);
}
/** {@inheritDoc} */
@Override protected long getTestTimeout() {
return 20_000;
}
}
{code}
was:
The problem was reproduced during org.apache.ignite.client.ReliabilityTest#testFailover test on TC. See attachments for the full log.
Exception:
{code:java}
java.lang.AssertionError
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.onPrimaryResponse(GridNearAtomicUpdateFuture.java:400)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateResponse(GridDhtAtomicCache.java:3236)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:311)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:306)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1084)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:594)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:396)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:306)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1862)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1483)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1387)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.thread.pool.IgniteStripedExecutor$Stripe.body(IgniteStripedExecutor.java:638)
[04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
[04:31:10] [org.apache.ignite:ignite-indexing] at java.base/java.lang.Thread.run(Thread.java:829)
{code}
The root cause of the assertion error:
Consider a cluster consisting of 3 nodes - node0, node1, node2
1. node2 is gracefully stopped
2. PME is started on node0 and node 1
3. PME is completed on node 0. So its topology version is now V+1.
4. Ignite#close method is called to gracefully stop the node 1
5. PME future on node1 is cancelled with the following exception
{code:java}
org.apache.ignite.internal.NodeStoppingException: Node is stopping: e2c0e324-70b6-45b8-925d-fe9a9fa5d877
at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.onKernalStop0(GridCachePartitionExchangeManager.java:842)
at org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.onKernalStop(GridCacheSharedManagerAdapter.java:131)
at org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStop(GridCacheProcessor.java:819)
at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:1782)
at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:1731)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2297)
at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2117)
at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:308)
at org.apache.ignite.Ignition.stop(Ignition.java:202)
at org.apache.ignite.internal.IgniteKernal.close(IgniteKernal.java:2686)
at org.apache.ignite.client.LocalIgniteCluster.failNode(LocalIgniteCluster.java:110)
at org.apache.ignite.client.ReliabilityTest.lambda$assertOnUnstableCluster$17(ReliabilityTest.java:714)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
{code}
6. GridCachePartitionExchangeManager#onExchangeDone and GridDhtPartitionTopology#onExchangeDone are not called on node 1 because the PME future was completed with the exception.
7.Even when stopped, node 1 can receive and process requests from other nodes for a short period of time. Node 1 (primary) receives GridDhtAtomicAbstractUpdateRequest from node 0 (near). Which was mapped by node 0 to V+1 topology version.
8. PME future was completed on node 1 so nothing prevents it from handling the request (see GridMessageListener#onMessage)
9. The cache request is handled by GridDhtAtomicCache#updateAllAsyncInternal0
10. GridDhtCacheAdapter#needRemap check returns true because cache request was mapped on V+1 topology version but node1 local version is V (see `cctx.topology().readyTopologyVersion()` call in GridCacheAffinityManager#isCompatibleWithCurrentTopologyVersion method and clause 6).
11. node1 sends response to node0 with remap topology version == V+1. See `res.remapTopologyVersion(top.lastTopologyChangeVersion());` call. `top.lastTopologyChangeVersion()` returns V+1 because even PME future failed, it was started and related to V+1 topology version.
12. node0 receives node1 response and fails with the mentioned above stack trace
Actually it may differ whether single cache put is called or putAll.
Reproducer:
{code:java}
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.ignite;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.atomic.AtomicInteger;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.apache.ignite.failure.StopNodeOrHaltFailureHandler;
import org.apache.ignite.internal.IgniteEx;
import org.apache.ignite.internal.IgniteInternalFuture;
import org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture;
import org.apache.ignite.internal.processors.cache.distributed.dht.preloader.PartitionsExchangeAware;
import org.apache.ignite.testframework.GridTestUtils;
import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
import org.junit.Test;
import static java.util.concurrent.TimeUnit.MILLISECONDS;
/** */
public class NodeStopCacheOperationRemappingTest extends GridCommonAbstractTest {
/** {@inheritDoc} */
@Override protected IgniteConfiguration getConfiguration() throws Exception {
return super.getConfiguration().setFailureHandler(new StopNodeOrHaltFailureHandler());
}
/** */
@Test
public void test() throws Exception {
startGrids(3);
grid(0).createCache(DEFAULT_CACHE_NAME);
IgniteEx firstNode = grid(1);
CountDownLatch zeroNodePmeFinishedLatch = new CountDownLatch(1);
CountDownLatch firstNodePmeStartedLatch = new CountDownLatch(1);
CountDownLatch firstNodePmeUnblockedLatch = new CountDownLatch(1);
CountDownLatch firstNodePmeFinishedLatch = new CountDownLatch(1);
CountDownLatch firstNodeStopUnblockedLatch = new CountDownLatch(1);
grid(0).context().cache().context().exchange().registerExchangeAwareComponent(new PartitionsExchangeAware() {
@Override public void onDoneAfterTopologyUnlock(GridDhtPartitionsExchangeFuture fut) {
zeroNodePmeFinishedLatch.countDown();
}
});
firstNode.context().cache().context().exchange().registerExchangeAwareComponent(new PartitionsExchangeAware() {
@Override public void onInitBeforeTopologyLock(GridDhtPartitionsExchangeFuture fut) {
try {
firstNodePmeStartedLatch.countDown();
firstNodePmeUnblockedLatch.await(getTestTimeout(), MILLISECONDS);
}
catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
});
stopGrid(2);
firstNodePmeStartedLatch.await(getTestTimeout(), MILLISECONDS);
firstNode.context().cache().context().exchange().lastTopologyFuture().listen(() -> {
try {
firstNodePmeFinishedLatch.countDown();
firstNodeStopUnblockedLatch.await(getTestTimeout(), MILLISECONDS);
}
catch (InterruptedException e) {
throw new RuntimeException(e);
}
});
IgniteInternalFuture<Object> firstNodeStopFut = GridTestUtils.runAsync(() -> stopGrid(1));
firstNodePmeUnblockedLatch.countDown();
firstNodePmeFinishedLatch.await(getTestTimeout(), MILLISECONDS);
zeroNodePmeFinishedLatch.await(getTestTimeout(), MILLISECONDS);
grid(0).cache(DEFAULT_CACHE_NAME).put(keyForNode(grid(0).affinity(DEFAULT_CACHE_NAME), new AtomicInteger(), firstNode.localNode()), "test-val");
firstNodeStopUnblockedLatch.countDown();
firstNodeStopFut.get(getTestTimeout(), MILLISECONDS);
}
/** {@inheritDoc} */
@Override protected long getTestTimeout() {
return 20_000;
}
}
{code}
> Node failure with Assertion Error during Atomic Cache operation topology remapping
> ----------------------------------------------------------------------------------
>
> Key: IGNITE-28381
> URL: https://issues.apache.org/jira/browse/IGNITE-28381
> Project: Ignite
> Issue Type: Bug
> Reporter: Mikhail Petrov
> Assignee: Mikhail Petrov
> Priority: Major
> Labels: ise
> Fix For: 2.19
>
> Attachments: _Apache_Ignite_2.x_Tests_Thin_Client_Java_16197.log.zip
>
>
> The problem was reproduced during org.apache.ignite.client.ReliabilityTest#testFailover test on TC. See attachments for the full log.
> Exception:
> {code:java}
> java.lang.AssertionError
> [04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.onPrimaryResponse(GridNearAtomicUpdateFuture.java:400)
> [04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateResponse(GridDhtAtomicCache.java:3236)
> [04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:311)
> [04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:306)
> [04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1084)
> [04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:594)
> [04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:396)
> [04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:306)
> [04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1862)
> [04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1483)
> [04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.managers.communication.GridIoManager$9.execute(GridIoManager.java:1387)
> [04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:55)
> [04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.thread.pool.IgniteStripedExecutor$Stripe.body(IgniteStripedExecutor.java:638)
> [04:31:10] [org.apache.ignite:ignite-indexing] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
> [04:31:10] [org.apache.ignite:ignite-indexing] at java.base/java.lang.Thread.run(Thread.java:829)
> {code}
> The root cause of the assertion error:
> Consider a cluster consisting of 3 nodes - node0, node1, node2
> 1. node2 is gracefully stopped
> 2. PME is started on node0 and node 1
> 3. PME is completed on node 0. So its topology version is now V+1.
> 4. Ignite#close method is called to gracefully stop the node 1
> 5. PME future on node1 is cancelled with the following exception
> {code:java}
> org.apache.ignite.internal.NodeStoppingException: Node is stopping: e2c0e324-70b6-45b8-925d-fe9a9fa5d877
> at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.onKernalStop0(GridCachePartitionExchangeManager.java:842)
> at org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.onKernalStop(GridCacheSharedManagerAdapter.java:131)
> at org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStop(GridCacheProcessor.java:819)
> at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:1782)
> at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:1731)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2297)
> at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2117)
> at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:308)
> at org.apache.ignite.Ignition.stop(Ignition.java:202)
> at org.apache.ignite.internal.IgniteKernal.close(IgniteKernal.java:2686)
> at org.apache.ignite.client.LocalIgniteCluster.failNode(LocalIgniteCluster.java:110)
> at org.apache.ignite.client.ReliabilityTest.lambda$assertOnUnstableCluster$17(ReliabilityTest.java:714)
> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)
> {code}
> 6. GridCachePartitionExchangeManager#onExchangeDone and GridDhtPartitionTopology#onExchangeDone are not called on node 1 because the PME future was completed with the exception.
> 7.Even when stopped, node 1 can receive and process requests from other nodes for a short period of time. Node 1 (primary) receives GridDhtAtomicAbstractUpdateRequest from node 0 (near). Which was mapped by node 0 to V+1 topology version.
> 8. PME future was completed on node 1 so nothing prevents it from handling the request (see GridMessageListener#onMessage)
> 9. The cache request is handled by GridDhtAtomicCache#updateAllAsyncInternal0
> 10. GridDhtCacheAdapter#needRemap check returns true because cache request was mapped on V+1 topology version but node1 local version is V (see `cctx.topology().readyTopologyVersion()` call in GridCacheAffinityManager#isCompatibleWithCurrentTopologyVersion method and clause 6).
> 11. node1 sends response to node0 with remap topology version == V+1. See `res.remapTopologyVersion(top.lastTopologyChangeVersion());` call. `top.lastTopologyChangeVersion()` returns V+1 because even PME future failed, it was started and related to V+1 topology version.
> 12. node0 receives node1 response and fails with the mentioned above stack trace
> Actually it may differ whether single cache put is called or putAll.
> Reproducer:
> {code:java}
> /*
> * Licensed to the Apache Software Foundation (ASF) under one or more
> * contributor license agreements. See the NOTICE file distributed with
> * this work for additional information regarding copyright ownership.
> * The ASF licenses this file to You under the Apache License, Version 2.0
> * (the "License"); you may not use this file except in compliance with
> * the License. You may obtain a copy of the License at
> *
> * http://www.apache.org/licenses/LICENSE-2.0
> *
> * Unless required by applicable law or agreed to in writing, software
> * distributed under the License is distributed on an "AS IS" BASIS,
> * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> * See the License for the specific language governing permissions and
> * limitations under the License.
> */
> package org.apache.ignite;
> import java.util.concurrent.CountDownLatch;
> import java.util.concurrent.atomic.AtomicInteger;
> import org.apache.ignite.configuration.IgniteConfiguration;
> import org.apache.ignite.failure.StopNodeOrHaltFailureHandler;
> import org.apache.ignite.internal.IgniteEx;
> import org.apache.ignite.internal.IgniteInternalFuture;
> import org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture;
> import org.apache.ignite.internal.processors.cache.distributed.dht.preloader.PartitionsExchangeAware;
> import org.apache.ignite.testframework.GridTestUtils;
> import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
> import org.junit.Test;
> import static java.util.concurrent.TimeUnit.MILLISECONDS;
> /** */
> public class AtomicCacheOperationRemappingOnNodeStopTest extends GridCommonAbstractTest {
> /** {@inheritDoc} */
> @Override protected IgniteConfiguration getConfiguration() throws Exception {
> return super.getConfiguration().setFailureHandler(new StopNodeOrHaltFailureHandler());
> }
> /** */
> @Test
> public void test() throws Exception {
> startGrids(3);
> grid(0).createCache(DEFAULT_CACHE_NAME);
> IgniteEx firstNode = grid(1);
> CountDownLatch zeroNodePmeFinishedLatch = new CountDownLatch(1);
> CountDownLatch firstNodePmeStartedLatch = new CountDownLatch(1);
> CountDownLatch firstNodePmeUnblockedLatch = new CountDownLatch(1);
> CountDownLatch firstNodePmeFinishedLatch = new CountDownLatch(1);
> CountDownLatch firstNodeStopUnblockedLatch = new CountDownLatch(1);
> grid(0).context().cache().context().exchange().registerExchangeAwareComponent(new PartitionsExchangeAware() {
> @Override public void onDoneAfterTopologyUnlock(GridDhtPartitionsExchangeFuture fut) {
> zeroNodePmeFinishedLatch.countDown();
> }
> });
> firstNode.context().cache().context().exchange().registerExchangeAwareComponent(new PartitionsExchangeAware() {
> @Override public void onInitBeforeTopologyLock(GridDhtPartitionsExchangeFuture fut) {
> try {
> firstNodePmeStartedLatch.countDown();
> firstNodePmeUnblockedLatch.await(getTestTimeout(), MILLISECONDS);
> }
> catch (InterruptedException e) {
> throw new RuntimeException(e);
> }
> }
> });
> stopGrid(2);
> firstNodePmeStartedLatch.await(getTestTimeout(), MILLISECONDS);
> firstNode.context().cache().context().exchange().lastTopologyFuture().listen(() -> {
> try {
> firstNodePmeFinishedLatch.countDown();
> firstNodeStopUnblockedLatch.await(getTestTimeout(), MILLISECONDS);
> }
> catch (InterruptedException e) {
> throw new RuntimeException(e);
> }
> });
> IgniteInternalFuture<Object> firstNodeStopFut = GridTestUtils.runAsync(() -> stopGrid(1));
> firstNodePmeUnblockedLatch.countDown();
> firstNodePmeFinishedLatch.await(getTestTimeout(), MILLISECONDS);
> zeroNodePmeFinishedLatch.await(getTestTimeout(), MILLISECONDS);
> grid(0).cache(DEFAULT_CACHE_NAME).put(keyForNode(grid(0).affinity(DEFAULT_CACHE_NAME), new AtomicInteger(), firstNode.localNode()), "test-val");
> firstNodeStopUnblockedLatch.countDown();
> firstNodeStopFut.get(getTestTimeout(), MILLISECONDS);
> }
> /** {@inheritDoc} */
> @Override protected long getTestTimeout() {
> return 20_000;
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)