You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "sergey-chugunov-1985 (via GitHub)" <gi...@apache.org> on 2025/07/08 14:31:41 UTC
[PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
sergey-chugunov-1985 opened a new pull request, #12178:
URL: https://github.com/apache/ignite/pull/12178
Thank you for submitting the pull request to the Apache Ignite.
In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:
### The Contribution Checklist
- [ ] There is a single JIRA ticket related to the pull request.
- [ ] The web-link to the pull request is attached to the JIRA ticket.
- [ ] The JIRA ticket has the _Patch Available_ state.
- [ ] The pull request body describes changes that have been made.
The description explains _WHAT_ and _WHY_ was made instead of _HOW_.
- [ ] The pull request title is treated as the final commit message.
The following pattern must be used: `IGNITE-XXXX Change summary` where `XXXX` - number of JIRA issue.
- [ ] A reviewer has been mentioned through the JIRA comments
(see [the Maintainers list](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers))
- [ ] The pull request has been checked by the Teamcity Bot and
the `green visa` attached to the JIRA ticket (see [TC.Bot: Check PR](https://mtcga.gridgain.com/prs.html))
### Notes
- [How to Contribute](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute)
- [Coding abbreviation rules](https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules)
- [Coding Guidelines](https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines)
- [Apache Ignite Teamcity Bot](https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Teamcity+Bot)
If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com _#ignite_ channel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sergey-chugunov-1985 (via GitHub)" <gi...@apache.org>.
sergey-chugunov-1985 merged PR #12178:
URL: https://github.com/apache/ignite/pull/12178
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #12178:
URL: https://github.com/apache/ignite/pull/12178#issuecomment-3078881729
## [](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_coverage&view=list)
 [3.7% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #12178:
URL: https://github.com/apache/ignite/pull/12178#issuecomment-3162708146
## [](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sergey-chugunov-1985 (via GitHub)" <gi...@apache.org>.
sergey-chugunov-1985 commented on code in PR #12178:
URL: https://github.com/apache/ignite/pull/12178#discussion_r2236055009
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
Review Comment:
At least this is how tx protocol works right now: it propagates exception from commit phase only when it happens on primary node.
Exceptions from backup are not propagated - I suspect implementing it that way was a deliberate choice but I don't have a proof of that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sergey-chugunov-1985 (via GitHub)" <gi...@apache.org>.
sergey-chugunov-1985 commented on code in PR #12178:
URL: https://github.com/apache/ignite/pull/12178#discussion_r2235949638
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
+
+ // check on affected node should pass as the node is alive and cache entries are cleaned up from the cache
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+ }
+ else {
+ // two servers and a client node which is a tx coordinator
+ waitForTopology(3);
+
+ assertTrue("Client node should survive test scenario",
+ G.allGrids()
+ .stream()
+ .filter(ignite -> ((IgniteEx)ignite).context().clientNode())
+ .count() == 1);
+ }
+
+ }
+ else {
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+
+ checkKeysOnHealthyNodes(keysOnFaultyNode);
+ }
+
+ /** */
+ private void fillCache(IgniteCache<Integer, Integer> cache, int numOfKeys) {
+ for (int i = 0; i < numOfKeys; i++) {
Review Comment:
Fixed.
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
+
+ // check on affected node should pass as the node is alive and cache entries are cleaned up from the cache
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+ }
+ else {
+ // two servers and a client node which is a tx coordinator
+ waitForTopology(3);
+
+ assertTrue("Client node should survive test scenario",
+ G.allGrids()
+ .stream()
+ .filter(ignite -> ((IgniteEx)ignite).context().clientNode())
+ .count() == 1);
+ }
+
+ }
+ else {
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+
+ checkKeysOnHealthyNodes(keysOnFaultyNode);
+ }
+
+ /** */
+ private void fillCache(IgniteCache<Integer, Integer> cache, int numOfKeys) {
+ for (int i = 0; i < numOfKeys; i++) {
+ cache.put(i, i);
+ }
+ }
+
+ /** */
+ private void checkKeysOnFaultyNode(List<Integer> keysToCheck) {
+ IgniteCache<Object, Object> cache = grid(FAULTY_NODE_IDX).cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals(storeStgy.getFromStore(key), cache.get(key));
+ }
+ }
+
+ /** */
+ private void checkKeysOnHealthyNodes(List<Integer> keysToCheck) {
+ for (int i = 0; i < gridCount(); i++) {
+ if (i != FAULTY_NODE_IDX) {
+ IgniteEx ig = grid(i);
+
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals("Key inconsistent with CacheStore found on node " + i + "; nodeName " + ig.name(),
+ storeStgy.getFromStore(key),
+ cache.get(key));
+ }
+ }
+ }
+ }
+
+ /** */
+ private void updateKeysInTx(Ignite ig, List<Integer> keys) {
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ try (Transaction tx = ig.transactions().txStart()) {
+ for (Integer key : keys) {
+ cache.put(key, KEY_UPDATE_FUNCTION.apply(key));
+ }
+
+ tx.commit();
+ }
+ }
+
+ /** */
+ private void updateKeysInTxWithExceptionCatching(Ignite ig, List<Integer> keys) {
+ try {
+ updateKeysInTx(ig, keys);
+ }
+ catch (Exception ignored) {
+ // No-op
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #12178:
URL: https://github.com/apache/ignite/pull/12178#issuecomment-3126741330
## [](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) **Quality Gate failed**
Failed conditions
 [2 New Code Smells](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) (required ≤ 1)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sergey-chugunov-1985 (via GitHub)" <gi...@apache.org>.
sergey-chugunov-1985 commented on code in PR #12178:
URL: https://github.com/apache/ignite/pull/12178#discussion_r2235988570
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
Review Comment:
This piece of code is rather entangled and hard to decipher, but the core distinction here is simple. When FH is not configured, no nodes are failing. When FH is configured nodes are always failing *except* a scenario when tx_coordinator is the same node as primary or backup. Only in that case things get tricky and personally I didn't quite get to the bottom of it.
I can create a ticket for this scenario and title it like "Errors during tx commit are not propagated to FH on Tx Coordinator". But I left this case intentionally out of scope of this PR as fixing it in a right way could require a lot of efforts.
Are you OK with that plan?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #12178:
URL: https://github.com/apache/ignite/pull/12178#issuecomment-3073793411
## [](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #12178:
URL: https://github.com/apache/ignite/pull/12178#issuecomment-3135969453
## [](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) **Quality Gate failed**
Failed conditions
 [2 New Code Smells](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) (required ≤ 1)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #12178:
URL: https://github.com/apache/ignite/pull/12178#issuecomment-3141124445
## [](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #12178:
URL: https://github.com/apache/ignite/pull/12178#issuecomment-3135238801
## [](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) **Quality Gate failed**
Failed conditions
 [2 New Code Smells](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) (required ≤ 1)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #12178:
URL: https://github.com/apache/ignite/pull/12178#issuecomment-3052381197
## [](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sergey-chugunov-1985 (via GitHub)" <gi...@apache.org>.
sergey-chugunov-1985 commented on code in PR #12178:
URL: https://github.com/apache/ignite/pull/12178#discussion_r2235948133
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxAdapter.java:
##########
@@ -504,6 +505,46 @@ public void storeEnabled(boolean storeEnabled) {
return storeEnabled() && txState().storeWriteThrough(cctx);
}
+ /**
+ * Clears tx entries if they could be re-read from a read-through {@link CacheStore}.
+ * <p>
+ * This method takes effect only if all Ignite caches involved into transaction configured with CacheStores in
+ * read-through mode.
+ */
+ protected void clearCacheStoreBackedEntries() {
+ if (txState().cacheIds() == null)
+ return;
+
+ GridIntIterator iter = txState().cacheIds().iterator();
+ while (iter.hasNext()) {
+ int cacheId = iter.next();
+
+ if (cctx.cacheContext(cacheId) == null)
Review Comment:
I removed this null check as it indeed looks redundant.
About not read-through caches. It is possible of cource to clear all involved caches unconditionally but I tried to avoid touching caches that are not backed by CacheStore thus we cannot recover their values easily. That was my motivation.
But it is possible to clear only read-through caches and leave all others - do you think it is a better idea?
Clear only the data you can easily recover.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sergey-chugunov-1985 (via GitHub)" <gi...@apache.org>.
sergey-chugunov-1985 commented on code in PR #12178:
URL: https://github.com/apache/ignite/pull/12178#discussion_r2246214350
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #12178:
URL: https://github.com/apache/ignite/pull/12178#issuecomment-3083054493
## [](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) **Quality Gate failed**
Failed conditions
 [2 New Code Smells](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) (required ≤ 1)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sergey-chugunov-1985 (via GitHub)" <gi...@apache.org>.
sergey-chugunov-1985 commented on code in PR #12178:
URL: https://github.com/apache/ignite/pull/12178#discussion_r2235962661
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxAdapter.java:
##########
@@ -504,6 +505,46 @@ public void storeEnabled(boolean storeEnabled) {
return storeEnabled() && txState().storeWriteThrough(cctx);
}
+ /**
+ * Clears tx entries if they could be re-read from a read-through {@link CacheStore}.
+ * <p>
+ * This method takes effect only if all Ignite caches involved into transaction configured with CacheStores in
+ * read-through mode.
+ */
+ protected void clearCacheStoreBackedEntries() {
+ if (txState().cacheIds() == null)
+ return;
+
+ GridIntIterator iter = txState().cacheIds().iterator();
+ while (iter.hasNext()) {
+ int cacheId = iter.next();
+
+ if (cctx.cacheContext(cacheId) == null)
+ return;
+
+ if (!cctx.cacheContext(cacheId).config().isReadThrough())
+ return;
+ }
+
+ for (IgniteTxEntry e : writeMap().values()) {
+ try {
+ GridCacheEntryEx entry = e.cached();
+
+ if (e.op() != NOOP) {
Review Comment:
I agree that this implementation looks similar to `uncommit` but the reason I decided to implement it in a separate method was that I wanted to keep `uncommit` logic untouched so no possible exceptions thrown inside `clearCacheStoreBackedEntries` affect `uncommit`. Does it make sense to you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "alex-plekhanov (via GitHub)" <gi...@apache.org>.
alex-plekhanov commented on code in PR #12178:
URL: https://github.com/apache/ignite/pull/12178#discussion_r2228458511
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxAdapter.java:
##########
@@ -504,6 +505,46 @@ public void storeEnabled(boolean storeEnabled) {
return storeEnabled() && txState().storeWriteThrough(cctx);
}
+ /**
+ * Clears tx entries if they could be re-read from a read-through {@link CacheStore}.
+ * <p>
+ * This method takes effect only if all Ignite caches involved into transaction configured with CacheStores in
+ * read-through mode.
+ */
+ protected void clearCacheStoreBackedEntries() {
+ if (txState().cacheIds() == null)
+ return;
+
+ GridIntIterator iter = txState().cacheIds().iterator();
+ while (iter.hasNext()) {
+ int cacheId = iter.next();
+
+ if (cctx.cacheContext(cacheId) == null)
Review Comment:
Looks weird. Is it really possible? Maybe get config from cache descriptor? (`cctx.cache().cacheDescriptor(cacheId).cacheConfiguration()`)
If there are caches in tx, that is not isReadThrough, entries will not be cleared. Is it ok?
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxAdapter.java:
##########
@@ -504,6 +505,46 @@ public void storeEnabled(boolean storeEnabled) {
return storeEnabled() && txState().storeWriteThrough(cctx);
}
+ /**
+ * Clears tx entries if they could be re-read from a read-through {@link CacheStore}.
+ * <p>
+ * This method takes effect only if all Ignite caches involved into transaction configured with CacheStores in
+ * read-through mode.
+ */
+ protected void clearCacheStoreBackedEntries() {
+ if (txState().cacheIds() == null)
+ return;
+
+ GridIntIterator iter = txState().cacheIds().iterator();
+ while (iter.hasNext()) {
+ int cacheId = iter.next();
+
+ if (cctx.cacheContext(cacheId) == null)
+ return;
+
+ if (!cctx.cacheContext(cacheId).config().isReadThrough())
+ return;
+ }
+
+ for (IgniteTxEntry e : writeMap().values()) {
+ try {
+ GridCacheEntryEx entry = e.cached();
+
+ if (e.op() != NOOP) {
+ entry.clear(xidVer, true);
Review Comment:
Redundant braces
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
+
+ // check on affected node should pass as the node is alive and cache entries are cleaned up from the cache
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+ }
+ else {
+ // two servers and a client node which is a tx coordinator
+ waitForTopology(3);
+
+ assertTrue("Client node should survive test scenario",
+ G.allGrids()
+ .stream()
+ .filter(ignite -> ((IgniteEx)ignite).context().clientNode())
+ .count() == 1);
+ }
+
+ }
+ else {
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+
+ checkKeysOnHealthyNodes(keysOnFaultyNode);
+ }
+
+ /** */
+ private void fillCache(IgniteCache<Integer, Integer> cache, int numOfKeys) {
+ for (int i = 0; i < numOfKeys; i++) {
+ cache.put(i, i);
+ }
+ }
+
+ /** */
+ private void checkKeysOnFaultyNode(List<Integer> keysToCheck) {
+ IgniteCache<Object, Object> cache = grid(FAULTY_NODE_IDX).cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals(storeStgy.getFromStore(key), cache.get(key));
+ }
+ }
+
+ /** */
+ private void checkKeysOnHealthyNodes(List<Integer> keysToCheck) {
+ for (int i = 0; i < gridCount(); i++) {
+ if (i != FAULTY_NODE_IDX) {
+ IgniteEx ig = grid(i);
+
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals("Key inconsistent with CacheStore found on node " + i + "; nodeName " + ig.name(),
+ storeStgy.getFromStore(key),
+ cache.get(key));
+ }
+ }
+ }
+ }
+
+ /** */
+ private void updateKeysInTx(Ignite ig, List<Integer> keys) {
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ try (Transaction tx = ig.transactions().txStart()) {
+ for (Integer key : keys) {
Review Comment:
Redundant braces
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
+
+ // check on affected node should pass as the node is alive and cache entries are cleaned up from the cache
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+ }
+ else {
+ // two servers and a client node which is a tx coordinator
+ waitForTopology(3);
+
+ assertTrue("Client node should survive test scenario",
+ G.allGrids()
+ .stream()
+ .filter(ignite -> ((IgniteEx)ignite).context().clientNode())
+ .count() == 1);
+ }
+
+ }
+ else {
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+
+ checkKeysOnHealthyNodes(keysOnFaultyNode);
+ }
+
+ /** */
+ private void fillCache(IgniteCache<Integer, Integer> cache, int numOfKeys) {
+ for (int i = 0; i < numOfKeys; i++) {
Review Comment:
Redundant braces
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
Review Comment:
Please fix comments (all) according to Ignite code style (start with upper-case, end with point).
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
+
+ // check on affected node should pass as the node is alive and cache entries are cleaned up from the cache
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+ }
+ else {
+ // two servers and a client node which is a tx coordinator
+ waitForTopology(3);
+
+ assertTrue("Client node should survive test scenario",
+ G.allGrids()
+ .stream()
+ .filter(ignite -> ((IgniteEx)ignite).context().clientNode())
+ .count() == 1);
+ }
+
+ }
+ else {
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+
+ checkKeysOnHealthyNodes(keysOnFaultyNode);
+ }
+
+ /** */
+ private void fillCache(IgniteCache<Integer, Integer> cache, int numOfKeys) {
+ for (int i = 0; i < numOfKeys; i++) {
+ cache.put(i, i);
+ }
+ }
+
+ /** */
+ private void checkKeysOnFaultyNode(List<Integer> keysToCheck) {
+ IgniteCache<Object, Object> cache = grid(FAULTY_NODE_IDX).cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals(storeStgy.getFromStore(key), cache.get(key));
+ }
+ }
+
+ /** */
+ private void checkKeysOnHealthyNodes(List<Integer> keysToCheck) {
+ for (int i = 0; i < gridCount(); i++) {
+ if (i != FAULTY_NODE_IDX) {
+ IgniteEx ig = grid(i);
+
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals("Key inconsistent with CacheStore found on node " + i + "; nodeName " + ig.name(),
+ storeStgy.getFromStore(key),
+ cache.get(key));
+ }
+ }
+ }
+ }
+
+ /** */
+ private void updateKeysInTx(Ignite ig, List<Integer> keys) {
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ try (Transaction tx = ig.transactions().txStart()) {
+ for (Integer key : keys) {
+ cache.put(key, KEY_UPDATE_FUNCTION.apply(key));
+ }
+
+ tx.commit();
+ }
+ }
+
+ /** */
+ private void updateKeysInTxWithExceptionCatching(Ignite ig, List<Integer> keys) {
+ try {
+ updateKeysInTx(ig, keys);
+ }
+ catch (Exception ignored) {
+ // No-op
+ }
+ }
+
+ /** */
+ private static class FaultyNodeInterceptor extends CacheInterceptorAdapter<Integer, Integer> {
+ /** */
+ private static final String FAULTY_NODE_SUFFIX = "Test" + FAULTY_NODE_IDX;
+
+ /** */
+ private final String instanceName;
+
+ /** */
+ private final FaultyNodeRole faultyNodeRole;
+
+ /** */
+ private final Map<Integer, Boolean> map = new ConcurrentHashMap<>();
+
+ /**
+ * @param instanceName Ignite node instance name.
+ * @param faultyNodeRole Flag if node is tx coordinator.
+ */
+ private FaultyNodeInterceptor(String instanceName, FaultyNodeRole faultyNodeRole) {
+ this.instanceName = instanceName;
+ this.faultyNodeRole = faultyNodeRole;
+ }
+
+ /** {@inheritDoc} */
+ @Override public @Nullable Integer onBeforePut(Cache.Entry<Integer, Integer> entry, Integer newVal) {
+ if (newVal < 2 * KEYS_NUMBER)
+ return newVal;
+
+ if (instanceName.contains(FAULTY_NODE_SUFFIX)) {
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (!map.containsKey(newVal))
Review Comment:
For tx coordinator we have two calls of this method for each key? Is it expected? Can you please add a comment?
Also, maybe use `Set` and simplify to `if (!set.add(newVal)) throw new exception`?
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
+
+ // check on affected node should pass as the node is alive and cache entries are cleaned up from the cache
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+ }
+ else {
+ // two servers and a client node which is a tx coordinator
+ waitForTopology(3);
+
+ assertTrue("Client node should survive test scenario",
+ G.allGrids()
+ .stream()
+ .filter(ignite -> ((IgniteEx)ignite).context().clientNode())
+ .count() == 1);
+ }
+
+ }
+ else {
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+
+ checkKeysOnHealthyNodes(keysOnFaultyNode);
+ }
+
+ /** */
+ private void fillCache(IgniteCache<Integer, Integer> cache, int numOfKeys) {
+ for (int i = 0; i < numOfKeys; i++) {
+ cache.put(i, i);
+ }
+ }
+
+ /** */
+ private void checkKeysOnFaultyNode(List<Integer> keysToCheck) {
+ IgniteCache<Object, Object> cache = grid(FAULTY_NODE_IDX).cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals(storeStgy.getFromStore(key), cache.get(key));
+ }
+ }
+
+ /** */
+ private void checkKeysOnHealthyNodes(List<Integer> keysToCheck) {
+ for (int i = 0; i < gridCount(); i++) {
+ if (i != FAULTY_NODE_IDX) {
+ IgniteEx ig = grid(i);
+
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals("Key inconsistent with CacheStore found on node " + i + "; nodeName " + ig.name(),
+ storeStgy.getFromStore(key),
+ cache.get(key));
+ }
+ }
+ }
+ }
+
+ /** */
+ private void updateKeysInTx(Ignite ig, List<Integer> keys) {
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ try (Transaction tx = ig.transactions().txStart()) {
+ for (Integer key : keys) {
+ cache.put(key, KEY_UPDATE_FUNCTION.apply(key));
+ }
+
+ tx.commit();
+ }
+ }
+
+ /** */
+ private void updateKeysInTxWithExceptionCatching(Ignite ig, List<Integer> keys) {
+ try {
+ updateKeysInTx(ig, keys);
+ }
+ catch (Exception ignored) {
+ // No-op
+ }
+ }
+
+ /** */
+ private static class FaultyNodeInterceptor extends CacheInterceptorAdapter<Integer, Integer> {
+ /** */
+ private static final String FAULTY_NODE_SUFFIX = "Test" + FAULTY_NODE_IDX;
+
+ /** */
+ private final String instanceName;
+
+ /** */
+ private final FaultyNodeRole faultyNodeRole;
+
+ /** */
+ private final Map<Integer, Boolean> map = new ConcurrentHashMap<>();
+
+ /**
+ * @param instanceName Ignite node instance name.
+ * @param faultyNodeRole Flag if node is tx coordinator.
+ */
+ private FaultyNodeInterceptor(String instanceName, FaultyNodeRole faultyNodeRole) {
+ this.instanceName = instanceName;
+ this.faultyNodeRole = faultyNodeRole;
+ }
+
+ /** {@inheritDoc} */
+ @Override public @Nullable Integer onBeforePut(Cache.Entry<Integer, Integer> entry, Integer newVal) {
+ if (newVal < 2 * KEYS_NUMBER)
+ return newVal;
+
+ if (instanceName.contains(FAULTY_NODE_SUFFIX)) {
Review Comment:
Fragile. Test will fail if test name fill be refactored or if someone change ignite instance name format for tests. Maybe it's better to declare `boolean faultyInstance`? And pass it like `getTestIgniteInstanceIndex(igniteInstanceName) == FAULTY_NODE_IDX`?
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
Review Comment:
Redundant braces
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
+
+ // check on affected node should pass as the node is alive and cache entries are cleaned up from the cache
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+ }
+ else {
+ // two servers and a client node which is a tx coordinator
+ waitForTopology(3);
+
+ assertTrue("Client node should survive test scenario",
+ G.allGrids()
+ .stream()
+ .filter(ignite -> ((IgniteEx)ignite).context().clientNode())
+ .count() == 1);
+ }
+
+ }
+ else {
Review Comment:
Redundant braces
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
+
+ // check on affected node should pass as the node is alive and cache entries are cleaned up from the cache
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+ }
+ else {
+ // two servers and a client node which is a tx coordinator
+ waitForTopology(3);
+
+ assertTrue("Client node should survive test scenario",
+ G.allGrids()
+ .stream()
+ .filter(ignite -> ((IgniteEx)ignite).context().clientNode())
+ .count() == 1);
+ }
+
Review Comment:
Redundant NL
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
Review Comment:
Please use generics to avoid IDE warnings
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
+
+ // check on affected node should pass as the node is alive and cache entries are cleaned up from the cache
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+ }
+ else {
+ // two servers and a client node which is a tx coordinator
+ waitForTopology(3);
+
+ assertTrue("Client node should survive test scenario",
+ G.allGrids()
+ .stream()
+ .filter(ignite -> ((IgniteEx)ignite).context().clientNode())
+ .count() == 1);
+ }
+
+ }
+ else {
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+
+ checkKeysOnHealthyNodes(keysOnFaultyNode);
+ }
+
+ /** */
+ private void fillCache(IgniteCache<Integer, Integer> cache, int numOfKeys) {
+ for (int i = 0; i < numOfKeys; i++) {
+ cache.put(i, i);
+ }
+ }
+
+ /** */
+ private void checkKeysOnFaultyNode(List<Integer> keysToCheck) {
+ IgniteCache<Object, Object> cache = grid(FAULTY_NODE_IDX).cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
Review Comment:
Redundant braces
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
Review Comment:
Abbreviation for `handler` should be used.
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
+
+ // check on affected node should pass as the node is alive and cache entries are cleaned up from the cache
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+ }
+ else {
+ // two servers and a client node which is a tx coordinator
+ waitForTopology(3);
+
+ assertTrue("Client node should survive test scenario",
+ G.allGrids()
+ .stream()
+ .filter(ignite -> ((IgniteEx)ignite).context().clientNode())
+ .count() == 1);
+ }
+
+ }
+ else {
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+
+ checkKeysOnHealthyNodes(keysOnFaultyNode);
+ }
+
+ /** */
+ private void fillCache(IgniteCache<Integer, Integer> cache, int numOfKeys) {
+ for (int i = 0; i < numOfKeys; i++) {
+ cache.put(i, i);
+ }
+ }
+
+ /** */
+ private void checkKeysOnFaultyNode(List<Integer> keysToCheck) {
+ IgniteCache<Object, Object> cache = grid(FAULTY_NODE_IDX).cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals(storeStgy.getFromStore(key), cache.get(key));
+ }
+ }
+
+ /** */
+ private void checkKeysOnHealthyNodes(List<Integer> keysToCheck) {
+ for (int i = 0; i < gridCount(); i++) {
+ if (i != FAULTY_NODE_IDX) {
+ IgniteEx ig = grid(i);
+
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals("Key inconsistent with CacheStore found on node " + i + "; nodeName " + ig.name(),
+ storeStgy.getFromStore(key),
+ cache.get(key));
+ }
+ }
+ }
+ }
+
+ /** */
+ private void updateKeysInTx(Ignite ig, List<Integer> keys) {
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ try (Transaction tx = ig.transactions().txStart()) {
+ for (Integer key : keys) {
+ cache.put(key, KEY_UPDATE_FUNCTION.apply(key));
+ }
+
+ tx.commit();
+ }
+ }
+
+ /** */
+ private void updateKeysInTxWithExceptionCatching(Ignite ig, List<Integer> keys) {
+ try {
+ updateKeysInTx(ig, keys);
+ }
+ catch (Exception ignored) {
+ // No-op
Review Comment:
Point at the end
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
Review Comment:
Is it expected behaviour, that some nodes fail with FH, some node not, some nodes fail without FH? Or we check this case just because it works that way? If it works that way and we don't know the reason, we shouldn't check this cases in tests. Someone could make an incorrect assumption based on the scenario described in this test and think that's how it's supposed to work. Perhaps here we can simplify the code and check keys only on alive nodes.
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
Review Comment:
Is it expected behaviour that commit throws an exception only when failed to store on primary nodes?
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
Review Comment:
The only difference: tx coordinator hosting primary partition fails when near cache is used. Perhaps it's not exceptional case (we have almost random node failures for other scenarios)
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxAdapter.java:
##########
@@ -504,6 +505,46 @@ public void storeEnabled(boolean storeEnabled) {
return storeEnabled() && txState().storeWriteThrough(cctx);
}
+ /**
+ * Clears tx entries if they could be re-read from a read-through {@link CacheStore}.
+ * <p>
+ * This method takes effect only if all Ignite caches involved into transaction configured with CacheStores in
+ * read-through mode.
+ */
+ protected void clearCacheStoreBackedEntries() {
+ if (txState().cacheIds() == null)
+ return;
+
+ GridIntIterator iter = txState().cacheIds().iterator();
+ while (iter.hasNext()) {
+ int cacheId = iter.next();
+
+ if (cctx.cacheContext(cacheId) == null)
+ return;
+
+ if (!cctx.cacheContext(cacheId).config().isReadThrough())
+ return;
+ }
+
+ for (IgniteTxEntry e : writeMap().values()) {
+ try {
+ GridCacheEntryEx entry = e.cached();
+
+ if (e.op() != NOOP) {
Review Comment:
Looks pretty similar to `uncommit()` method. Maybe move this code to `uncommit`?
Like additional condition:
```
if (entry.context().readThrough())
entry.clear(xidVer, true);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sergey-chugunov-1985 (via GitHub)" <gi...@apache.org>.
sergey-chugunov-1985 commented on code in PR #12178:
URL: https://github.com/apache/ignite/pull/12178#discussion_r2253486408
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
Review Comment:
Fixed: failures in this scenario indeed were because of data races in test code and not a sign of another problem. Thus I mitigated this race and removed tests for near configuration setting completely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #12178:
URL: https://github.com/apache/ignite/pull/12178#issuecomment-3135322075
## [](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) **Quality Gate failed**
Failed conditions
 [2 New Code Smells](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) (required ≤ 1)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178)
##
 Catch issues before they fail your Quality Gate with our IDE extension  [SonarQube for IDE](https://www.sonarsource.com/products/sonarlint/features/connected-mode/?referrer=pull-request)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #12178:
URL: https://github.com/apache/ignite/pull/12178#issuecomment-3136203343
## [](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sergey-chugunov-1985 (via GitHub)" <gi...@apache.org>.
sergey-chugunov-1985 commented on code in PR #12178:
URL: https://github.com/apache/ignite/pull/12178#discussion_r2246206981
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #12178:
URL: https://github.com/apache/ignite/pull/12178#issuecomment-3049230178
## [](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sergey-chugunov-1985 (via GitHub)" <gi...@apache.org>.
sergey-chugunov-1985 commented on code in PR #12178:
URL: https://github.com/apache/ignite/pull/12178#discussion_r2235988570
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
Review Comment:
This piece of code is rather entangled and hard to decipher, but the core distinction here is simple. When FH is not configured, no nodes are failing. When FH is configured nodes are always failing *except* a scenario when tx_coordinator is the same node as primary or backup. Only in that case things get tricky and personally I didn't quite get to the bottom of it.
I can create a ticket for this scenario and title it like "Errors during tx commit are not propagated to FH on Tx Coordinator". But I left this case intentionally out of scope of this PR as fixing it in a right way could require a lot of efforts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sergey-chugunov-1985 (via GitHub)" <gi...@apache.org>.
sergey-chugunov-1985 commented on code in PR #12178:
URL: https://github.com/apache/ignite/pull/12178#discussion_r2246212944
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
+
+ // check on affected node should pass as the node is alive and cache entries are cleaned up from the cache
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+ }
+ else {
+ // two servers and a client node which is a tx coordinator
+ waitForTopology(3);
+
+ assertTrue("Client node should survive test scenario",
+ G.allGrids()
+ .stream()
+ .filter(ignite -> ((IgniteEx)ignite).context().clientNode())
+ .count() == 1);
+ }
+
+ }
+ else {
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+
+ checkKeysOnHealthyNodes(keysOnFaultyNode);
+ }
+
+ /** */
+ private void fillCache(IgniteCache<Integer, Integer> cache, int numOfKeys) {
+ for (int i = 0; i < numOfKeys; i++) {
+ cache.put(i, i);
+ }
+ }
+
+ /** */
+ private void checkKeysOnFaultyNode(List<Integer> keysToCheck) {
+ IgniteCache<Object, Object> cache = grid(FAULTY_NODE_IDX).cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals(storeStgy.getFromStore(key), cache.get(key));
+ }
+ }
+
+ /** */
+ private void checkKeysOnHealthyNodes(List<Integer> keysToCheck) {
+ for (int i = 0; i < gridCount(); i++) {
+ if (i != FAULTY_NODE_IDX) {
+ IgniteEx ig = grid(i);
+
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals("Key inconsistent with CacheStore found on node " + i + "; nodeName " + ig.name(),
+ storeStgy.getFromStore(key),
+ cache.get(key));
+ }
+ }
+ }
+ }
+
+ /** */
+ private void updateKeysInTx(Ignite ig, List<Integer> keys) {
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ try (Transaction tx = ig.transactions().txStart()) {
+ for (Integer key : keys) {
+ cache.put(key, KEY_UPDATE_FUNCTION.apply(key));
+ }
+
+ tx.commit();
+ }
+ }
+
+ /** */
+ private void updateKeysInTxWithExceptionCatching(Ignite ig, List<Integer> keys) {
+ try {
+ updateKeysInTx(ig, keys);
+ }
+ catch (Exception ignored) {
+ // No-op
+ }
+ }
+
+ /** */
+ private static class FaultyNodeInterceptor extends CacheInterceptorAdapter<Integer, Integer> {
+ /** */
+ private static final String FAULTY_NODE_SUFFIX = "Test" + FAULTY_NODE_IDX;
+
+ /** */
+ private final String instanceName;
+
+ /** */
+ private final FaultyNodeRole faultyNodeRole;
+
+ /** */
+ private final Map<Integer, Boolean> map = new ConcurrentHashMap<>();
+
+ /**
+ * @param instanceName Ignite node instance name.
+ * @param faultyNodeRole Flag if node is tx coordinator.
+ */
+ private FaultyNodeInterceptor(String instanceName, FaultyNodeRole faultyNodeRole) {
+ this.instanceName = instanceName;
+ this.faultyNodeRole = faultyNodeRole;
+ }
+
+ /** {@inheritDoc} */
+ @Override public @Nullable Integer onBeforePut(Cache.Entry<Integer, Integer> entry, Integer newVal) {
+ if (newVal < 2 * KEYS_NUMBER)
+ return newVal;
+
+ if (instanceName.contains(FAULTY_NODE_SUFFIX)) {
Review Comment:
Fixed.
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sergey-chugunov-1985 (via GitHub)" <gi...@apache.org>.
sergey-chugunov-1985 commented on code in PR #12178:
URL: https://github.com/apache/ignite/pull/12178#discussion_r2246213975
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
+
+ // check on affected node should pass as the node is alive and cache entries are cleaned up from the cache
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+ }
+ else {
+ // two servers and a client node which is a tx coordinator
+ waitForTopology(3);
+
+ assertTrue("Client node should survive test scenario",
+ G.allGrids()
+ .stream()
+ .filter(ignite -> ((IgniteEx)ignite).context().clientNode())
+ .count() == 1);
+ }
+
+ }
+ else {
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+
+ checkKeysOnHealthyNodes(keysOnFaultyNode);
+ }
+
+ /** */
+ private void fillCache(IgniteCache<Integer, Integer> cache, int numOfKeys) {
+ for (int i = 0; i < numOfKeys; i++) {
+ cache.put(i, i);
+ }
+ }
+
+ /** */
+ private void checkKeysOnFaultyNode(List<Integer> keysToCheck) {
+ IgniteCache<Object, Object> cache = grid(FAULTY_NODE_IDX).cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals(storeStgy.getFromStore(key), cache.get(key));
+ }
+ }
+
+ /** */
+ private void checkKeysOnHealthyNodes(List<Integer> keysToCheck) {
+ for (int i = 0; i < gridCount(); i++) {
+ if (i != FAULTY_NODE_IDX) {
+ IgniteEx ig = grid(i);
+
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals("Key inconsistent with CacheStore found on node " + i + "; nodeName " + ig.name(),
+ storeStgy.getFromStore(key),
+ cache.get(key));
+ }
+ }
+ }
+ }
+
+ /** */
+ private void updateKeysInTx(Ignite ig, List<Integer> keys) {
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ try (Transaction tx = ig.transactions().txStart()) {
+ for (Integer key : keys) {
+ cache.put(key, KEY_UPDATE_FUNCTION.apply(key));
+ }
+
+ tx.commit();
+ }
+ }
+
+ /** */
+ private void updateKeysInTxWithExceptionCatching(Ignite ig, List<Integer> keys) {
+ try {
+ updateKeysInTx(ig, keys);
+ }
+ catch (Exception ignored) {
+ // No-op
+ }
+ }
+
+ /** */
+ private static class FaultyNodeInterceptor extends CacheInterceptorAdapter<Integer, Integer> {
+ /** */
+ private static final String FAULTY_NODE_SUFFIX = "Test" + FAULTY_NODE_IDX;
+
+ /** */
+ private final String instanceName;
+
+ /** */
+ private final FaultyNodeRole faultyNodeRole;
+
+ /** */
+ private final Map<Integer, Boolean> map = new ConcurrentHashMap<>();
+
+ /**
+ * @param instanceName Ignite node instance name.
+ * @param faultyNodeRole Flag if node is tx coordinator.
+ */
+ private FaultyNodeInterceptor(String instanceName, FaultyNodeRole faultyNodeRole) {
+ this.instanceName = instanceName;
+ this.faultyNodeRole = faultyNodeRole;
+ }
+
+ /** {@inheritDoc} */
+ @Override public @Nullable Integer onBeforePut(Cache.Entry<Integer, Integer> entry, Integer newVal) {
+ if (newVal < 2 * KEYS_NUMBER)
+ return newVal;
+
+ if (instanceName.contains(FAULTY_NODE_SUFFIX)) {
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (!map.containsKey(newVal))
Review Comment:
I added a comment and simplified the logic with a Set instead of Map.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #12178:
URL: https://github.com/apache/ignite/pull/12178#issuecomment-3052285438
## [](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #12178:
URL: https://github.com/apache/ignite/pull/12178#issuecomment-3052315327
## [](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sergey-chugunov-1985 (via GitHub)" <gi...@apache.org>.
sergey-chugunov-1985 commented on code in PR #12178:
URL: https://github.com/apache/ignite/pull/12178#discussion_r2246207802
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #12178:
URL: https://github.com/apache/ignite/pull/12178#issuecomment-3143619082
## [](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_ignite&pullRequest=12178&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_ignite&pullRequest=12178&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_ignite&pullRequest=12178&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_ignite&pullRequest=12178)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] IGNITE-25641 Fix discrepancy with CacheStore in case of an error during tx commit phase [ignite]
Posted by "sergey-chugunov-1985 (via GitHub)" <gi...@apache.org>.
sergey-chugunov-1985 commented on code in PR #12178:
URL: https://github.com/apache/ignite/pull/12178#discussion_r2235948930
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxAdapter.java:
##########
@@ -504,6 +505,46 @@ public void storeEnabled(boolean storeEnabled) {
return storeEnabled() && txState().storeWriteThrough(cctx);
}
+ /**
+ * Clears tx entries if they could be re-read from a read-through {@link CacheStore}.
+ * <p>
+ * This method takes effect only if all Ignite caches involved into transaction configured with CacheStores in
+ * read-through mode.
+ */
+ protected void clearCacheStoreBackedEntries() {
+ if (txState().cacheIds() == null)
+ return;
+
+ GridIntIterator iter = txState().cacheIds().iterator();
+ while (iter.hasNext()) {
+ int cacheId = iter.next();
+
+ if (cctx.cacheContext(cacheId) == null)
+ return;
+
+ if (!cctx.cacheContext(cacheId).config().isReadThrough())
+ return;
+ }
+
+ for (IgniteTxEntry e : writeMap().values()) {
+ try {
+ GridCacheEntryEx entry = e.cached();
+
+ if (e.op() != NOOP) {
+ entry.clear(xidVer, true);
Review Comment:
Fixed.
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
+
+ // check on affected node should pass as the node is alive and cache entries are cleaned up from the cache
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+ }
+ else {
+ // two servers and a client node which is a tx coordinator
+ waitForTopology(3);
+
+ assertTrue("Client node should survive test scenario",
+ G.allGrids()
+ .stream()
+ .filter(ignite -> ((IgniteEx)ignite).context().clientNode())
+ .count() == 1);
+ }
+
+ }
+ else {
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+
+ checkKeysOnHealthyNodes(keysOnFaultyNode);
+ }
+
+ /** */
+ private void fillCache(IgniteCache<Integer, Integer> cache, int numOfKeys) {
+ for (int i = 0; i < numOfKeys; i++) {
+ cache.put(i, i);
+ }
+ }
+
+ /** */
+ private void checkKeysOnFaultyNode(List<Integer> keysToCheck) {
+ IgniteCache<Object, Object> cache = grid(FAULTY_NODE_IDX).cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals(storeStgy.getFromStore(key), cache.get(key));
+ }
+ }
+
+ /** */
+ private void checkKeysOnHealthyNodes(List<Integer> keysToCheck) {
+ for (int i = 0; i < gridCount(); i++) {
+ if (i != FAULTY_NODE_IDX) {
+ IgniteEx ig = grid(i);
+
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
+ assertEquals("Key inconsistent with CacheStore found on node " + i + "; nodeName " + ig.name(),
+ storeStgy.getFromStore(key),
+ cache.get(key));
+ }
+ }
+ }
+ }
+
+ /** */
+ private void updateKeysInTx(Ignite ig, List<Integer> keys) {
+ IgniteCache<Object, Object> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ try (Transaction tx = ig.transactions().txStart()) {
+ for (Integer key : keys) {
Review Comment:
Fixed.
##########
modules/core/src/test/java/org/apache/ignite/cache/store/CacheStoreWithIgniteTxFailureTest.java:
##########
@@ -0,0 +1,322 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cache.store;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.function.IntFunction;
+import javax.cache.Cache;
+import org.apache.ignite.Ignite;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.cache.CacheInterceptor;
+import org.apache.ignite.cache.CacheInterceptorAdapter;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.NearCacheConfiguration;
+import org.apache.ignite.failure.FailureHandler;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.processors.cache.GridCacheAbstractSelfTest;
+import org.apache.ignite.internal.util.typedef.G;
+import org.apache.ignite.transactions.Transaction;
+import org.jetbrains.annotations.Nullable;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+/**
+ * Tests to check scenarios with system failures during transaction commit. Internal system failures are simulated by
+ * {@link CacheInterceptor} custom implementation throwing an exception during final commit phase.
+ */
+@RunWith(Parameterized.class)
+public class CacheStoreWithIgniteTxFailureTest extends GridCacheAbstractSelfTest {
+ /** */
+ private static final int GRID_COUNT = 3;
+
+ /** */
+ private static final int KEYS_NUMBER = 50;
+
+ /** */
+ private static final int FAULTY_NODE_IDX = 1;
+
+ /** */
+ private static final IntFunction<Integer> KEY_UPDATE_FUNCTION = key -> key + KEYS_NUMBER * 3;
+
+ /**
+ * Type of node for keys involved into transaction: primary or backup.
+ */
+ private enum FaultyNodeType {
+ /** */
+ PRIMARY,
+ /** */
+ BACKUP
+ }
+
+ /**
+ * Role of faulty node in transaction management: tx coordinator or regular node.
+ */
+ private enum FaultyNodeRole {
+ /** */
+ REGULAR,
+ /** */
+ TX_COORDINATOR
+ }
+
+ /** */
+ @Parameterized.Parameter
+ public FaultyNodeType faultyNodeType;
+
+ /** */
+ @Parameterized.Parameter(1)
+ public FaultyNodeRole faultyNodeRole;
+
+ /** */
+ @Parameterized.Parameter(2)
+ public boolean withFaulireHandler;
+
+ /** */
+ @Parameterized.Parameter(3)
+ public boolean withNearCacheConfiguration;
+
+ /** */
+ @Parameterized.Parameters(name = "faultyNodeType={0}, faultyNodeRole={1}, withFaulireHandler={2}, withNearCacheConfiguration={3}")
+ public static List<Object[]> parameters() {
+ List<Object[]> params = new ArrayList<>();
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, false});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, false});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, false});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.REGULAR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.REGULAR, false, true});
+
+ params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, false, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, false, true});
+ // TODO https://issues.apache.org/jira/browse/IGNITE-25924
+ // params.add(new Object[] {FaultyNodeType.PRIMARY, FaultyNodeRole.TX_COORDINATOR, true, true});
+ params.add(new Object[] {FaultyNodeType.BACKUP, FaultyNodeRole.TX_COORDINATOR, true, true});
+
+ return params;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTestsStarted() throws Exception {
+ initStoreStrategy();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ // No-op.
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ stopAllGrids();
+
+ storeStgy.resetStore();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int gridCount() {
+ return GRID_COUNT;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected int backups() {
+ return 2;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected FailureHandler getFailureHandler(String igniteInstanceName) {
+ return withFaulireHandler ? new StopNodeFailureHandler() : super.getFailureHandler(igniteInstanceName);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected NearCacheConfiguration nearConfiguration() {
+ return withNearCacheConfiguration ? super.nearConfiguration() : null;
+ }
+
+ /** {@inheritDoc} */
+ @Override protected CacheConfiguration cacheConfiguration(String igniteInstanceName) throws Exception {
+ CacheConfiguration ccfg = super.cacheConfiguration(igniteInstanceName);
+
+ ccfg.setInterceptor(new FaultyNodeInterceptor(igniteInstanceName, faultyNodeRole));
+
+ return ccfg;
+ }
+
+ /**
+ *
+ */
+ @Test
+ public void testSystemExceptionAfterCacheStoreCommit() throws Exception {
+ IgniteEx ig = startGrids(gridCount());
+ IgniteCache<Integer, Integer> cache = ig.cache(DEFAULT_CACHE_NAME);
+
+ fillCache(cache, KEYS_NUMBER);
+
+ int keysType = faultyNodeType == FaultyNodeType.PRIMARY ? 0 : 1;
+
+ List<Integer> keysOnFaultyNode = findKeys(grid(FAULTY_NODE_IDX).localNode(), cache, 5, 0, keysType);
+
+ IgniteEx txCoordinator = faultyNodeRole == FaultyNodeRole.TX_COORDINATOR ? grid(FAULTY_NODE_IDX) : startClientGrid();
+ if (faultyNodeType == FaultyNodeType.PRIMARY)
+ updateKeysInTxWithExceptionCatching(txCoordinator, keysOnFaultyNode);
+ else
+ updateKeysInTx(txCoordinator, keysOnFaultyNode);
+
+ if (withFaulireHandler) { // FH doesn't fail tx coordinator
+ if (faultyNodeRole == FaultyNodeRole.TX_COORDINATOR) {
+ if (faultyNodeType == FaultyNodeType.BACKUP) {
+ waitForTopology(2); // two servers - tx coordinator hosting backup partition fails
+ }
+ else {
+ waitForTopology(3); // three servers - tx coordinator hosting primary partition doesn't fail
+
+ // check on affected node should pass as the node is alive and cache entries are cleaned up from the cache
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+ }
+ else {
+ // two servers and a client node which is a tx coordinator
+ waitForTopology(3);
+
+ assertTrue("Client node should survive test scenario",
+ G.allGrids()
+ .stream()
+ .filter(ignite -> ((IgniteEx)ignite).context().clientNode())
+ .count() == 1);
+ }
+
+ }
+ else {
+ checkKeysOnFaultyNode(keysOnFaultyNode);
+ }
+
+ checkKeysOnHealthyNodes(keysOnFaultyNode);
+ }
+
+ /** */
+ private void fillCache(IgniteCache<Integer, Integer> cache, int numOfKeys) {
+ for (int i = 0; i < numOfKeys; i++) {
+ cache.put(i, i);
+ }
+ }
+
+ /** */
+ private void checkKeysOnFaultyNode(List<Integer> keysToCheck) {
+ IgniteCache<Object, Object> cache = grid(FAULTY_NODE_IDX).cache(DEFAULT_CACHE_NAME);
+
+ for (Integer key : keysToCheck) {
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org