You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2021/08/19 10:56:57 UTC

[GitHub] [ignite] nizhikov opened a new pull request #9345: IGNITE-15117

nizhikov opened a new pull request #9345:
URL: https://github.com/apache/ignite/pull/9345


   Thank you for submitting the pull request to the Apache Ignite.
   
   In order to streamline the review of the contribution 
   we ask you to ensure the following steps have been taken:
   
   ### The Contribution Checklist
   - [ ] There is a single JIRA ticket related to the pull request. 
   - [ ] The web-link to the pull request is attached to the JIRA ticket.
   - [ ] The JIRA ticket has the _Patch Available_ state.
   - [ ] The pull request body describes changes that have been made. 
   The description explains _WHAT_ and _WHY_ was made instead of _HOW_.
   - [ ] The pull request title is treated as the final commit message. 
   The following pattern must be used: `IGNITE-XXXX Change summary` where `XXXX` - number of JIRA issue.
   - [ ] A reviewer has been mentioned through the JIRA comments 
   (see [the Maintainers list](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers)) 
   - [ ] The pull request has been checked by the Teamcity Bot and 
   the `green visa` attached to the JIRA ticket (see [TC.Bot: Check PR](https://mtcga.gridgain.com/prs.html))
   
   ### Notes
   - [How to Contribute](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute)
   - [Coding abbreviation rules](https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules)
   - [Coding Guidelines](https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines)
   - [Apache Ignite Teamcity Bot](https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Teamcity+Bot)
   
   If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com _#ignite_ channel.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803357952



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheUtils.java
##########
@@ -1987,6 +2009,48 @@ public static boolean isPersistenceEnabled(IgniteConfiguration cfg) {
         return isPersistenceEnabled(cfg.getDataStorageConfiguration());
     }
 
+    /**
+     * @return {@code true} if CDC enabled.
+     */
+    public static boolean isCdcEnabled(DataStorageConfiguration cfg) {
+        if (cfg == null)
+            return false;
+
+        DataRegionConfiguration dfltReg = cfg.getDefaultDataRegionConfiguration();
+
+        if (dfltReg != null && dfltReg.isCdcEnabled())
+            return true;
+
+        DataRegionConfiguration[] regCfgs = cfg.getDataRegionConfigurations();
+
+        if (regCfgs == null)
+            return false;
+
+        for (DataRegionConfiguration regCfg : regCfgs) {
+            if (regCfg.isCdcEnabled())
+                return true;
+        }
+
+        return false;
+    }
+
+    /**
+     * @param ctx Group context.
+     * @return {@code True} if {@link DataRecord} should be logged into WAL.
+     */
+    public static boolean isDataRecordsEnabled(CacheGroupContext ctx) {
+        if (ctx.dataRegion() == null)
+            return false;
+
+        if (ctx.persistenceEnabled())
+            return ctx.walEnabled();
+
+        if (ctx.systemCache())
+            return false;
+
+        return ctx.dataRegion().config().isCdcEnabled() && ctx.walEnabled();

Review comment:
       Yes.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824845857



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java
##########
@@ -1204,12 +1207,28 @@ public void onReconnected() {
     }
 
     /**
-     * WAL enabled flag.
+     * Value returned by this method can be changed runtime by the user or during rebalance.
+     *
+     * @return WAL enabled flag.
+     * @see IgniteCluster#disableWal(String)
+     * @see IgniteCluster#enableWal(String)
      */
     public boolean walEnabled() {

Review comment:
       every `walEnabled ` usage (except CheckpointWorkflow) now covered by existing or added `&& grp.persistenceEnabled()`.
   It's good to encapsulate this to `walEnabled ` with renaming of current `walEnabled` to `localWalEnabled`
   Just to minimize length of if's conditions.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824723642



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.BiConsumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)
+            .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(true)));

Review comment:
       Why?
   We may have unexpected behaviour here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r825887292



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java
##########
@@ -1204,12 +1207,28 @@ public void onReconnected() {
     }
 
     /**
-     * WAL enabled flag.
+     * Value returned by this method can be changed runtime by the user or during rebalance.
+     *
+     * @return WAL enabled flag.
+     * @see IgniteCluster#disableWal(String)
+     * @see IgniteCluster#enableWal(String)
      */
     public boolean walEnabled() {

Review comment:
       The goal is to have good design each time.
    Current case seems to be unfinished. And can be finished in way I described.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824529054



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java
##########
@@ -1095,6 +1096,14 @@ public boolean persistenceEnabled() {
         return persistenceEnabled;
     }
 
+    /**
+     * @return Persistence or CDC enabled flag.
+     */
+    public boolean cdcEnabled() {

Review comment:
       renamed to the `logDataRecords`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803444731



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
##########
@@ -355,7 +356,7 @@ private CdcFileLockHolder lockPds() throws IgniteCheckedException {
             new PdsFolderResolver<>(igniteCfg, log, igniteCfg.getConsistentId(), this::tryLock).resolve();
 
         if (settings == null) {
-            throw new IgniteException("Can't find folder to read WAL segments from based on provided configuration! " +
+            throw new IgniteException("Can't find folder to read WAL segments from! " +

Review comment:
       Sound like Yoda style, but it's ok if wish you it to have! :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803443473



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/BinaryMetadataFileStore.java
##########
@@ -89,10 +89,11 @@
     ) throws IgniteCheckedException {
         this.metadataLocCache = metadataLocCache;
         this.ctx = ctx;
-        this.isPersistenceEnabled = CU.isPersistenceEnabled(ctx.config());
+        this.isPersistenceMetadataEnabled = CU.isPersistenceEnabled(ctx.config())

Review comment:
       Not sure I've got an answer :(




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803345674



##########
File path: modules/indexing/src/test/java/org/apache/ignite/internal/cdc/SqlCdcTest.java
##########
@@ -55,14 +60,31 @@
     /** */
     public static final String MSK = "Moscow";
 
+    /** */
+    @Parameterized.Parameter
+    public boolean persistenceEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "persistence={0}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (boolean persistenceEnabled : new boolean[] {true, false})
+            params.add(new Object[] {persistenceEnabled});
+
+        return params;
+    }
+
+
     /** {@inheritDoc} */
     @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
         IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
 
         cfg.setDataStorageConfiguration(new DataStorageConfiguration()
-            .setCdcEnabled(true)
             .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
-            .setDefaultDataRegionConfiguration(new DataRegionConfiguration().setPersistenceEnabled(true)));
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(true)

Review comment:
       Yes. Thanks.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824618748



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.getFieldValue;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    private boolean cdcEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)
+            .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(cdcEnabled)));
+
+        cfg.setConsistentId(igniteInstanceName);
+
+        return cfg;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void beforeTest() throws Exception {
+        stopAllGrids();
+
+        cleanPersistenceDir();
+
+        cdcEnabled = true;
+        persistenceEnabled = false;
+    }
+
+    /** */
+    @Test
+    public void testOnlyDataRecordWritten() throws Exception {
+        IgniteEx ignite = startGrid(0);
+
+        ignite.cluster().state(ClusterState.ACTIVE);
+
+        AtomicInteger cntr = new AtomicInteger();
+
+        // Check only `DataRecords` written in WAL for in-memory cache with CDC enabled.
+        doTestWal(ignite, cache -> {
+            for (int i = 0; i < RECORD_COUNT; i++)
+                cache.put(keyForNode(ignite.affinity(DEFAULT_CACHE_NAME), cntr, ignite.localNode()), i);
+        });
+
+        // Check no WAL written during rebalance.
+        IgniteEx ignite1 = startGrid(1);
+
+        awaitPartitionMapExchange(false, true, null);
+
+        // Can't use `waitForCondition` because if test passed
+        // then no `DataRecords` loged therefore no segment archivation.
+        Thread.sleep(3 * WAL_ARCHIVE_TIMEOUT);
+
+        int walRecCnt = checkDataRecords(ignite1);
+
+        assertEquals(0, walRecCnt);
+
+        // Check `DataRecords` written on second node after rebalance.
+        doTestWal(ignite1, cache -> {

Review comment:
       Good point. Check added.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824621824



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheMapEntry.java
##########
@@ -1213,8 +1213,8 @@ else if (res.resultType() == ResultType.REMOVED_NOT_NULL) {
                 counters.accumulateSizeDelta(cctx.cacheId(), partition(), -1);
             }
 
-            if (cctx.group().persistenceEnabled() && cctx.group().walEnabled()) {
-                logPtr = cctx.shared().wal().log(new MvccDataRecord(new MvccDataEntry(
+            if (cctx.group().cdcEnabled() && cctx.group().walEnabled()) {
+                logPtr = cctx.shared().wal(true).log(new MvccDataRecord(new MvccDataEntry(

Review comment:
       Replaced.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824605837



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.getFieldValue;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    private boolean cdcEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)

Review comment:
       Changed to use default WAL mode for test.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824605453



##########
File path: modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/IndexStoragePageMemoryImplTest.java
##########
@@ -92,15 +93,18 @@
         cctx.add(new GridMetricManager(cctx));
         cctx.add(new GridSystemViewManager(cctx));
 
+        IgniteWriteAheadLogManager walMgr = new NoOpWALManager();
+
         GridCacheSharedContext<Object, Object> sharedCtx = new GridCacheSharedContext<>(
             cctx,
             null,
             null,
             null,
             new NoOpPageStoreManager(),
-            new NoOpWALManager(),
+            walMgr,
+            walMgr,

Review comment:
       Can't do it. Please, take a look at assert:
   
   `assert walMgr == null || walMgr == cdcWalMgr;`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824605224



##########
File path: modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/wal/IgniteWalRebalanceTest.java
##########
@@ -879,7 +879,7 @@ public void testSwitchHistoricalRebalanceToFullWhileIteratingOverWAL() throws Ex
             supplier1 -> {
                 try {
                     // Corrupt wal record in order to fail historical rebalance from supplier1 node.
-                    IgniteWriteAheadLogManager walMgr = supplier1.context().cache().context().wal();
+                    IgniteWriteAheadLogManager walMgr = supplier1.context().cache().context().wal(true);

Review comment:
       Change reverted, this test shouldn't change in scope of this patch.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824838787



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.BiConsumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)
+            .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(true)));
+
+        return cfg;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void beforeTest() throws Exception {
+        stopAllGrids();
+
+        cleanPersistenceDir();
+    }
+
+    /** */
+    @Test
+    public void testOnlyDataRecordWritten() throws Exception {
+        persistenceEnabled = false;
+
+        doTestWal((ignite, cache) -> {
+            for (int i = 0; i < RECORD_COUNT; i++)
+                cache.put(i, i);
+        });
+    }
+
+    /** */
+    @Test
+    public void testWalDisable() throws Exception {
+        persistenceEnabled = true;

Review comment:
       Could you please promt some?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r823824446



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/CdcSelfTest.java
##########
@@ -105,13 +98,11 @@
             cfg.setConsistentId(igniteInstanceName);
 
         cfg.setDataStorageConfiguration(new DataStorageConfiguration()
-            .setCdcEnabled(true)
             .setWalMode(walMode)
             .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
-            .setDefaultDataRegionConfiguration(new DataRegionConfiguration().setPersistenceEnabled(true)));
-
-        if (pageSz != 0)
-            cfg.getDataStorageConfiguration().setPageSize(pageSz);
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(true)));

Review comment:
       Yes. `WalForCdcTest#testWalDisabledIfPersistenceAndCdcDisabled` added.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824553681



##########
File path: modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/BPlusTreePageMemoryImplTest.java
##########
@@ -77,15 +78,18 @@
         cctx.add(new GridMetricManager(cctx));
         cctx.add(new GridSystemViewManager(cctx));
 
+        IgniteWriteAheadLogManager walMgr = new NoOpWALManager();
+
         GridCacheSharedContext<Object, Object> sharedCtx = new GridCacheSharedContext<>(
             cctx,
             null,
             null,
             null,
             new NoOpPageStoreManager(),
-            new NoOpWALManager(),
+            walMgr,
+            walMgr,

Review comment:
       Can't do it. Please, take a look at assert:
   
    `assert walMgr == null || walMgr == cdcWalMgr;`
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r809936554



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/topology/GridDhtLocalPartition.java
##########
@@ -229,7 +229,7 @@ public GridDhtLocalPartition(
             store = grp.offheap().createCacheDataStore(id);
 
             // Log partition creation for further crash recovery purposes.
-            if (grp.walEnabled() && !recovery)
+            if (grp.persistenceEnabled() && grp.walEnabled() && !recovery)

Review comment:
       Yes. `PartitionMetaStateRecord` should be logged only if `persistenceEnabled`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803384144



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FileWriteAheadLogManager.java
##########
@@ -276,9 +281,12 @@
     /** WAL archive directory (including consistent ID as subfolder). */
     private File walArchiveDir;
 
-    /** WAL cdc directory (including consistent ID as subfolder) */
+    /** WAL CDC directory (including consistent ID as subfolder). */
     private File walCdcDir;
 
+    /** CDC enabled flag. */
+    private final boolean cdcEnabled;

Review comment:
       Yes. Thanks. Fixed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r809936743



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/topology/GridDhtLocalPartition.java
##########
@@ -1001,7 +1001,7 @@ protected long clearAll(EvictionContext evictionCtx) throws NodeStoppingExceptio
             // and repeat clearing on applying updates from WAL if the record was read.
             // It's need for atomic cache only. Transactional cache start a rebalance due to outdated counter in this case,
             // because atomic and transactional caches use different partition counters implementation.
-            if (state() == MOVING && !recoveryMode && grp.walEnabled() &&
+            if (state() == MOVING && !recoveryMode && grp.persistenceEnabled() && grp.walEnabled() &&

Review comment:
       Yes. `PartitionClearingStartRecord` should be logged only if `persistenceEnabled`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r809936241



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java
##########
@@ -1041,7 +1041,7 @@ public void destroyPartitionStore(int partId) throws IgniteCheckedException {
 
         int tag = pageMemory.invalidate(grp.groupId(), partId);
 
-        if (grp.walEnabled())
+        if (grp.persistenceEnabled() && grp.walEnabled())

Review comment:
       Yes. `PartitionDestroyRecord` should be logged only if `persistenceEnabled`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r823803463



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/CdcCacheVersionTest.java
##########
@@ -243,6 +260,7 @@ public void testConflictVersionWritten() throws Exception {
             new CacheConfiguration<Integer, User>(DEFAULT_CACHE_NAME)
                 .setCacheMode(cacheMode)
                 .setAtomicityMode(atomicityMode)
+                .setDataRegionName("cdc")

Review comment:
       It's just a self-explaining name.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824619288



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.getFieldValue;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    private boolean cdcEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)
+            .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(cdcEnabled)));
+
+        cfg.setConsistentId(igniteInstanceName);
+
+        return cfg;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void beforeTest() throws Exception {
+        stopAllGrids();
+
+        cleanPersistenceDir();
+
+        cdcEnabled = true;
+        persistenceEnabled = false;
+    }
+
+    /** */
+    @Test
+    public void testOnlyDataRecordWritten() throws Exception {
+        IgniteEx ignite = startGrid(0);
+
+        ignite.cluster().state(ClusterState.ACTIVE);
+
+        AtomicInteger cntr = new AtomicInteger();
+
+        // Check only `DataRecords` written in WAL for in-memory cache with CDC enabled.
+        doTestWal(ignite, cache -> {
+            for (int i = 0; i < RECORD_COUNT; i++)
+                cache.put(keyForNode(ignite.affinity(DEFAULT_CACHE_NAME), cntr, ignite.localNode()), i);
+        });
+
+        // Check no WAL written during rebalance.
+        IgniteEx ignite1 = startGrid(1);
+
+        awaitPartitionMapExchange(false, true, null);
+
+        // Can't use `waitForCondition` because if test passed
+        // then no `DataRecords` loged therefore no segment archivation.
+        Thread.sleep(3 * WAL_ARCHIVE_TIMEOUT);
+
+        int walRecCnt = checkDataRecords(ignite1);
+
+        assertEquals(0, walRecCnt);
+
+        // Check `DataRecords` written on second node after rebalance.
+        doTestWal(ignite1, cache -> {
+            for (int i = 0; i < RECORD_COUNT; i++)
+                cache.put(keyForNode(ignite1.affinity(DEFAULT_CACHE_NAME), cntr, ignite1.localNode()), i);
+        });
+    }
+
+    /** */
+    @Test
+    public void testWalDisable() throws Exception {
+        persistenceEnabled = true;
+
+        IgniteEx ignite = startGrid(0);
+
+        ignite.cluster().state(ClusterState.ACTIVE);
+
+        doTestWal(ignite, cache -> {
+            for (int i = 0; i < RECORD_COUNT / 2; i++)
+                cache.put(i, i);
+
+            ignite.cluster().disableWal(DEFAULT_CACHE_NAME);
+
+            for (int i = 0; i < RECORD_COUNT; i++)
+                cache.put(i, i);
+
+            ignite.cluster().enableWal(DEFAULT_CACHE_NAME);
+
+            for (int i = RECORD_COUNT / 2; i < RECORD_COUNT; i++)
+                cache.put(i, i);
+        });
+    }
+
+    /** */
+    @Test
+    public void testWalDisabledIfPersistenceAndCdcDisabled() throws Exception {
+        persistenceEnabled = false;
+        cdcEnabled = false;
+
+        IgniteEx ignite = startGrid(0);
+
+        ignite.cluster().state(ClusterState.ACTIVE);
+
+        ignite.getOrCreateCache(new CacheConfiguration<Integer, Integer>(DEFAULT_CACHE_NAME)
+                .setCacheMode(mode)
+                .setAtomicityMode(atomicityMode));
+
+        assertNull(ignite.context().cache().context().wal());
+        assertNull(getFieldValue(ignite.context().cache().context(), "cdcWalMgr"));
+    }
+
+    /** */
+    private void doTestWal(IgniteEx ignite, Consumer<IgniteCache<Integer, Integer>> putData) throws Exception {
+        IgniteCache<Integer, Integer> cache = ignite.getOrCreateCache(
+            new CacheConfiguration<Integer, Integer>(DEFAULT_CACHE_NAME)
+                .setCacheMode(mode)
+                .setAtomicityMode(atomicityMode));
+
+        long archiveIdx = ignite.context().cache().context().wal(true).lastArchivedSegment();

Review comment:
       Method added.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] timoninmaxim commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r800645728



##########
File path: modules/indexing/src/test/java/org/apache/ignite/internal/cdc/SqlCdcTest.java
##########
@@ -55,14 +60,31 @@
     /** */
     public static final String MSK = "Moscow";
 
+    /** */
+    @Parameterized.Parameter
+    public boolean persistenceEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "persistence={0}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (boolean persistenceEnabled : new boolean[] {true, false})
+            params.add(new Object[] {persistenceEnabled});
+
+        return params;
+    }
+
+
     /** {@inheritDoc} */
     @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
         IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
 
         cfg.setDataStorageConfiguration(new DataStorageConfiguration()
-            .setCdcEnabled(true)
             .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
-            .setDefaultDataRegionConfiguration(new DataRegionConfiguration().setPersistenceEnabled(true)));
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(true)

Review comment:
       There should be a var `persistenceEnabled` instead of `true`, shouldn't it?

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
##########
@@ -110,7 +111,7 @@
  */
 public class CdcMain implements Runnable {
     /** */
-    public static final String ERR_MSG = "Persistence disabled. Capture Data Change can't run!";
+    public static final String ERR_MSG = "Persistence and CDC disabled. Capture Data Change can't run!";

Review comment:
       Persistance isn't a requirement anymore, is it?

##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/AbstractCdcTest.java
##########
@@ -75,7 +75,7 @@
     public static final int WAL_ARCHIVE_TIMEOUT = 5_000;
 
     /** Keys count. */
-    public static final int KEYS_CNT = 50;
+    public static final int KEYS_CNT = 10;

Review comment:
       Why not 50?

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheUtils.java
##########
@@ -1987,6 +2009,48 @@ public static boolean isPersistenceEnabled(IgniteConfiguration cfg) {
         return isPersistenceEnabled(cfg.getDataStorageConfiguration());
     }
 
+    /**
+     * @return {@code true} if CDC enabled.
+     */
+    public static boolean isCdcEnabled(DataStorageConfiguration cfg) {
+        if (cfg == null)
+            return false;
+
+        DataRegionConfiguration dfltReg = cfg.getDefaultDataRegionConfiguration();
+
+        if (dfltReg != null && dfltReg.isCdcEnabled())
+            return true;
+
+        DataRegionConfiguration[] regCfgs = cfg.getDataRegionConfigurations();
+
+        if (regCfgs == null)
+            return false;
+
+        for (DataRegionConfiguration regCfg : regCfgs) {
+            if (regCfg.isCdcEnabled())
+                return true;
+        }
+
+        return false;
+    }
+
+    /**
+     * @param ctx Group context.
+     * @return {@code True} if {@link DataRecord} should be logged into WAL.
+     */
+    public static boolean isDataRecordsEnabled(CacheGroupContext ctx) {
+        if (ctx.dataRegion() == null)
+            return false;
+
+        if (ctx.persistenceEnabled())
+            return ctx.walEnabled();
+
+        if (ctx.systemCache())
+            return false;
+
+        return ctx.dataRegion().config().isCdcEnabled() && ctx.walEnabled();

Review comment:
       Am I correct that CDC doesn't work without WAL?

##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/CdcCacheVersionTest.java
##########
@@ -81,9 +81,10 @@
         IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
 
         cfg.setDataStorageConfiguration(new DataStorageConfiguration()
-            .setCdcEnabled(true)
             .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
-            .setDefaultDataRegionConfiguration(new DataRegionConfiguration().setPersistenceEnabled(true)));
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(true)

Review comment:
       Can we remove persistence here?

##########
File path: modules/indexing/src/test/java/org/apache/ignite/internal/cdc/SqlCdcTest.java
##########
@@ -55,14 +60,31 @@
     /** */
     public static final String MSK = "Moscow";
 
+    /** */
+    @Parameterized.Parameter
+    public boolean persistenceEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "persistence={0}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();

Review comment:
       full method can be replaced with 
   ```
   return new Object[] {false, true};
   ```
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803796619



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java
##########
@@ -1191,7 +1194,30 @@ public void onReconnected() {
      * WAL enabled flag.
      */
     public boolean walEnabled() {
-        return localWalEnabled && globalWalEnabled;
+        return walEnabled(false);
+    }
+
+    /**
+     * @param forCdc {@code True} record can be consumed by {@link CdcConsumer}.
+     * @return {@code True} if {@link DataRecord} should be logged into WAL.

Review comment:
       Thanks. Fixed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803795406



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FileWriteAheadLogManager.java
##########
@@ -482,9 +487,30 @@ public void setFileIOFactory(FileIOFactory ioFactory) {
 
             serializer = new RecordSerializerFactoryImpl(cctx).createSerializer(serializerVer);
 
-            GridCacheDatabaseSharedManager dbMgr = (GridCacheDatabaseSharedManager)cctx.database();
+            IgniteCacheDatabaseSharedManager dbMgr = cctx.database();
 
-            metrics = dbMgr.persistentStoreMetricsImpl();
+            if (dbMgr instanceof GridCacheDatabaseSharedManager)
+                metrics = ((GridCacheDatabaseSharedManager)dbMgr).persistentStoreMetricsImpl();
+            else {
+                DataStorageConfiguration dsCfg = cctx.gridConfig().getDataStorageConfiguration();

Review comment:
       Good catch! Thanks. Fixed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824782468



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java
##########
@@ -1204,12 +1207,28 @@ public void onReconnected() {
     }
 
     /**
-     * WAL enabled flag.
+     * Value returned by this method can be changed runtime by the user or during rebalance.
+     *
+     * @return WAL enabled flag.
+     * @see IgniteCluster#disableWal(String)
+     * @see IgniteCluster#enableWal(String)
      */
     public boolean walEnabled() {

Review comment:
       > That's in scope, see comment above.
   
   No, it's not.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824851393



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java
##########
@@ -1204,12 +1207,28 @@ public void onReconnected() {
     }
 
     /**
-     * WAL enabled flag.
+     * Value returned by this method can be changed runtime by the user or during rebalance.
+     *
+     * @return WAL enabled flag.
+     * @see IgniteCluster#disableWal(String)
+     * @see IgniteCluster#enableWal(String)
      */
     public boolean walEnabled() {

Review comment:
       > It's good to encapsulate this to walEnabled with renaming of current walEnabled to localWalEnabled
   
   Refactoring is not the goal of this ticket.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r825908823



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java
##########
@@ -1204,12 +1207,28 @@ public void onReconnected() {
     }
 
     /**
-     * WAL enabled flag.
+     * Value returned by this method can be changed runtime by the user or during rebalance.
+     *
+     * @return WAL enabled flag.
+     * @see IgniteCluster#disableWal(String)
+     * @see IgniteCluster#enableWal(String)
      */
     public boolean walEnabled() {

Review comment:
       `walEnabled` method doesn't changed in the current ticket.
   I don't see how renaming relates to the current PR.
   Let's do renaming in other ticket?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r823804909



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.BiConsumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)
+            .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(true)));
+
+        return cfg;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void beforeTest() throws Exception {
+        stopAllGrids();
+
+        cleanPersistenceDir();
+    }
+
+    /** */
+    @Test
+    public void testOnlyDataRecordWritten() throws Exception {
+        persistenceEnabled = false;
+
+        doTestWal((ignite, cache) -> {
+            for (int i = 0; i < RECORD_COUNT; i++)
+                cache.put(i, i);
+        });
+    }
+
+    /** */
+    @Test
+    public void testWalDisable() throws Exception {
+        persistenceEnabled = true;

Review comment:
       We don't support user-initiated WAL disabling for now




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r823804389



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.BiConsumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {

Review comment:
       What name do you have in mind? For me name is pretty clear.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803474543



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheUtils.java
##########
@@ -1987,6 +2009,48 @@ public static boolean isPersistenceEnabled(IgniteConfiguration cfg) {
         return isPersistenceEnabled(cfg.getDataStorageConfiguration());
     }
 
+    /**
+     * @return {@code true} if CDC enabled.
+     */
+    public static boolean isCdcEnabled(DataStorageConfiguration cfg) {
+        if (cfg == null)
+            return false;
+
+        DataRegionConfiguration dfltReg = cfg.getDefaultDataRegionConfiguration();
+
+        if (dfltReg != null && dfltReg.isCdcEnabled())
+            return true;
+
+        DataRegionConfiguration[] regCfgs = cfg.getDataRegionConfigurations();
+
+        if (regCfgs == null)
+            return false;
+
+        for (DataRegionConfiguration regCfg : regCfgs) {
+            if (regCfg.isCdcEnabled())
+                return true;
+        }
+
+        return false;
+    }
+
+    /**
+     * @param ctx Group context.
+     * @return {@code True} if {@link DataRecord} should be logged into WAL.
+     */
+    public static boolean isDataRecordsEnabled(CacheGroupContext ctx) {
+        if (ctx.dataRegion() == null)
+            return false;
+
+        if (ctx.persistenceEnabled())
+            return ctx.walEnabled();
+
+        if (ctx.systemCache())
+            return false;
+
+        return ctx.dataRegion().config().isCdcEnabled() && ctx.walEnabled();

Review comment:
       walEnabled will be false during rebalance




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803348555



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheMapEntry.java
##########
@@ -6961,7 +6961,7 @@ private void versionCheck(@Nullable IgniteBiTuple<Object, Exception> invokeRes)
             if (cctx.offheap().mvccApplyHistoryIfAbsent(this, entryHist)) {
                 updated = true;
 
-                if (!cctx.isNear() && cctx.group().persistenceEnabled() && cctx.group().walEnabled()) {
+                if (!cctx.isNear() && CU.isDataRecordsEnabled(cctx.group())) {

Review comment:
       The whole branch under the `if` is aboud MVCC*DataRecords*.
   So I found `is*DataRecords*Enabled` pretty self explaining name.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r809931641



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/GridDistributedTxRemoteAdapter.java
##########
@@ -800,13 +801,28 @@ else if (op == READ) {
 
                         cctx.mvccCaching().onTxFinished(this, true);
 
+                        boolean walEnabled = false;
+
+                        // Log only there are at least one persistent or cdc enabled group.
                         if (!near() && !F.isEmpty(dataEntries) && cctx.wal() != null) {

Review comment:
       Fixed. Thanks.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r805951924



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/IgniteCacheDatabaseSharedManager.java
##########
@@ -1093,7 +1094,20 @@ public void beforeExchange(GridDhtPartitionsExchangeFuture discoEvt) throws Igni
      * @throws IgniteCheckedException If fails.
      */
     public void startMemoryRestore(GridKernalContext kctx, TimeBag startTimer) throws IgniteCheckedException {

Review comment:
       Restart added to `CdcSelfTest#testReReadWhenStateWasNotStored`.
   Also, please, take a look into ticket [IGNITE-14825](https://issues.apache.org/jira/browse/IGNITE-14825)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r804537537



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/BinaryMetadataFileStore.java
##########
@@ -89,10 +89,11 @@
     ) throws IgniteCheckedException {
         this.metadataLocCache = metadataLocCache;
         this.ctx = ctx;
-        this.isPersistenceEnabled = CU.isPersistenceEnabled(ctx.config());
+        this.isPersistenceMetadataEnabled = CU.isPersistenceEnabled(ctx.config())

Review comment:
       No reason to cast here.
   Use original ~name~ semantic isPersistenceEnabled in place to resolve PersistenceMetadata fate :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824553852



##########
File path: modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/BPlusTreeReuseListPageMemoryImplTest.java
##########
@@ -76,15 +77,18 @@
         cctx.add(new GridMetricManager(cctx));
         cctx.add(new GridSystemViewManager(cctx));
 
+        IgniteWriteAheadLogManager walMgr = new NoOpWALManager();
+
         GridCacheSharedContext<Object, Object> sharedCtx = new GridCacheSharedContext<>(
             cctx,
             null,
             null,
             null,
             new NoOpPageStoreManager(),
-            new NoOpWALManager(),
+            walMgr,
+            walMgr,

Review comment:
       Can't do it. Please, take a look at assert:
   
   `assert walMgr == null || walMgr == cdcWalMgr;`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r826040104



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java
##########
@@ -3116,6 +3118,7 @@ private GridCacheSharedContext createSharedContext(
             verMgr,
             mvccMgr,
             pageStoreMgr,
+            CU.isPersistenceEnabled(ctx.config()) ? walMgr : null,

Review comment:
       Could we encapsulate this inside `GridCacheSharedContext`?
   No need to pass `cdcWalMgr` param, we able to resolve everything from `kernalCtx` and `walMgr` inside the constructor atomically.

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheMapEntry.java
##########
@@ -4410,7 +4410,7 @@ protected WALPointer logMvccUpdate(IgniteInternalTx tx, CacheObject val, long ex
             else
                 op = this.val == null ? GridCacheOperation.CREATE : UPDATE;
 
-            return cctx.shared().wal().log(new MvccDataRecord(new MvccDataEntry(
+            return cctx.shared().wal(cctx.group().cdcEnabled()).log(new MvccDataRecord(new MvccDataEntry(

Review comment:
       Looks like we able to change access point from `cctx.shared().wal()` to `cctx.group().wal()` and resolve `cctx.group().cdcEnabled()` inside the new method. 
   This will minimize the changes just to `shared()` -> `group()`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803359498



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
##########
@@ -355,7 +356,7 @@ private CdcFileLockHolder lockPds() throws IgniteCheckedException {
             new PdsFolderResolver<>(igniteCfg, log, igniteCfg.getConsistentId(), this::tryLock).resolve();
 
         if (settings == null) {
-            throw new IgniteException("Can't find folder to read WAL segments from based on provided configuration! " +
+            throw new IgniteException("Can't find folder to read WAL segments from! " +

Review comment:
       Actually, it simplified right now)
   
   > Can't find the folder to read WAL segments from!
   
   Can you, please, clarify, what is proposed message text?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803360270



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/BinaryMetadataFileStore.java
##########
@@ -89,10 +89,11 @@
     ) throws IgniteCheckedException {
         this.metadataLocCache = metadataLocCache;
         this.ctx = ctx;
-        this.isPersistenceEnabled = CU.isPersistenceEnabled(ctx.config());
+        this.isPersistenceMetadataEnabled = CU.isPersistenceEnabled(ctx.config())

Review comment:
       pesistenceMetadataEnabled in two cases 
   
   1. Persistence enabled.
   2. CDC for in-memory enabled.
   
   I still think rename required.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] timoninmaxim commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803652959



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FileWriteAheadLogManager.java
##########
@@ -482,9 +487,30 @@ public void setFileIOFactory(FileIOFactory ioFactory) {
 
             serializer = new RecordSerializerFactoryImpl(cctx).createSerializer(serializerVer);
 
-            GridCacheDatabaseSharedManager dbMgr = (GridCacheDatabaseSharedManager)cctx.database();
+            IgniteCacheDatabaseSharedManager dbMgr = cctx.database();
 
-            metrics = dbMgr.persistentStoreMetricsImpl();
+            if (dbMgr instanceof GridCacheDatabaseSharedManager)
+                metrics = ((GridCacheDatabaseSharedManager)dbMgr).persistentStoreMetricsImpl();
+            else {
+                DataStorageConfiguration dsCfg = cctx.gridConfig().getDataStorageConfiguration();

Review comment:
       I think we should to move this logic up to `IgniteCacheDatabaseSharedManager` by analogue with the `#startMemoryRestore()` method. WDYT?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803404428



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/CdcCacheVersionTest.java
##########
@@ -81,9 +81,10 @@
         IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
 
         cfg.setDataStorageConfiguration(new DataStorageConfiguration()
-            .setCdcEnabled(true)
             .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
-            .setDefaultDataRegionConfiguration(new DataRegionConfiguration().setPersistenceEnabled(true)));
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(true)

Review comment:
       Yes. Thanks. Fixed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] timoninmaxim commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803659603



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/IgniteCacheDatabaseSharedManager.java
##########
@@ -1093,7 +1094,20 @@ public void beforeExchange(GridDhtPartitionsExchangeFuture discoEvt) throws Igni
      * @throws IgniteCheckedException If fails.
      */
     public void startMemoryRestore(GridKernalContext kctx, TimeBag startTimer) throws IgniteCheckedException {

Review comment:
       Let's add a test for node restart.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r805951924



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/IgniteCacheDatabaseSharedManager.java
##########
@@ -1093,7 +1094,20 @@ public void beforeExchange(GridDhtPartitionsExchangeFuture discoEvt) throws Igni
      * @throws IgniteCheckedException If fails.
      */
     public void startMemoryRestore(GridKernalContext kctx, TimeBag startTimer) throws IgniteCheckedException {

Review comment:
       Restart added to `CdcSelfTest#testReReadWhenStateWasNotStored`.
   Also, please, take a look into ticket [IGNITE-14825](https://issues.apache.org/jira/browse/IGNITE-14825)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824621675



##########
File path: modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/wal/IgniteWalIteratorSwitchSegmentTest.java
##########
@@ -463,6 +464,7 @@ private void checkSwitchReadingSegmentDuringIteration(int serVer) throws Excepti
             null,
             null,
             walMgr,
+            walMgr,

Review comment:
       > we should not test CDC here at all
   
   Yes you are correct.
   Unnecessary changes reverted.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824729410



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.BiConsumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)
+            .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(true)));
+
+        return cfg;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void beforeTest() throws Exception {
+        stopAllGrids();
+
+        cleanPersistenceDir();
+    }
+
+    /** */
+    @Test
+    public void testOnlyDataRecordWritten() throws Exception {
+        persistenceEnabled = false;
+
+        doTestWal((ignite, cache) -> {
+            for (int i = 0; i < RECORD_COUNT; i++)
+                cache.put(i, i);
+        });
+    }
+
+    /** */
+    @Test
+    public void testWalDisable() throws Exception {
+        persistenceEnabled = true;

Review comment:
       Correct, 
   We just must check it's impossible (check we gain an exception on attempt).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824718036



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java
##########
@@ -1041,7 +1041,7 @@ public void destroyPartitionStore(int partId) throws IgniteCheckedException {
 
         int tag = pageMemory.invalidate(grp.groupId(), partId);
 
-        if (grp.walEnabled())
+        if (grp.persistenceEnabled() && grp.walEnabled())

Review comment:
       `cdcEnabled` still may (must) use localWalEnabled(), no problems here




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r823595640



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java
##########
@@ -1204,12 +1207,28 @@ public void onReconnected() {
     }
 
     /**
-     * WAL enabled flag.
+     * Value returned by this method can be changed runtime by the user or during rebalance.
+     *
+     * @return WAL enabled flag.
+     * @see IgniteCluster#disableWal(String)
+     * @see IgniteCluster#enableWal(String)
      */
     public boolean walEnabled() {

Review comment:
       Could this be renamed to `walLoggingEnabled` or `localWalEnabled` or similar?

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/GridDistributedTxRemoteAdapter.java
##########
@@ -801,17 +802,32 @@ else if (op == READ) {
 
                         cctx.mvccCaching().onTxFinished(this, true);
 
-                        if (!near() && !F.isEmpty(dataEntries) && cctx.wal() != null) {
+                        boolean walEnabled = false;
+
+                        // Log only there are at least one persistent or cdc enabled group.
+                        if (!near() && !F.isEmpty(dataEntries) && cctx.cdcWal() != null) {
+                            for (int i = 0; i < dataEntries.size(); i++) {
+                                CacheGroupContext grpCtx = dataEntries.get(i).get2().context().group();
+
+                                if (grpCtx.walOrCdcEnabled()) {
+                                    walEnabled = true;
+
+                                    break;
+                                }
+                            }
+                        }
+
+                        if (walEnabled) {
                             // Set new update counters for data entries received from persisted tx entries.
                             List<DataEntry> entriesWithCounters = dataEntries.stream()
                                 .map(tuple -> tuple.get1().partitionCounter(tuple.get2().updateCounter()))
                                 .collect(Collectors.toList());
 
-                            ptr = cctx.wal().log(new DataRecord(entriesWithCounters));
+                            ptr = cctx.cdcWal().log(new DataRecord(entriesWithCounters));
                         }
 
                         if (ptr != null)
-                            cctx.wal().flush(ptr, false);
+                            cctx.cdcWal().flush(ptr, false);

Review comment:
       This is a `wal or if cdc enabled then also the wal`, not a `cdcWal`.
   `cctx.cdcWal()` looks almost good for me, but we must use ternary operation or something like this to gain wal instance in such code.
   
   `wal = cctx.wal() != null ? cctx.wal() : cctx.cdcWal()`
   and use `wal` variable further at any application code.

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java
##########
@@ -1041,7 +1041,7 @@ public void destroyPartitionStore(int partId) throws IgniteCheckedException {
 
         int tag = pageMemory.invalidate(grp.groupId(), partId);
 
-        if (grp.walEnabled())
+        if (grp.persistenceEnabled() && grp.walEnabled())

Review comment:
       Could we encapsulate this to the `walEnabled` method?
   Looks like now everywhere we use this combination, except `CheckpointWorkflow#fillCacheGroupState` which may be the mistake

##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/CdcCacheVersionTest.java
##########
@@ -139,9 +147,18 @@
         IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
 
         cfg.setDataStorageConfiguration(new DataStorageConfiguration()
-            .setCdcEnabled(true)
             .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
-            .setDefaultDataRegionConfiguration(new DataRegionConfiguration().setPersistenceEnabled(true)));
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled))
+            .setDataRegionConfigurations(
+                new DataRegionConfiguration()
+                    .setName("cdc")
+                    .setPersistenceEnabled(persistenceEnabled)
+                    .setCdcEnabled(true),
+                new DataRegionConfiguration()
+                    .setName("not-cdc")
+                    .setPersistenceEnabled(false)

Review comment:
       Could we check all case matrix? 
   Lets have here `.setPersistenceEnabled(persistenceEnabled)`?

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheSharedContext.java
##########
@@ -764,6 +776,13 @@ public IgniteSnapshotManager snapshotMgr() {
         return walMgr;
     }
 
+    /**
+     * @return Write ahead log manager.
+     */
+    @Nullable public IgniteWriteAheadLogManager cdcWal() {

Review comment:
       We must assert it's not null here, since we accessing (must) this method only after `walOrCdcEnabled()` check

##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.BiConsumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)
+            .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(true)));
+
+        return cfg;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void beforeTest() throws Exception {
+        stopAllGrids();
+
+        cleanPersistenceDir();
+    }
+
+    /** */
+    @Test
+    public void testOnlyDataRecordWritten() throws Exception {
+        persistenceEnabled = false;
+
+        doTestWal((ignite, cache) -> {
+            for (int i = 0; i < RECORD_COUNT; i++)
+                cache.put(i, i);
+        });
+    }
+
+    /** */
+    @Test
+    public void testWalDisable() throws Exception {
+        persistenceEnabled = true;

Review comment:
       Inmem case should be tested as well.

##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/AbstractCdcTest.java
##########
@@ -75,7 +75,7 @@
     public static final int WAL_ARCHIVE_TIMEOUT = 5_000;
 
     /** Keys count. */
-    public static final int KEYS_CNT = 50;
+    public static final int KEYS_CNT = 10;

Review comment:
       How may 50 keys freeze the test?
   What test and why?
   Let's investigate this.

##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/CdcCacheVersionTest.java
##########
@@ -243,6 +260,7 @@ public void testConflictVersionWritten() throws Exception {
             new CacheConfiguration<Integer, User>(DEFAULT_CACHE_NAME)
                 .setCacheMode(cacheMode)
                 .setAtomicityMode(atomicityMode)
+                .setDataRegionName("cdc")

Review comment:
       Is this necessary?
   Is this checked somewhere and should be equal to "cdc"?
   If so, lets use const.

##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.BiConsumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)
+            .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(true)));

Review comment:
       Should we check disabled cdc behaviour as well?

##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/CdcSelfTest.java
##########
@@ -105,13 +98,11 @@
             cfg.setConsistentId(igniteInstanceName);
 
         cfg.setDataStorageConfiguration(new DataStorageConfiguration()
-            .setCdcEnabled(true)
             .setWalMode(walMode)
             .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
-            .setDefaultDataRegionConfiguration(new DataRegionConfiguration().setPersistenceEnabled(true)));
-
-        if (pageSz != 0)
-            cfg.getDataStorageConfiguration().setPageSize(pageSz);
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(true)));

Review comment:
       Should we check that disabled CDC streams nothing as well?

##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.BiConsumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {

Review comment:
       Lets call this test in honor to its aim.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824722058



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java
##########
@@ -1204,12 +1207,28 @@ public void onReconnected() {
     }
 
     /**
-     * WAL enabled flag.
+     * Value returned by this method can be changed runtime by the user or during rebalance.
+     *
+     * @return WAL enabled flag.
+     * @see IgniteCluster#disableWal(String)
+     * @see IgniteCluster#enableWal(String)
      */
     public boolean walEnabled() {

Review comment:
       That's in scope, see comment above.
   To simplify everything (you added `grp.persistenceEnabled()` everywhere, while it should be encapsulated) after the change we must have 3 methods:
   `localWalEnabled`, `walEnabled`, `walOrCdcEnabled` where last two uses the first.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r823813598



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java
##########
@@ -1041,7 +1041,7 @@ public void destroyPartitionStore(int partId) throws IgniteCheckedException {
 
         int tag = pageMemory.invalidate(grp.groupId(), partId);
 
-        if (grp.walEnabled())
+        if (grp.persistenceEnabled() && grp.walEnabled())

Review comment:
       > Could we encapsulate this to the walEnabled method?
   
   no, we can't. `walEnabled` combined with `cdcEnabled` to check if we need to log data records.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r823813972



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java
##########
@@ -1204,12 +1207,28 @@ public void onReconnected() {
     }
 
     /**
-     * WAL enabled flag.
+     * Value returned by this method can be changed runtime by the user or during rebalance.
+     *
+     * @return WAL enabled flag.
+     * @see IgniteCluster#disableWal(String)
+     * @see IgniteCluster#enableWal(String)
      */
     public boolean walEnabled() {

Review comment:
       Yes, but it's an out of scope of current ticket.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824603686



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.getFieldValue;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    private boolean cdcEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)

Review comment:
       Here we test the fact of logging records, but not the WAL modes itself.
   So I think no, we shouldn't test other modes.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r823803697



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.BiConsumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)
+            .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(true)));

Review comment:
       no




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r823803010



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/CdcCacheVersionTest.java
##########
@@ -139,9 +147,18 @@
         IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
 
         cfg.setDataStorageConfiguration(new DataStorageConfiguration()
-            .setCdcEnabled(true)
             .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
-            .setDefaultDataRegionConfiguration(new DataRegionConfiguration().setPersistenceEnabled(true)));
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled))
+            .setDataRegionConfigurations(
+                new DataRegionConfiguration()
+                    .setName("cdc")
+                    .setPersistenceEnabled(persistenceEnabled)
+                    .setCdcEnabled(true),
+                new DataRegionConfiguration()
+                    .setName("not-cdc")
+                    .setPersistenceEnabled(false)

Review comment:
       No we can't.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r823812263



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java
##########
@@ -1041,7 +1041,7 @@ public void destroyPartitionStore(int partId) throws IgniteCheckedException {
 
         int tag = pageMemory.invalidate(grp.groupId(), partId);
 
-        if (grp.walEnabled())
+        if (grp.persistenceEnabled() && grp.walEnabled())

Review comment:
       > CheckpointWorkflow#fillCacheGroupState which may be the mistake
   
   Checkpoint can only happens when persistence enabled.
   So code flow don't get to `CheckpointWorkflow` is persistence disabled.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824748104



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.BiConsumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)
+            .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(true)));

Review comment:
       > We may have unexpected behaviour here.
   
   I doesn't understand this point.
   What do you want to test?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] timoninmaxim commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803643222



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java
##########
@@ -1191,7 +1194,30 @@ public void onReconnected() {
      * WAL enabled flag.
      */
     public boolean walEnabled() {
-        return localWalEnabled && globalWalEnabled;
+        return walEnabled(false);
+    }
+
+    /**
+     * @param forCdc {@code True} record can be consumed by {@link CdcConsumer}.
+     * @return {@code True} if {@link DataRecord} should be logged into WAL.

Review comment:
       It also returns true for every other records if WAL is enabled. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803445850



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheMapEntry.java
##########
@@ -6961,7 +6961,7 @@ private void versionCheck(@Nullable IgniteBiTuple<Object, Exception> invokeRes)
             if (cctx.offheap().mvccApplyHistoryIfAbsent(this, entryHist)) {
                 updated = true;
 
-                if (!cctx.isNear() && cctx.group().persistenceEnabled() && cctx.group().walEnabled()) {
+                if (!cctx.isNear() && CU.isDataRecordsEnabled(cctx.group())) {

Review comment:
       Wal mention should be useful




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803444731



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
##########
@@ -355,7 +356,7 @@ private CdcFileLockHolder lockPds() throws IgniteCheckedException {
             new PdsFolderResolver<>(igniteCfg, log, igniteCfg.getConsistentId(), this::tryLock).resolve();
 
         if (settings == null) {
-            throw new IgniteException("Can't find folder to read WAL segments from based on provided configuration! " +
+            throw new IgniteException("Can't find folder to read WAL segments from! " +

Review comment:
       Sound like Yoda style, but it's ok if wish you it to have :)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803347214



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/AbstractCdcTest.java
##########
@@ -75,7 +75,7 @@
     public static final int WAL_ARCHIVE_TIMEOUT = 5_000;
 
     /** Keys count. */
-    public static final int KEYS_CNT = 50;
+    public static final int KEYS_CNT = 10;

Review comment:
       Because, on the TC with the slow disks some tests take more then one minute that is treated like a failure.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803492537



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/BinaryMetadataFileStore.java
##########
@@ -89,10 +89,11 @@
     ) throws IgniteCheckedException {
         this.metadataLocCache = metadataLocCache;
         this.ctx = ctx;
-        this.isPersistenceEnabled = CU.isPersistenceEnabled(ctx.config());
+        this.isPersistenceMetadataEnabled = CU.isPersistenceEnabled(ctx.config())

Review comment:
       The flag `BinaryMetadataFileStore#isPersistenceEnabled` used to decide - Do we have to store binary metadata on the disk or not?
   
   Before this patch flag always equal to `CU.isPersistenceEnabled` value.
   But, now, it a little different so I rename flag to `isPersistenceMetadataEnabled` to make purpose of the flag more clear.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824854959



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.BiConsumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)
+            .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(true)));
+
+        return cfg;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void beforeTest() throws Exception {
+        stopAllGrids();
+
+        cleanPersistenceDir();
+    }
+
+    /** */
+    @Test
+    public void testOnlyDataRecordWritten() throws Exception {
+        persistenceEnabled = false;
+
+        doTestWal((ignite, cache) -> {
+            for (int i = 0; i < RECORD_COUNT; i++)
+                cache.put(i, i);
+        });
+    }
+
+    /** */
+    @Test
+    public void testWalDisable() throws Exception {
+        persistenceEnabled = true;

Review comment:
       `WalModeChangeAbstractSelfTest#testPersistenceDisabled`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] timoninmaxim commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824425547



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java
##########
@@ -1095,6 +1096,14 @@ public boolean persistenceEnabled() {
         return persistenceEnabled;
     }
 
+    /**
+     * @return Persistence or CDC enabled flag.
+     */
+    public boolean cdcEnabled() {

Review comment:
       Name is misleading, as it can return `true` for disabled CDC.

##########
File path: modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/BPlusTreePageMemoryImplTest.java
##########
@@ -77,15 +78,18 @@
         cctx.add(new GridMetricManager(cctx));
         cctx.add(new GridSystemViewManager(cctx));
 
+        IgniteWriteAheadLogManager walMgr = new NoOpWALManager();
+
         GridCacheSharedContext<Object, Object> sharedCtx = new GridCacheSharedContext<>(
             cctx,
             null,
             null,
             null,
             new NoOpPageStoreManager(),
-            new NoOpWALManager(),
+            walMgr,
+            walMgr,

Review comment:
       I think, we should setup `cdcWalMgr` with `null` for most of tests.

##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/AbstractCdcTest.java
##########
@@ -102,7 +103,7 @@ protected CdcMain createCdc(
 
         cdcCfg.setConsumer(cnsmr);
         cdcCfg.setKeepBinary(keepBinary());
-        cdcCfg.setMetricExporterSpi(metricExporters());

Review comment:
       Now `metricExporters()` is obsolete method, let's remove it.

##########
File path: modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/IndexStoragePageMemoryImplTest.java
##########
@@ -92,15 +93,18 @@
         cctx.add(new GridMetricManager(cctx));
         cctx.add(new GridSystemViewManager(cctx));
 
+        IgniteWriteAheadLogManager walMgr = new NoOpWALManager();
+
         GridCacheSharedContext<Object, Object> sharedCtx = new GridCacheSharedContext<>(
             cctx,
             null,
             null,
             null,
             new NoOpPageStoreManager(),
-            new NoOpWALManager(),
+            walMgr,
+            walMgr,

Review comment:
       I think, we should setup `cdcWalMgr` with `null` for most of tests.

##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.getFieldValue;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    private boolean cdcEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)
+            .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(cdcEnabled)));
+
+        cfg.setConsistentId(igniteInstanceName);
+
+        return cfg;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void beforeTest() throws Exception {
+        stopAllGrids();
+
+        cleanPersistenceDir();
+
+        cdcEnabled = true;
+        persistenceEnabled = false;
+    }
+
+    /** */
+    @Test
+    public void testOnlyDataRecordWritten() throws Exception {
+        IgniteEx ignite = startGrid(0);
+
+        ignite.cluster().state(ClusterState.ACTIVE);
+
+        AtomicInteger cntr = new AtomicInteger();
+
+        // Check only `DataRecords` written in WAL for in-memory cache with CDC enabled.
+        doTestWal(ignite, cache -> {
+            for (int i = 0; i < RECORD_COUNT; i++)
+                cache.put(keyForNode(ignite.affinity(DEFAULT_CACHE_NAME), cntr, ignite.localNode()), i);
+        });
+
+        // Check no WAL written during rebalance.
+        IgniteEx ignite1 = startGrid(1);
+
+        awaitPartitionMapExchange(false, true, null);
+
+        // Can't use `waitForCondition` because if test passed
+        // then no `DataRecords` loged therefore no segment archivation.
+        Thread.sleep(3 * WAL_ARCHIVE_TIMEOUT);
+
+        int walRecCnt = checkDataRecords(ignite1);
+
+        assertEquals(0, walRecCnt);
+
+        // Check `DataRecords` written on second node after rebalance.
+        doTestWal(ignite1, cache -> {
+            for (int i = 0; i < RECORD_COUNT; i++)
+                cache.put(keyForNode(ignite1.affinity(DEFAULT_CACHE_NAME), cntr, ignite1.localNode()), i);
+        });
+    }
+
+    /** */
+    @Test
+    public void testWalDisable() throws Exception {
+        persistenceEnabled = true;
+
+        IgniteEx ignite = startGrid(0);
+
+        ignite.cluster().state(ClusterState.ACTIVE);
+
+        doTestWal(ignite, cache -> {
+            for (int i = 0; i < RECORD_COUNT / 2; i++)
+                cache.put(i, i);
+
+            ignite.cluster().disableWal(DEFAULT_CACHE_NAME);
+
+            for (int i = 0; i < RECORD_COUNT; i++)
+                cache.put(i, i);
+
+            ignite.cluster().enableWal(DEFAULT_CACHE_NAME);
+
+            for (int i = RECORD_COUNT / 2; i < RECORD_COUNT; i++)
+                cache.put(i, i);
+        });
+    }
+
+    /** */
+    @Test
+    public void testWalDisabledIfPersistenceAndCdcDisabled() throws Exception {
+        persistenceEnabled = false;
+        cdcEnabled = false;
+
+        IgniteEx ignite = startGrid(0);
+
+        ignite.cluster().state(ClusterState.ACTIVE);
+
+        ignite.getOrCreateCache(new CacheConfiguration<Integer, Integer>(DEFAULT_CACHE_NAME)
+                .setCacheMode(mode)
+                .setAtomicityMode(atomicityMode));
+
+        assertNull(ignite.context().cache().context().wal());
+        assertNull(getFieldValue(ignite.context().cache().context(), "cdcWalMgr"));
+    }
+
+    /** */
+    private void doTestWal(IgniteEx ignite, Consumer<IgniteCache<Integer, Integer>> putData) throws Exception {
+        IgniteCache<Integer, Integer> cache = ignite.getOrCreateCache(
+            new CacheConfiguration<Integer, Integer>(DEFAULT_CACHE_NAME)
+                .setCacheMode(mode)
+                .setAtomicityMode(atomicityMode));
+
+        long archiveIdx = ignite.context().cache().context().wal(true).lastArchivedSegment();

Review comment:
       WDYT, if we add default method `wal()` --> `wal(false)`?

##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.getFieldValue;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    private boolean cdcEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)

Review comment:
       Should we test it with other modes too?

##########
File path: modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/wal/IgniteWalRebalanceTest.java
##########
@@ -879,7 +879,7 @@ public void testSwitchHistoricalRebalanceToFullWhileIteratingOverWAL() throws Ex
             supplier1 -> {
                 try {
                     // Corrupt wal record in order to fail historical rebalance from supplier1 node.
-                    IgniteWriteAheadLogManager walMgr = supplier1.context().cache().context().wal();
+                    IgniteWriteAheadLogManager walMgr = supplier1.context().cache().context().wal(true);

Review comment:
       Let's parameterize this test, or don't test CDC here at all.

##########
File path: modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/pagemem/BPlusTreeReuseListPageMemoryImplTest.java
##########
@@ -76,15 +77,18 @@
         cctx.add(new GridMetricManager(cctx));
         cctx.add(new GridSystemViewManager(cctx));
 
+        IgniteWriteAheadLogManager walMgr = new NoOpWALManager();
+
         GridCacheSharedContext<Object, Object> sharedCtx = new GridCacheSharedContext<>(
             cctx,
             null,
             null,
             null,
             new NoOpPageStoreManager(),
-            new NoOpWALManager(),
+            walMgr,
+            walMgr,

Review comment:
       I think, we should setup `cdcWalMgr` with `null` for most of tests.

##########
File path: modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/wal/IgniteWalIteratorSwitchSegmentTest.java
##########
@@ -463,6 +464,7 @@ private void checkSwitchReadingSegmentDuringIteration(int serVer) throws Excepti
             null,
             null,
             walMgr,
+            walMgr,

Review comment:
       It looks like this test can be parameterized with CDC or not, WDYT? Or, we should not test CDC here at all.

##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Consumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.getFieldValue;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    private boolean cdcEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)
+            .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(cdcEnabled)));
+
+        cfg.setConsistentId(igniteInstanceName);
+
+        return cfg;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void beforeTest() throws Exception {
+        stopAllGrids();
+
+        cleanPersistenceDir();
+
+        cdcEnabled = true;
+        persistenceEnabled = false;
+    }
+
+    /** */
+    @Test
+    public void testOnlyDataRecordWritten() throws Exception {
+        IgniteEx ignite = startGrid(0);
+
+        ignite.cluster().state(ClusterState.ACTIVE);
+
+        AtomicInteger cntr = new AtomicInteger();
+
+        // Check only `DataRecords` written in WAL for in-memory cache with CDC enabled.
+        doTestWal(ignite, cache -> {
+            for (int i = 0; i < RECORD_COUNT; i++)
+                cache.put(keyForNode(ignite.affinity(DEFAULT_CACHE_NAME), cntr, ignite.localNode()), i);
+        });
+
+        // Check no WAL written during rebalance.
+        IgniteEx ignite1 = startGrid(1);
+
+        awaitPartitionMapExchange(false, true, null);
+
+        // Can't use `waitForCondition` because if test passed
+        // then no `DataRecords` loged therefore no segment archivation.
+        Thread.sleep(3 * WAL_ARCHIVE_TIMEOUT);
+
+        int walRecCnt = checkDataRecords(ignite1);
+
+        assertEquals(0, walRecCnt);
+
+        // Check `DataRecords` written on second node after rebalance.
+        doTestWal(ignite1, cache -> {

Review comment:
       Let's add a check for ignite0 too, we will test that rebalance doesn't break CDC on existing nodes.

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheMapEntry.java
##########
@@ -1213,8 +1213,8 @@ else if (res.resultType() == ResultType.REMOVED_NOT_NULL) {
                 counters.accumulateSizeDelta(cctx.cacheId(), partition(), -1);
             }
 
-            if (cctx.group().persistenceEnabled() && cctx.group().walEnabled()) {
-                logPtr = cctx.shared().wal().log(new MvccDataRecord(new MvccDataEntry(
+            if (cctx.group().cdcEnabled() && cctx.group().walEnabled()) {
+                logPtr = cctx.shared().wal(true).log(new MvccDataRecord(new MvccDataEntry(

Review comment:
       Let's replace `wal(true)` with `wal(cctx.group().cdcEnabled())`. 
   
   Otherwise, it's a little bit misleading to setup the flag `forCdc` to `true` for cases when CDC is actually disabled. 
   
   Also, `cctx.group().cdcEnabled()` should return whether CDC enabled or not, and shouldn't check persistence flag inside.

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/GridDistributedTxRemoteAdapter.java
##########
@@ -801,17 +802,32 @@ else if (op == READ) {
 
                         cctx.mvccCaching().onTxFinished(this, true);
 
-                        if (!near() && !F.isEmpty(dataEntries) && cctx.wal() != null) {
+                        boolean walEnabled = false;
+
+                        // Log only there are at least one persistent or cdc enabled group.
+                        if (!near() && !F.isEmpty(dataEntries) && cctx.wal(true) != null) {
+                            for (int i = 0; i < dataEntries.size(); i++) {
+                                CacheGroupContext grpCtx = dataEntries.get(i).get2().context().group();
+
+                                if (grpCtx.cdcEnabled() && grpCtx.walEnabled()) {

Review comment:
       Previous logic didn't check `grpCtx.walEnabled()`. Why is it needed now?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824747081



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.BiConsumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)
+            .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(true)));
+
+        return cfg;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected void beforeTest() throws Exception {
+        stopAllGrids();
+
+        cleanPersistenceDir();
+    }
+
+    /** */
+    @Test
+    public void testOnlyDataRecordWritten() throws Exception {
+        persistenceEnabled = false;
+
+        doTestWal((ignite, cache) -> {
+            for (int i = 0; i < RECORD_COUNT; i++)
+                cache.put(i, i);
+        });
+    }
+
+    /** */
+    @Test
+    public void testWalDisable() throws Exception {
+        persistenceEnabled = true;

Review comment:
       > We just must check it's impossible (check we gain an exception on attempt).
   
   Such tests already should exists.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824783257



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java
##########
@@ -1204,12 +1207,28 @@ public void onReconnected() {
     }
 
     /**
-     * WAL enabled flag.
+     * Value returned by this method can be changed runtime by the user or during rebalance.
+     *
+     * @return WAL enabled flag.
+     * @see IgniteCluster#disableWal(String)
+     * @see IgniteCluster#enableWal(String)
      */
     public boolean walEnabled() {

Review comment:
       > To simplify everything (you added grp.persistenceEnabled() everywhere, while it should be encapsulated) after the change we must have 3 methods:
   
   Sorry, this can't be done during this ticket.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r823814533



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheSharedContext.java
##########
@@ -764,6 +776,13 @@ public IgniteSnapshotManager snapshotMgr() {
         return walMgr;
     }
 
+    /**
+     * @return Write ahead log manager.
+     */
+    @Nullable public IgniteWriteAheadLogManager cdcWal() {

Review comment:
       assert added.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov merged pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov merged pull request #9345:
URL: https://github.com/apache/ignite/pull/9345


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r826150239



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java
##########
@@ -3116,6 +3118,7 @@ private GridCacheSharedContext createSharedContext(
             verMgr,
             mvccMgr,
             pageStoreMgr,
+            CU.isPersistenceEnabled(ctx.config()) ? walMgr : null,

Review comment:
       Done.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] timoninmaxim commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r808910724



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/GridCacheOffheapManager.java
##########
@@ -1041,7 +1041,7 @@ public void destroyPartitionStore(int partId) throws IgniteCheckedException {
 
         int tag = pageMemory.invalidate(grp.groupId(), partId);
 
-        if (grp.walEnabled())
+        if (grp.persistenceEnabled() && grp.walEnabled())

Review comment:
       Do we need this change, after adding `walOrCdcEnabled()` method?

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/GridDistributedTxRemoteAdapter.java
##########
@@ -800,13 +801,28 @@ else if (op == READ) {
 
                         cctx.mvccCaching().onTxFinished(this, true);
 
+                        boolean walEnabled = false;
+
+                        // Log only there are at least one persistent or cdc enabled group.
                         if (!near() && !F.isEmpty(dataEntries) && cctx.wal() != null) {

Review comment:
       I think we should check `cdcWal()` here too?

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheSharedContext.java
##########
@@ -114,6 +114,9 @@
     /** Write ahead log manager. {@code Null} if persistence is not enabled. */
     @Nullable private IgniteWriteAheadLogManager walMgr;
 
+    /** Write ahead log manager for CDC. {@code Null} if persistence and cdc is not enabled. */

Review comment:
       Let's make it enabled only when CDC is enabled?

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/topology/GridDhtLocalPartition.java
##########
@@ -1001,7 +1001,7 @@ protected long clearAll(EvictionContext evictionCtx) throws NodeStoppingExceptio
             // and repeat clearing on applying updates from WAL if the record was read.
             // It's need for atomic cache only. Transactional cache start a rebalance due to outdated counter in this case,
             // because atomic and transactional caches use different partition counters implementation.
-            if (state() == MOVING && !recoveryMode && grp.walEnabled() &&
+            if (state() == MOVING && !recoveryMode && grp.persistenceEnabled() && grp.walEnabled() &&

Review comment:
       The same, do we need this change now, if now we have separated walOrCdcEnabled() method?

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/topology/GridDhtLocalPartition.java
##########
@@ -229,7 +229,7 @@ public GridDhtLocalPartition(
             store = grp.offheap().createCacheDataStore(id);
 
             // Log partition creation for further crash recovery purposes.
-            if (grp.walEnabled() && !recovery)
+            if (grp.persistenceEnabled() && grp.walEnabled() && !recovery)

Review comment:
       The same, do we need this change now, if now we have separated `walOrCdcEnabled()` method?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824555320



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/GridDistributedTxRemoteAdapter.java
##########
@@ -801,17 +802,32 @@ else if (op == READ) {
 
                         cctx.mvccCaching().onTxFinished(this, true);
 
-                        if (!near() && !F.isEmpty(dataEntries) && cctx.wal() != null) {
+                        boolean walEnabled = false;
+
+                        // Log only there are at least one persistent or cdc enabled group.
+                        if (!near() && !F.isEmpty(dataEntries) && cctx.wal(true) != null) {
+                            for (int i = 0; i < dataEntries.size(); i++) {
+                                CacheGroupContext grpCtx = dataEntries.get(i).get2().context().group();
+
+                                if (grpCtx.cdcEnabled() && grpCtx.walEnabled()) {

Review comment:
       > Previous logic didn't check grpCtx.walEnabled()
   
   It seesm like a bug to me.
   We log `DataRecords` for a group has persistence disabled.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824789939



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java
##########
@@ -1204,12 +1207,28 @@ public void onReconnected() {
     }
 
     /**
-     * WAL enabled flag.
+     * Value returned by this method can be changed runtime by the user or during rebalance.
+     *
+     * @return WAL enabled flag.
+     * @see IgniteCluster#disableWal(String)
+     * @see IgniteCluster#enableWal(String)
      */
     public boolean walEnabled() {

Review comment:
       > you added grp.persistenceEnabled() everywhere
   
   Can you, please, clarify what do you mean by "everywhere"?
   `persistenceEnabled` added 2 times where it should be to exclude unnecessary records in the WAL for in-memory groups.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r823806218



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/GridDistributedTxRemoteAdapter.java
##########
@@ -801,17 +802,32 @@ else if (op == READ) {
 
                         cctx.mvccCaching().onTxFinished(this, true);
 
-                        if (!near() && !F.isEmpty(dataEntries) && cctx.wal() != null) {
+                        boolean walEnabled = false;
+
+                        // Log only there are at least one persistent or cdc enabled group.
+                        if (!near() && !F.isEmpty(dataEntries) && cctx.cdcWal() != null) {
+                            for (int i = 0; i < dataEntries.size(); i++) {
+                                CacheGroupContext grpCtx = dataEntries.get(i).get2().context().group();
+
+                                if (grpCtx.walOrCdcEnabled()) {
+                                    walEnabled = true;
+
+                                    break;
+                                }
+                            }
+                        }
+
+                        if (walEnabled) {
                             // Set new update counters for data entries received from persisted tx entries.
                             List<DataEntry> entriesWithCounters = dataEntries.stream()
                                 .map(tuple -> tuple.get1().partitionCounter(tuple.get2().updateCounter()))
                                 .collect(Collectors.toList());
 
-                            ptr = cctx.wal().log(new DataRecord(entriesWithCounters));
+                            ptr = cctx.cdcWal().log(new DataRecord(entriesWithCounters));
                         }
 
                         if (ptr != null)
-                            cctx.wal().flush(ptr, false);
+                            cctx.cdcWal().flush(ptr, false);

Review comment:
       > wal = cctx.wal() != null ? cctx.wal() : cctx.cdcWal()
   
   OK. Implemented inside`wal(boolean)` method.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r824842162



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/WalForCdcTest.java
##########
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.cdc;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.function.BiConsumer;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.cache.CacheAtomicityMode;
+import org.apache.ignite.cache.CacheMode;
+import org.apache.ignite.cluster.ClusterState;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.configuration.WALMode;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.DataRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cache.CacheAtomicityMode.ATOMIC;
+import static org.apache.ignite.cache.CacheAtomicityMode.TRANSACTIONAL;
+import static org.apache.ignite.cache.CacheMode.PARTITIONED;
+import static org.apache.ignite.cache.CacheMode.REPLICATED;
+import static org.apache.ignite.cdc.CdcSelfTest.WAL_ARCHIVE_TIMEOUT;
+import static org.apache.ignite.testframework.GridTestUtils.waitForCondition;
+
+/** Check only {@link DataRecord} written to the WAL for in-memory cache. */
+@RunWith(Parameterized.class)
+public class WalForCdcTest extends GridCommonAbstractTest {
+    /** */
+    private static final int RECORD_COUNT = 10;
+
+    /** */
+    @Parameterized.Parameter
+    public CacheMode mode;
+
+    /** */
+    @Parameterized.Parameter(1)
+    public CacheAtomicityMode atomicityMode;
+
+    /** */
+    private boolean persistenceEnabled;
+
+    /** */
+    @Parameterized.Parameters(name = "mode={0}, atomicityMode={1}")
+    public static Collection<?> parameters() {
+        List<Object[]> params = new ArrayList<>();
+
+        for (CacheMode mode : Arrays.asList(REPLICATED, PARTITIONED))
+            for (CacheAtomicityMode atomicityMode : Arrays.asList(ATOMIC, TRANSACTIONAL))
+                params.add(new Object[] {mode, atomicityMode});
+
+        return params;
+    }
+
+    /** {@inheritDoc} */
+    @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+        IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
+
+        cfg.setDataStorageConfiguration(new DataStorageConfiguration()
+            .setWalMode(WALMode.FSYNC)
+            .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(persistenceEnabled)
+                .setCdcEnabled(true)));

Review comment:
       You already updated the code to use the field. 
   Problem solved here.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] anton-vinogradov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
anton-vinogradov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r799487590



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheMapEntry.java
##########
@@ -6961,7 +6961,7 @@ private void versionCheck(@Nullable IgniteBiTuple<Object, Exception> invokeRes)
             if (cctx.offheap().mvccApplyHistoryIfAbsent(this, entryHist)) {
                 updated = true;
 
-                if (!cctx.isNear() && cctx.group().persistenceEnabled() && cctx.group().walEnabled()) {
+                if (!cctx.isNear() && CU.isDataRecordsEnabled(cctx.group())) {

Review comment:
       What is `isDataRecordsEnabled`, how to guess it's about WAL?

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
##########
@@ -110,7 +111,7 @@
  */
 public class CdcMain implements Runnable {
     /** */
-    public static final String ERR_MSG = "Persistence disabled. Capture Data Change can't run!";
+    public static final String ERR_MSG = "Persistence and CDC disabled. Capture Data Change can't run!";

Review comment:
       Why `and` while `or`?
   Should we print proper states here?

##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/CdcCacheVersionTest.java
##########
@@ -81,9 +81,10 @@
         IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
 
         cfg.setDataStorageConfiguration(new DataStorageConfiguration()
-            .setCdcEnabled(true)
             .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
-            .setDefaultDataRegionConfiguration(new DataRegionConfiguration().setPersistenceEnabled(true)));
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(true)

Review comment:
       we should have a test with at least 2 data regions, one is with CDC and another is without.

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/binary/BinaryMetadataFileStore.java
##########
@@ -89,10 +89,11 @@
     ) throws IgniteCheckedException {
         this.metadataLocCache = metadataLocCache;
         this.ctx = ctx;
-        this.isPersistenceEnabled = CU.isPersistenceEnabled(ctx.config());
+        this.isPersistenceMetadataEnabled = CU.isPersistenceEnabled(ctx.config())

Review comment:
       Looks like a bad cast. 
   `CU.isPersistenceEnabled` may mean `isPersistenceMetadataEnabled` as well, but we should have special metadata aware method or keep it as is.
   Such cast just adds complexity without pros.

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
##########
@@ -355,7 +356,7 @@ private CdcFileLockHolder lockPds() throws IgniteCheckedException {
             new PdsFolderResolver<>(igniteCfg, log, igniteCfg.getConsistentId(), this::tryLock).resolve();
 
         if (settings == null) {
-            throw new IgniteException("Can't find folder to read WAL segments from based on provided configuration! " +
+            throw new IgniteException("Can't find folder to read WAL segments from! " +

Review comment:
       ... the folder ...
   Could we simplify `folder to read ... from`?

##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FileWriteAheadLogManager.java
##########
@@ -276,9 +281,12 @@
     /** WAL archive directory (including consistent ID as subfolder). */
     private File walArchiveDir;
 
-    /** WAL cdc directory (including consistent ID as subfolder) */
+    /** WAL CDC directory (including consistent ID as subfolder). */
     private File walCdcDir;
 
+    /** CDC enabled flag. */
+    private final boolean cdcEnabled;

Review comment:
       Looks like `walCdcDir != null` means cdcEnabled?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] timoninmaxim commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
timoninmaxim commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803473151



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheUtils.java
##########
@@ -1987,6 +2009,48 @@ public static boolean isPersistenceEnabled(IgniteConfiguration cfg) {
         return isPersistenceEnabled(cfg.getDataStorageConfiguration());
     }
 
+    /**
+     * @return {@code true} if CDC enabled.
+     */
+    public static boolean isCdcEnabled(DataStorageConfiguration cfg) {
+        if (cfg == null)
+            return false;
+
+        DataRegionConfiguration dfltReg = cfg.getDefaultDataRegionConfiguration();
+
+        if (dfltReg != null && dfltReg.isCdcEnabled())
+            return true;
+
+        DataRegionConfiguration[] regCfgs = cfg.getDataRegionConfigurations();
+
+        if (regCfgs == null)
+            return false;
+
+        for (DataRegionConfiguration regCfg : regCfgs) {
+            if (regCfg.isCdcEnabled())
+                return true;
+        }
+
+        return false;
+    }
+
+    /**
+     * @param ctx Group context.
+     * @return {@code True} if {@link DataRecord} should be logged into WAL.
+     */
+    public static boolean isDataRecordsEnabled(CacheGroupContext ctx) {
+        if (ctx.dataRegion() == null)
+            return false;
+
+        if (ctx.persistenceEnabled())
+            return ctx.walEnabled();
+
+        if (ctx.systemCache())
+            return false;
+
+        return ctx.dataRegion().config().isCdcEnabled() && ctx.walEnabled();

Review comment:
       Then no need in `ctx.walEnabled()` here?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803346839



##########
File path: modules/core/src/test/java/org/apache/ignite/cdc/CdcCacheVersionTest.java
##########
@@ -81,9 +81,10 @@
         IgniteConfiguration cfg = super.getConfiguration(igniteInstanceName);
 
         cfg.setDataStorageConfiguration(new DataStorageConfiguration()
-            .setCdcEnabled(true)
             .setWalForceArchiveTimeout(WAL_ARCHIVE_TIMEOUT)
-            .setDefaultDataRegionConfiguration(new DataRegionConfiguration().setPersistenceEnabled(true)));
+            .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+                .setPersistenceEnabled(true)

Review comment:
       It seems better to have persistenceEnabled={true,false) test.
   Fixed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803358725



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/cdc/CdcMain.java
##########
@@ -110,7 +111,7 @@
  */
 public class CdcMain implements Runnable {
     /** */
-    public static final String ERR_MSG = "Persistence disabled. Capture Data Change can't run!";
+    public static final String ERR_MSG = "Persistence and CDC disabled. Capture Data Change can't run!";

Review comment:
       Both, persistence and cdc disabled.
   This mean we can't run CDC application.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [ignite] nizhikov commented on a change in pull request #9345: IGNITE-15117 CDC for in-memory caches

Posted by GitBox <gi...@apache.org>.
nizhikov commented on a change in pull request #9345:
URL: https://github.com/apache/ignite/pull/9345#discussion_r803490043



##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheMapEntry.java
##########
@@ -6961,7 +6961,7 @@ private void versionCheck(@Nullable IgniteBiTuple<Object, Exception> invokeRes)
             if (cctx.offheap().mvccApplyHistoryIfAbsent(this, entryHist)) {
                 updated = true;
 
-                if (!cctx.isNear() && cctx.group().persistenceEnabled() && cctx.group().walEnabled()) {
+                if (!cctx.isNear() && CU.isDataRecordsEnabled(cctx.group())) {

Review comment:
       Right now it's looks like
   
   ```
   if (cctx.group().logDataRecordsToWal()) {
       // log data record to WAL.
   }
   ```
   
   WDYT?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org