You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by "nizhikov (via GitHub)" <gi...@apache.org> on 2023/02/10 11:41:44 UTC
[GitHub] [ignite] nizhikov opened a new pull request, #10533: IGNITE-18721 Disable WAL for index partition during full rebuild
nizhikov opened a new pull request, #10533:
URL: https://github.com/apache/ignite/pull/10533
Thank you for submitting the pull request to the Apache Ignite.
In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:
### The Contribution Checklist
- [ ] There is a single JIRA ticket related to the pull request.
- [ ] The web-link to the pull request is attached to the JIRA ticket.
- [ ] The JIRA ticket has the _Patch Available_ state.
- [ ] The pull request body describes changes that have been made.
The description explains _WHAT_ and _WHY_ was made instead of _HOW_.
- [ ] The pull request title is treated as the final commit message.
The following pattern must be used: `IGNITE-XXXX Change summary` where `XXXX` - number of JIRA issue.
- [ ] A reviewer has been mentioned through the JIRA comments
(see [the Maintainers list](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers))
- [ ] The pull request has been checked by the Teamcity Bot and
the `green visa` attached to the JIRA ticket (see [TC.Bot: Check PR](https://mtcga.gridgain.com/prs.html))
### Notes
- [How to Contribute](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute)
- [Coding abbreviation rules](https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules)
- [Coding Guidelines](https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines)
- [Apache Ignite Teamcity Bot](https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Teamcity+Bot)
If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com _#ignite_ channel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a diff in pull request #10533: IGNITE-18721 Disable WAL for index partition during full rebuild
Posted by "anton-vinogradov (via GitHub)" <gi...@apache.org>.
anton-vinogradov commented on code in PR #10533:
URL: https://github.com/apache/ignite/pull/10533#discussion_r1107114090
##########
modules/core/src/main/java/org/apache/ignite/internal/managers/indexing/IndexesRebuildTask.java:
##########
@@ -86,6 +88,13 @@ public class IndexesRebuildTask {
// Closure prepared, do rebuild.
cctx.kernalContext().query().markAsRebuildNeeded(cctx, true);
+ if (recreate) {
+ cctx.kernalContext().query().markIndexRecreate(cctx);
+
+ if (cctx.group().persistenceEnabled())
Review Comment:
lets encapsulate (or ever get rid of, do we really neet it?) this check
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java:
##########
@@ -1235,6 +1240,16 @@ public boolean globalWalEnabled() {
return globalWalEnabled;
}
+ /** @return {@code True} if WAL for index operations enabled. */
+ public boolean indexWalEnabled() {
+ return idxWalEnabled && globalWalEnabled;
Review Comment:
idxWalEnabled && walEnabled()
##########
modules/core/src/main/java/org/apache/ignite/internal/managers/indexing/IndexesRebuildTask.java:
##########
@@ -101,14 +110,17 @@ public class IndexesRebuildTask {
// Check that the previous rebuild is completed.
assert prevIntRebFut == null;
- cctx.kernalContext().query().onStartRebuildIndexes(cctx);
+ cctx.kernalContext().query().onStartRebuildIndexes(cctx, recreate);
rebuildCacheIdxFut.listen(fut -> {
Throwable err = fut.error();
if (err == null) {
try {
cctx.kernalContext().query().markAsRebuildNeeded(cctx, false);
+
+ if (cctx.group().persistenceEnabled())
Review Comment:
lets encapsulate (or ever get rid of) this check
##########
modules/indexing/src/test/java/org/apache/ignite/internal/processors/database/WalDisabledDuringIndexRecreateTest.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.database;
+
+import java.io.File;
+import java.util.Arrays;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Predicate;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryBasicIdMapper;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.CdcIndexRebuildTest.TestVal;
+import org.apache.ignite.internal.metric.IoStatisticsHolder;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord.RecordType;
+import org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree;
+import org.apache.ignite.internal.processors.cache.persistence.tree.io.PageIO;
+import org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.testframework.ListeningTestLogger;
+import org.apache.ignite.testframework.LogListener;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.cluster.ClusterState.INACTIVE;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_WAL_ARCHIVE_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_WAL_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.UNLIMITED_WAL_ARCHIVE;
+import static org.apache.ignite.internal.pagemem.wal.record.WALRecord.RecordType.BTREE_PAGE_INSERT;
+import static org.apache.ignite.internal.processors.cache.GridCacheUtils.cacheGroupId;
+import static org.apache.ignite.internal.processors.cache.GridCacheUtils.cacheId;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.INDEX_FILE_NAME;
+import static org.apache.ignite.internal.processors.query.schema.management.SortedIndexDescriptorFactory.H2_TREE;
+
+/** */
+@RunWith(Parameterized.class)
+@SuppressWarnings({"resource"})
+public class WalDisabledDuringIndexRecreateTest extends GridCommonAbstractTest {
+ /** Batches count. */
+ public static final int ENTRIES_CNT = 1_000;
+
+ /** */
+ public static final String GRP_NAME = "my-group";
+
+ /** */
+ private ListeningTestLogger testLog;
+
+ /** */
+ @Parameterized.Parameter()
+ public boolean cacheGrps;
+
+ /** */
+ @Parameterized.Parameters(name = "cacheGroups={0}")
+ public static Iterable<Object> data() {
+ return Arrays.asList(true, false);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ return super.getConfiguration(igniteInstanceName)
+ .setGridLogger(testLog)
+ .setConsistentId(igniteInstanceName)
+ .setClusterStateOnStart(INACTIVE)
+ .setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true))
+ .setMaxWalArchiveSize(UNLIMITED_WAL_ARCHIVE));
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+
+ cleanPersistenceDir();
+ }
+
+ /** */
+ @Test
+ public void testDisabled() throws Exception {
+ WALPointer walStartPtr = createDataAndDeleteIndexBin();
+
+ awaitRebuild();
+
+ assertEquals(
+ (Long)3L,
+ countWalRecordsByTypes(wp -> wp.compareTo(walStartPtr) > 0).getOrDefault(BTREE_PAGE_INSERT, 0L)
+ );
+ }
+
+ /** */
+ @Test
+ public void testRestartInCaseOfFailure() throws Exception {
+ WALPointer walStartPtr = createDataAndDeleteIndexBin();
+
+ String treeName = BPlusTree.treeName(
+ new BinaryBasicIdMapper().typeId(TestVal.class.getName()) + "_"
+ + TestVal.class.getSimpleName().toUpperCase() + "_F0_IDX",
+ H2_TREE
+ );
+
+ try {
+ AtomicInteger errCntr = new AtomicInteger(10);
+
+ // Setting up wrapper that will fail on 10 put operation.
+ BPlusTree.testHndWrapper = (tree, hnd) -> {
+ if (!tree.name().equals(treeName))
+ return hnd;
+
+ PageHandler<Object, BPlusTree.Result> delegate = (PageHandler<Object, BPlusTree.Result>)hnd;
+
+ return new PageHandler<BPlusTree.Get, BPlusTree.Result>() {
+ @Override public BPlusTree.Result run(
+ int cacheId,
+ long pageId,
+ long page,
+ long pageAddr,
+ PageIO io,
+ Boolean walPlc,
+ BPlusTree.Get arg,
+ int lvl,
+ IoStatisticsHolder statHolder
+ ) throws IgniteCheckedException {
+ if (arg instanceof BPlusTree.Put && errCntr.decrementAndGet() == 0)
+ throw new IgniteCheckedException("Test error!");
+
+ return delegate.run(cacheId, pageId, page, pageAddr, io, walPlc, arg, lvl, statHolder);
+ }
+
+ @Override public boolean releaseAfterWrite(
+ int cacheId, long pageId, long page, long pageAddr, BPlusTree.Get g, int lvl
+ ) {
+ return g.canRelease(pageId, lvl);
+ }
+ };
+ };
+
+ try {
+ awaitRebuild();
+
+ throw new RuntimeException("Rebuild must not succeed!");
+ }
+ catch (Exception ignore) {
+ BPlusTree.testHndWrapper = null;
+
+ stopAllGrids();
+ }
+
+ LogListener lsnr = LogListener.matches(
+ "Recreate of index.bin don't finish before node stop, index.bin can be inconsistent. " +
+ "Removing it to recreate one more time " +
+ "[grpId=" + cacheGroupId(cacheName(), cacheGroupName())
+ ).build();
+
+ testLog = new ListeningTestLogger(log);
+
+ testLog.registerListener(lsnr);
+
+ awaitRebuild();
+
+ assertTrue(lsnr.check());
+
+ assertEquals(
+ (Long)6L,
+ countWalRecordsByTypes(wp -> wp.compareTo(walStartPtr) > 0).getOrDefault(BTREE_PAGE_INSERT, 0L)
+ );
+ }
+ finally {
+ BPlusTree.testHndWrapper = null;
+
+ testLog = null;
+ }
+ }
+
+ /** */
+ private void awaitRebuild() throws Exception {
+ IgniteEx srv = startGrid(0);
+
+ srv.cluster().state(ACTIVE);
+
+ indexRebuildFuture(srv, cacheId(cacheName())).get(getTestTimeout());
+
+ forceCheckpoint(srv);
+
+ checkIdxFile();
+ }
+
+ /** */
+ private WALPointer createDataAndDeleteIndexBin() throws Exception {
+ IgniteEx srv = startGrid(0);
+
+ srv.cluster().state(ACTIVE);
+
+ for (int i = 0; i < (cacheGrps ? 1 : 3); i++)
Review Comment:
lets have 3 as const, and use it at all places
##########
modules/indexing/src/test/java/org/apache/ignite/internal/processors/database/WalDisabledDuringIndexRecreateTest.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.database;
+
+import java.io.File;
+import java.util.Arrays;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Predicate;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryBasicIdMapper;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.CdcIndexRebuildTest.TestVal;
+import org.apache.ignite.internal.metric.IoStatisticsHolder;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord.RecordType;
+import org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree;
+import org.apache.ignite.internal.processors.cache.persistence.tree.io.PageIO;
+import org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.testframework.ListeningTestLogger;
+import org.apache.ignite.testframework.LogListener;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.cluster.ClusterState.INACTIVE;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_WAL_ARCHIVE_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_WAL_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.UNLIMITED_WAL_ARCHIVE;
+import static org.apache.ignite.internal.pagemem.wal.record.WALRecord.RecordType.BTREE_PAGE_INSERT;
+import static org.apache.ignite.internal.processors.cache.GridCacheUtils.cacheGroupId;
+import static org.apache.ignite.internal.processors.cache.GridCacheUtils.cacheId;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.INDEX_FILE_NAME;
+import static org.apache.ignite.internal.processors.query.schema.management.SortedIndexDescriptorFactory.H2_TREE;
+
+/** */
+@RunWith(Parameterized.class)
+@SuppressWarnings({"resource"})
+public class WalDisabledDuringIndexRecreateTest extends GridCommonAbstractTest {
+ /** Batches count. */
+ public static final int ENTRIES_CNT = 1_000;
+
+ /** */
+ public static final String GRP_NAME = "my-group";
+
+ /** */
+ private ListeningTestLogger testLog;
+
+ /** */
+ @Parameterized.Parameter()
+ public boolean cacheGrps;
+
+ /** */
+ @Parameterized.Parameters(name = "cacheGroups={0}")
+ public static Iterable<Object> data() {
+ return Arrays.asList(true, false);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ return super.getConfiguration(igniteInstanceName)
+ .setGridLogger(testLog)
+ .setConsistentId(igniteInstanceName)
+ .setClusterStateOnStart(INACTIVE)
+ .setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true))
+ .setMaxWalArchiveSize(UNLIMITED_WAL_ARCHIVE));
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+
+ cleanPersistenceDir();
+ }
+
+ /** */
+ @Test
+ public void testDisabled() throws Exception {
+ WALPointer walStartPtr = createDataAndDeleteIndexBin();
+
+ awaitRebuild();
+
+ assertEquals(
+ (Long)3L,
+ countWalRecordsByTypes(wp -> wp.compareTo(walStartPtr) > 0).getOrDefault(BTREE_PAGE_INSERT, 0L)
+ );
+ }
+
+ /** */
+ @Test
+ public void testRestartInCaseOfFailure() throws Exception {
+ WALPointer walStartPtr = createDataAndDeleteIndexBin();
+
+ String treeName = BPlusTree.treeName(
+ new BinaryBasicIdMapper().typeId(TestVal.class.getName()) + "_"
+ + TestVal.class.getSimpleName().toUpperCase() + "_F0_IDX",
+ H2_TREE
+ );
+
+ try {
+ AtomicInteger errCntr = new AtomicInteger(10);
+
+ // Setting up wrapper that will fail on 10 put operation.
+ BPlusTree.testHndWrapper = (tree, hnd) -> {
+ if (!tree.name().equals(treeName))
+ return hnd;
+
+ PageHandler<Object, BPlusTree.Result> delegate = (PageHandler<Object, BPlusTree.Result>)hnd;
+
+ return new PageHandler<BPlusTree.Get, BPlusTree.Result>() {
+ @Override public BPlusTree.Result run(
+ int cacheId,
+ long pageId,
+ long page,
+ long pageAddr,
+ PageIO io,
+ Boolean walPlc,
+ BPlusTree.Get arg,
+ int lvl,
+ IoStatisticsHolder statHolder
+ ) throws IgniteCheckedException {
+ if (arg instanceof BPlusTree.Put && errCntr.decrementAndGet() == 0)
+ throw new IgniteCheckedException("Test error!");
+
+ return delegate.run(cacheId, pageId, page, pageAddr, io, walPlc, arg, lvl, statHolder);
+ }
+
+ @Override public boolean releaseAfterWrite(
+ int cacheId, long pageId, long page, long pageAddr, BPlusTree.Get g, int lvl
+ ) {
+ return g.canRelease(pageId, lvl);
+ }
+ };
+ };
+
+ try {
+ awaitRebuild();
+
+ throw new RuntimeException("Rebuild must not succeed!");
+ }
+ catch (Exception ignore) {
+ BPlusTree.testHndWrapper = null;
+
+ stopAllGrids();
+ }
+
+ LogListener lsnr = LogListener.matches(
+ "Recreate of index.bin don't finish before node stop, index.bin can be inconsistent. " +
+ "Removing it to recreate one more time " +
+ "[grpId=" + cacheGroupId(cacheName(), cacheGroupName())
+ ).build();
+
+ testLog = new ListeningTestLogger(log);
+
+ testLog.registerListener(lsnr);
+
+ awaitRebuild();
+
+ assertTrue(lsnr.check());
+
+ assertEquals(
+ (Long)6L,
+ countWalRecordsByTypes(wp -> wp.compareTo(walStartPtr) > 0).getOrDefault(BTREE_PAGE_INSERT, 0L)
+ );
+ }
+ finally {
+ BPlusTree.testHndWrapper = null;
+
+ testLog = null;
+ }
+ }
+
+ /** */
+ private void awaitRebuild() throws Exception {
+ IgniteEx srv = startGrid(0);
+
+ srv.cluster().state(ACTIVE);
+
+ indexRebuildFuture(srv, cacheId(cacheName())).get(getTestTimeout());
+
+ forceCheckpoint(srv);
Review Comment:
file existence should be quaranted by indexRebuildFuture
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java:
##########
@@ -1263,6 +1278,8 @@ public void localWalEnabled(boolean enabled, boolean persist) {
}
localWalEnabled = enabled;
+
+ idxWalEnabled = localWalEnabled;
Review Comment:
Any reason to do this?
This localWalEnabled && idxWalEnabled (false && true) already gives you a false
##########
modules/indexing/src/test/java/org/apache/ignite/internal/processors/database/WalDisabledDuringIndexRecreateTest.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.database;
+
+import java.io.File;
+import java.util.Arrays;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Predicate;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryBasicIdMapper;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.CdcIndexRebuildTest.TestVal;
+import org.apache.ignite.internal.metric.IoStatisticsHolder;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord.RecordType;
+import org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree;
+import org.apache.ignite.internal.processors.cache.persistence.tree.io.PageIO;
+import org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.testframework.ListeningTestLogger;
+import org.apache.ignite.testframework.LogListener;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.cluster.ClusterState.INACTIVE;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_WAL_ARCHIVE_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_WAL_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.UNLIMITED_WAL_ARCHIVE;
+import static org.apache.ignite.internal.pagemem.wal.record.WALRecord.RecordType.BTREE_PAGE_INSERT;
+import static org.apache.ignite.internal.processors.cache.GridCacheUtils.cacheGroupId;
+import static org.apache.ignite.internal.processors.cache.GridCacheUtils.cacheId;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.INDEX_FILE_NAME;
+import static org.apache.ignite.internal.processors.query.schema.management.SortedIndexDescriptorFactory.H2_TREE;
+
+/** */
+@RunWith(Parameterized.class)
+@SuppressWarnings({"resource"})
+public class WalDisabledDuringIndexRecreateTest extends GridCommonAbstractTest {
+ /** Batches count. */
+ public static final int ENTRIES_CNT = 1_000;
+
+ /** */
+ public static final String GRP_NAME = "my-group";
+
+ /** */
+ private ListeningTestLogger testLog;
+
+ /** */
+ @Parameterized.Parameter()
+ public boolean cacheGrps;
+
+ /** */
+ @Parameterized.Parameters(name = "cacheGroups={0}")
+ public static Iterable<Object> data() {
+ return Arrays.asList(true, false);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ return super.getConfiguration(igniteInstanceName)
+ .setGridLogger(testLog)
+ .setConsistentId(igniteInstanceName)
+ .setClusterStateOnStart(INACTIVE)
+ .setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true))
+ .setMaxWalArchiveSize(UNLIMITED_WAL_ARCHIVE));
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+
+ cleanPersistenceDir();
+ }
+
+ /** */
+ @Test
+ public void testDisabled() throws Exception {
+ WALPointer walStartPtr = createDataAndDeleteIndexBin();
+
+ awaitRebuild();
+
+ assertEquals(
+ (Long)3L,
+ countWalRecordsByTypes(wp -> wp.compareTo(walStartPtr) > 0).getOrDefault(BTREE_PAGE_INSERT, 0L)
+ );
+ }
+
+ /** */
+ @Test
+ public void testRestartInCaseOfFailure() throws Exception {
+ WALPointer walStartPtr = createDataAndDeleteIndexBin();
+
+ String treeName = BPlusTree.treeName(
+ new BinaryBasicIdMapper().typeId(TestVal.class.getName()) + "_"
+ + TestVal.class.getSimpleName().toUpperCase() + "_F0_IDX",
+ H2_TREE
+ );
+
+ try {
+ AtomicInteger errCntr = new AtomicInteger(10);
+
+ // Setting up wrapper that will fail on 10 put operation.
+ BPlusTree.testHndWrapper = (tree, hnd) -> {
+ if (!tree.name().equals(treeName))
+ return hnd;
+
+ PageHandler<Object, BPlusTree.Result> delegate = (PageHandler<Object, BPlusTree.Result>)hnd;
+
+ return new PageHandler<BPlusTree.Get, BPlusTree.Result>() {
+ @Override public BPlusTree.Result run(
+ int cacheId,
+ long pageId,
+ long page,
+ long pageAddr,
+ PageIO io,
+ Boolean walPlc,
+ BPlusTree.Get arg,
+ int lvl,
+ IoStatisticsHolder statHolder
+ ) throws IgniteCheckedException {
+ if (arg instanceof BPlusTree.Put && errCntr.decrementAndGet() == 0)
+ throw new IgniteCheckedException("Test error!");
Review Comment:
Proper explanation required.
Looks like you also must check errCntr.get()==0 at the final?
##########
modules/indexing/src/test/java/org/apache/ignite/internal/processors/database/WalDisabledDuringIndexRecreateTest.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.database;
+
+import java.io.File;
+import java.util.Arrays;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Predicate;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryBasicIdMapper;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.CdcIndexRebuildTest.TestVal;
+import org.apache.ignite.internal.metric.IoStatisticsHolder;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord.RecordType;
+import org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree;
+import org.apache.ignite.internal.processors.cache.persistence.tree.io.PageIO;
+import org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.testframework.ListeningTestLogger;
+import org.apache.ignite.testframework.LogListener;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.cluster.ClusterState.INACTIVE;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_WAL_ARCHIVE_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_WAL_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.UNLIMITED_WAL_ARCHIVE;
+import static org.apache.ignite.internal.pagemem.wal.record.WALRecord.RecordType.BTREE_PAGE_INSERT;
+import static org.apache.ignite.internal.processors.cache.GridCacheUtils.cacheGroupId;
+import static org.apache.ignite.internal.processors.cache.GridCacheUtils.cacheId;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.INDEX_FILE_NAME;
+import static org.apache.ignite.internal.processors.query.schema.management.SortedIndexDescriptorFactory.H2_TREE;
+
+/** */
+@RunWith(Parameterized.class)
+@SuppressWarnings({"resource"})
+public class WalDisabledDuringIndexRecreateTest extends GridCommonAbstractTest {
+ /** Batches count. */
+ public static final int ENTRIES_CNT = 1_000;
+
+ /** */
+ public static final String GRP_NAME = "my-group";
+
+ /** */
+ private ListeningTestLogger testLog;
+
+ /** */
+ @Parameterized.Parameter()
+ public boolean cacheGrps;
+
+ /** */
+ @Parameterized.Parameters(name = "cacheGroups={0}")
+ public static Iterable<Object> data() {
+ return Arrays.asList(true, false);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ return super.getConfiguration(igniteInstanceName)
+ .setGridLogger(testLog)
+ .setConsistentId(igniteInstanceName)
+ .setClusterStateOnStart(INACTIVE)
+ .setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true))
+ .setMaxWalArchiveSize(UNLIMITED_WAL_ARCHIVE));
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+
+ cleanPersistenceDir();
+ }
+
+ /** */
+ @Test
+ public void testDisabled() throws Exception {
+ WALPointer walStartPtr = createDataAndDeleteIndexBin();
+
+ awaitRebuild();
+
+ assertEquals(
+ (Long)3L,
+ countWalRecordsByTypes(wp -> wp.compareTo(walStartPtr) > 0).getOrDefault(BTREE_PAGE_INSERT, 0L)
+ );
+ }
+
+ /** */
+ @Test
+ public void testRestartInCaseOfFailure() throws Exception {
+ WALPointer walStartPtr = createDataAndDeleteIndexBin();
+
+ String treeName = BPlusTree.treeName(
+ new BinaryBasicIdMapper().typeId(TestVal.class.getName()) + "_"
+ + TestVal.class.getSimpleName().toUpperCase() + "_F0_IDX",
+ H2_TREE
+ );
+
+ try {
+ AtomicInteger errCntr = new AtomicInteger(10);
+
+ // Setting up wrapper that will fail on 10 put operation.
+ BPlusTree.testHndWrapper = (tree, hnd) -> {
+ if (!tree.name().equals(treeName))
+ return hnd;
+
+ PageHandler<Object, BPlusTree.Result> delegate = (PageHandler<Object, BPlusTree.Result>)hnd;
+
+ return new PageHandler<BPlusTree.Get, BPlusTree.Result>() {
+ @Override public BPlusTree.Result run(
+ int cacheId,
+ long pageId,
+ long page,
+ long pageAddr,
+ PageIO io,
+ Boolean walPlc,
+ BPlusTree.Get arg,
+ int lvl,
+ IoStatisticsHolder statHolder
+ ) throws IgniteCheckedException {
+ if (arg instanceof BPlusTree.Put && errCntr.decrementAndGet() == 0)
+ throw new IgniteCheckedException("Test error!");
+
+ return delegate.run(cacheId, pageId, page, pageAddr, io, walPlc, arg, lvl, statHolder);
+ }
+
+ @Override public boolean releaseAfterWrite(
+ int cacheId, long pageId, long page, long pageAddr, BPlusTree.Get g, int lvl
+ ) {
+ return g.canRelease(pageId, lvl);
+ }
+ };
+ };
+
+ try {
+ awaitRebuild();
+
+ throw new RuntimeException("Rebuild must not succeed!");
+ }
+ catch (Exception ignore) {
+ BPlusTree.testHndWrapper = null;
+
+ stopAllGrids();
+ }
+
+ LogListener lsnr = LogListener.matches(
+ "Recreate of index.bin don't finish before node stop, index.bin can be inconsistent. " +
+ "Removing it to recreate one more time " +
+ "[grpId=" + cacheGroupId(cacheName(), cacheGroupName())
+ ).build();
+
+ testLog = new ListeningTestLogger(log);
+
+ testLog.registerListener(lsnr);
+
+ awaitRebuild();
+
+ assertTrue(lsnr.check());
+
+ assertEquals(
+ (Long)6L,
+ countWalRecordsByTypes(wp -> wp.compareTo(walStartPtr) > 0).getOrDefault(BTREE_PAGE_INSERT, 0L)
+ );
+ }
+ finally {
+ BPlusTree.testHndWrapper = null;
+
+ testLog = null;
+ }
+ }
+
+ /** */
+ private void awaitRebuild() throws Exception {
+ IgniteEx srv = startGrid(0);
+
+ srv.cluster().state(ACTIVE);
+
+ indexRebuildFuture(srv, cacheId(cacheName())).get(getTestTimeout());
+
+ forceCheckpoint(srv);
+
+ checkIdxFile();
+ }
+
+ /** */
+ private WALPointer createDataAndDeleteIndexBin() throws Exception {
+ IgniteEx srv = startGrid(0);
+
+ srv.cluster().state(ACTIVE);
+
+ for (int i = 0; i < (cacheGrps ? 1 : 3); i++)
+ produceData(srv, cacheName() + (i > 0 ? i + 1 : ""));
Review Comment:
please simplify using `cacheGrps ?`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a diff in pull request #10533: IGNITE-18721 Disable WAL for index partition during full rebuild
Posted by "anton-vinogradov (via GitHub)" <gi...@apache.org>.
anton-vinogradov commented on code in PR #10533:
URL: https://github.com/apache/ignite/pull/10533#discussion_r1107189790
##########
modules/indexing/src/test/java/org/apache/ignite/internal/processors/database/WalDisabledDuringIndexRecreateTest.java:
##########
@@ -0,0 +1,308 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.database;
+
+import java.io.File;
+import java.util.Arrays;
+import java.util.EnumMap;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Predicate;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryBasicIdMapper;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.cdc.CdcIndexRebuildTest.TestVal;
+import org.apache.ignite.internal.metric.IoStatisticsHolder;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord.RecordType;
+import org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree;
+import org.apache.ignite.internal.processors.cache.persistence.tree.io.PageIO;
+import org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.testframework.ListeningTestLogger;
+import org.apache.ignite.testframework.LogListener;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.cluster.ClusterState.INACTIVE;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_WAL_ARCHIVE_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_WAL_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.UNLIMITED_WAL_ARCHIVE;
+import static org.apache.ignite.internal.pagemem.wal.record.WALRecord.RecordType.BTREE_PAGE_INSERT;
+import static org.apache.ignite.internal.processors.cache.GridCacheUtils.cacheGroupId;
+import static org.apache.ignite.internal.processors.cache.GridCacheUtils.cacheId;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.INDEX_FILE_NAME;
+import static org.apache.ignite.internal.processors.query.schema.management.SortedIndexDescriptorFactory.H2_TREE;
+
+/** */
+@RunWith(Parameterized.class)
+@SuppressWarnings({"resource"})
+public class WalDisabledDuringIndexRecreateTest extends GridCommonAbstractTest {
+ /** Batches count. */
+ public static final int ENTRIES_CNT = 1_000;
+
+ /** */
+ public static final String GRP_NAME = "my-group";
+
+ /** */
+ private ListeningTestLogger testLog;
+
+ /** */
+ @Parameterized.Parameter()
+ public boolean cacheGrps;
+
+ /** */
+ @Parameterized.Parameters(name = "cacheGroups={0}")
+ public static Iterable<Object> data() {
+ return Arrays.asList(true, false);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ return super.getConfiguration(igniteInstanceName)
+ .setGridLogger(testLog)
+ .setConsistentId(igniteInstanceName)
+ .setClusterStateOnStart(INACTIVE)
+ .setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true))
+ .setMaxWalArchiveSize(UNLIMITED_WAL_ARCHIVE));
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+
+ cleanPersistenceDir();
+ }
+
+ /** */
+ @Test
+ public void testDisabled() throws Exception {
+ WALPointer walStartPtr = createDataAndDeleteIndexBin();
+
+ awaitRebuild();
+
+ assertEquals(
+ (Long)3L,
+ countWalRecordsByTypes(wp -> wp.compareTo(walStartPtr) > 0).getOrDefault(BTREE_PAGE_INSERT, 0L)
+ );
+ }
+
+ /** */
+ @Test
+ public void testRestartInCaseOfFailure() throws Exception {
+ WALPointer walStartPtr = createDataAndDeleteIndexBin();
+
+ String treeName = BPlusTree.treeName(
+ new BinaryBasicIdMapper().typeId(TestVal.class.getName()) + "_"
+ + TestVal.class.getSimpleName().toUpperCase() + "_F0_IDX",
+ H2_TREE
+ );
+
+ try {
+ AtomicInteger errCntr = new AtomicInteger(10);
+
+ // Setting up wrapper that will fail on 10 put operation.
+ BPlusTree.testHndWrapper = (tree, hnd) -> {
+ if (!tree.name().equals(treeName))
+ return hnd;
+
+ PageHandler<Object, BPlusTree.Result> delegate = (PageHandler<Object, BPlusTree.Result>)hnd;
+
+ return new PageHandler<BPlusTree.Get, BPlusTree.Result>() {
+ @Override public BPlusTree.Result run(
+ int cacheId,
+ long pageId,
+ long page,
+ long pageAddr,
+ PageIO io,
+ Boolean walPlc,
+ BPlusTree.Get arg,
+ int lvl,
+ IoStatisticsHolder statHolder
+ ) throws IgniteCheckedException {
+ if (arg instanceof BPlusTree.Put && errCntr.decrementAndGet() == 0)
+ throw new IgniteCheckedException("Test error!");
Review Comment:
Proper explanation required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a diff in pull request #10533: IGNITE-18721 Disable WAL for index partition during full rebuild
Posted by "anton-vinogradov (via GitHub)" <gi...@apache.org>.
anton-vinogradov commented on code in PR #10533:
URL: https://github.com/apache/ignite/pull/10533#discussion_r1105952708
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/WalStateManager.java:
##########
@@ -1013,12 +1015,15 @@ private WalStateResult awaitCheckpoint(CheckpointProgress cpFut, WalStatePropose
* Checks WAL disabled for cache group.
*
* @param grpId Group id.
+ * @param pageId Page id.
* @return {@code True} if WAL disable for group. {@code False} If not.
*/
- public boolean isDisabled(int grpId) {
+ public boolean isDisabled(int grpId, long pageId) {
CacheGroupContext ctx = cctx.cache().cacheGroup(grpId);
- return ctx != null && !ctx.walEnabled();
+ return ctx != null && (PageIdUtils.partId(pageId) == INDEX_PARTITION
Review Comment:
Lets invert to cause calculation only when !ctx.indexWalEnabled()
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/query/aware/IndexBuildStatusStorage.java:
##########
@@ -109,8 +109,8 @@ public void stop() {
* @param cacheCtx Cache context.
* @see #onFinishRebuildIndexes
*/
- public void onStartRebuildIndexes(GridCacheContext cacheCtx) {
- onStartOperation(cacheCtx, true);
+ public void onStartRebuildIndexes(GridCacheContext cacheCtx, boolean fullRebuild) {
Review Comment:
javadoc param specification missed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a diff in pull request #10533: IGNITE-18721 Disable WAL for index partition during full rebuild
Posted by "anton-vinogradov (via GitHub)" <gi...@apache.org>.
anton-vinogradov commented on code in PR #10533:
URL: https://github.com/apache/ignite/pull/10533#discussion_r1108785879
##########
modules/indexing/src/test/java/org/apache/ignite/internal/processors/database/WalDisabledDuringIndexRecreateTest.java:
##########
@@ -0,0 +1,382 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.database;
+
+import java.io.File;
+import java.util.Arrays;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.function.Predicate;
+import org.apache.ignite.IgniteCache;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.binary.BinaryBasicIdMapper;
+import org.apache.ignite.configuration.CacheConfiguration;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.DataStorageConfiguration;
+import org.apache.ignite.configuration.IgniteConfiguration;
+import org.apache.ignite.failure.StopNodeFailureHandler;
+import org.apache.ignite.internal.IgniteEx;
+import org.apache.ignite.internal.IgniteInternalFuture;
+import org.apache.ignite.internal.cdc.CdcIndexRebuildTest.TestVal;
+import org.apache.ignite.internal.metric.IoStatisticsHolder;
+import org.apache.ignite.internal.pagemem.PageIdUtils;
+import org.apache.ignite.internal.pagemem.wal.WALIterator;
+import org.apache.ignite.internal.pagemem.wal.record.PageSnapshot;
+import org.apache.ignite.internal.pagemem.wal.record.WALRecord;
+import org.apache.ignite.internal.pagemem.wal.record.WalRecordCacheGroupAware;
+import org.apache.ignite.internal.pagemem.wal.record.delta.InsertRecord;
+import org.apache.ignite.internal.pagemem.wal.record.delta.PageDeltaRecord;
+import org.apache.ignite.internal.processors.cache.persistence.IndexStorageImpl;
+import org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree;
+import org.apache.ignite.internal.processors.cache.persistence.tree.io.BPlusIO;
+import org.apache.ignite.internal.processors.cache.persistence.tree.io.PageIO;
+import org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler;
+import org.apache.ignite.internal.processors.cache.persistence.wal.WALPointer;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory;
+import org.apache.ignite.internal.processors.cache.persistence.wal.reader.IgniteWalIteratorFactory.IteratorParametersBuilder;
+import org.apache.ignite.internal.util.typedef.internal.CU;
+import org.apache.ignite.internal.util.typedef.internal.U;
+import org.apache.ignite.lang.IgniteBiTuple;
+import org.apache.ignite.testframework.ListeningTestLogger;
+import org.apache.ignite.testframework.LogListener;
+import org.apache.ignite.testframework.junits.common.GridCommonAbstractTest;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import static org.apache.ignite.cluster.ClusterState.ACTIVE;
+import static org.apache.ignite.cluster.ClusterState.INACTIVE;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_WAL_ARCHIVE_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.DFLT_WAL_PATH;
+import static org.apache.ignite.configuration.DataStorageConfiguration.UNLIMITED_WAL_ARCHIVE;
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+import static org.apache.ignite.internal.processors.cache.GridCacheUtils.cacheGroupId;
+import static org.apache.ignite.internal.processors.cache.GridCacheUtils.cacheId;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.CACHE_GRP_DIR_PREFIX;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.DFLT_STORE_DIR;
+import static org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.INDEX_FILE_NAME;
+import static org.apache.ignite.internal.processors.query.schema.management.SortedIndexDescriptorFactory.H2_TREE;
+
+/** */
+@RunWith(Parameterized.class)
+@SuppressWarnings({"resource"})
+public class WalDisabledDuringIndexRecreateTest extends GridCommonAbstractTest {
+ /** Batches count. */
+ public static final int ENTRIES_CNT = 1_000;
+
+ /** */
+ public static final String GRP_NAME = "my-group";
+
+ /** */
+ public static final int GRP_CACHES_CNT = 3;
+
+ /** */
+ public static final long UNKNOWN_PAGE_ID = -1;
+
+ /** */
+ private ListeningTestLogger testLog;
+
+ /** */
+ @Parameterized.Parameter()
+ public boolean cacheGrps;
+
+ /** */
+ @Parameterized.Parameters(name = "cacheGroups={0}")
+ public static Iterable<Object> data() {
+ return Arrays.asList(true, false);
+ }
+
+ /** {@inheritDoc} */
+ @Override protected IgniteConfiguration getConfiguration(String igniteInstanceName) throws Exception {
+ return super.getConfiguration(igniteInstanceName)
+ .setGridLogger(testLog)
+ .setConsistentId(igniteInstanceName)
+ .setClusterStateOnStart(INACTIVE)
+ .setFailureHandler(new StopNodeFailureHandler())
+ .setDataStorageConfiguration(new DataStorageConfiguration()
+ .setDefaultDataRegionConfiguration(new DataRegionConfiguration()
+ .setPersistenceEnabled(true))
+ .setMaxWalArchiveSize(UNLIMITED_WAL_ARCHIVE));
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void beforeTest() throws Exception {
+ cleanPersistenceDir();
+ }
+
+ /** {@inheritDoc} */
+ @Override protected void afterTest() throws Exception {
+ super.afterTest();
+
+ stopAllGrids();
+
+ cleanPersistenceDir();
+ }
+
+ /** */
+ @Test
+ public void testDisabled() throws Exception {
+ WALPointer walStartPtr = createDataAndDeleteIndexBin();
+
+ testLog = new ListeningTestLogger(log);
+
+ awaitRebuild();
+
+ assertEquals(
+ 0,
+ countWalGrpRecords(wp -> wp.compareTo(walStartPtr) > 0, CU.cacheGroupId(cacheName(), cacheGroupName()))
Review Comment:
`CU.` seems to be redundant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] nizhikov merged pull request #10533: IGNITE-18721 Disable WAL for index partition during full rebuild
Posted by "nizhikov (via GitHub)" <gi...@apache.org>.
nizhikov merged PR #10533:
URL: https://github.com/apache/ignite/pull/10533
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] anton-vinogradov commented on a diff in pull request #10533: IGNITE-18721 Disable WAL for index partition during full rebuild
Posted by "anton-vinogradov (via GitHub)" <gi...@apache.org>.
anton-vinogradov commented on code in PR #10533:
URL: https://github.com/apache/ignite/pull/10533#discussion_r1108515099
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/CacheGroupContext.java:
##########
@@ -1235,6 +1240,16 @@ public boolean globalWalEnabled() {
return globalWalEnabled;
}
+ /** @return {@code True} if WAL for index operations enabled. */
+ public boolean indexWalEnabled() {
+ return idxWalEnabled && walEnabled();
Review Comment:
`return idxWalEnabled;` seems to be enough at the current code
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/file/FilePageStoreManager.java:
##########
@@ -437,17 +439,25 @@ private List<CacheConfiguration> findCacheGroupsWithDisabledWal() {
}
/** {@inheritDoc} */
- @Override public void initialize(int cacheId, int partitions, String workingDir, PageMetrics pageMetrics)
+ @Override public void initialize(int cacheId, int partitions, String cacheName, PageMetrics pageMetrics)
throws IgniteCheckedException {
assert storeWorkDir != null;
if (!idxCacheStores.containsKey(cacheId)) {
+ GridCacheContext<?, ?> cctx = this.cctx.cacheContext(cacheId);
+
+ Collection<String> grpCaches = cctx != null
+ ? cctx.group().caches().stream().map(GridCacheContext::name).collect(Collectors.toSet())
+ : Collections.singleton(cacheName);
Review Comment:
Since you still transfering cacheName, could you encapsulate this to initDir?
BTW, cacheId seems to be enough (no reason to transter cacheName)
##########
modules/core/src/main/java/org/apache/ignite/internal/managers/indexing/IndexesRebuildTask.java:
##########
@@ -121,6 +134,28 @@ public class IndexesRebuildTask {
cctx.kernalContext().query().onFinishRebuildIndexes(cctx);
idxRebuildFuts.remove(cctx.cacheId(), intRebFut);
+
+ if (recreate) {
+ boolean recreateContinues = false;
+
+ for (GridCacheContext<?, ?> cctx0 : cctx.group().caches()) {
+ if (idxRebuildFuts.containsKey(cctx0.cacheId())) {
+ recreateContinues = true;
+
+ break;
+ }
+ }
+
+ if (!recreateContinues) {
+ cctx.group().indexWalEnabled(cctx.group().localWalEnabled());
Review Comment:
Must be cctx.group().indexWalEnabled(true)?
Local wal may be disabled for now, but on enabling you'll still have indexWal disabled?
##########
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/WalStateManager.java:
##########
@@ -1013,12 +1015,14 @@ private WalStateResult awaitCheckpoint(CheckpointProgress cpFut, WalStatePropose
* Checks WAL disabled for cache group.
*
* @param grpId Group id.
+ * @param pageId Page id.
* @return {@code True} if WAL disable for group. {@code False} If not.
*/
- public boolean isDisabled(int grpId) {
+ public boolean isDisabled(int grpId, long pageId) {
CacheGroupContext ctx = cctx.cache().cacheGroup(grpId);
- return ctx != null && !ctx.walEnabled();
+ return ctx != null && (!ctx.walEnabled()
+ || (!ctx.indexWalEnabled() && PageIdUtils.partId(pageId) == INDEX_PARTITION));
Review Comment:
Relocation of `PageIdUtils.partId(pageId) == INDEX_PARTITION))` to the `ctx.indexWalEnabled()` should simplify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@ignite.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org