You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@ignite.apache.org by GitBox <gi...@apache.org> on 2020/08/12 15:58:32 UTC
[GitHub] [ignite] tkalkirill opened a new pull request #8148: IGNITE-13345
tkalkirill opened a new pull request #8148:
URL: https://github.com/apache/ignite/pull/8148
Thank you for submitting the pull request to the Apache Ignite.
In order to streamline the review of the contribution
we ask you to ensure the following steps have been taken:
### The Contribution Checklist
- [ ] There is a single JIRA ticket related to the pull request.
- [ ] The web-link to the pull request is attached to the JIRA ticket.
- [ ] The JIRA ticket has the _Patch Available_ state.
- [ ] The pull request body describes changes that have been made.
The description explains _WHAT_ and _WHY_ was made instead of _HOW_.
- [ ] The pull request title is treated as the final commit message.
The following pattern must be used: `IGNITE-XXXX Change summary` where `XXXX` - number of JIRA issue.
- [ ] A reviewer has been mentioned through the JIRA comments
(see [the Maintainers list](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-ReviewProcessandMaintainers))
- [ ] The pull request has been checked by the Teamcity Bot and
the `green visa` attached to the JIRA ticket (see [TC.Bot: Check PR](https://mtcga.gridgain.com/prs.html))
### Notes
- [How to Contribute](https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute)
- [Coding abbreviation rules](https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules)
- [Coding Guidelines](https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines)
- [Apache Ignite Teamcity Bot](https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+Teamcity+Bot)
If you need any help, please email dev@ignite.apache.org or ask anу advice on http://asf.slack.com _#ignite_ channel.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] asfgit closed pull request #8148: IGNITE-13345
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #8148:
URL: https://github.com/apache/ignite/pull/8148
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] akalash commented on a change in pull request #8148: IGNITE-13345
Posted by GitBox <gi...@apache.org>.
akalash commented on a change in pull request #8148:
URL: https://github.com/apache/ignite/pull/8148#discussion_r473045197
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/warmup/WarmUpStrategy.java
##########
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.warmup;
+
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.configuration.WarmUpConfiguration;
+import org.apache.ignite.internal.GridKernalContext;
+import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
+
+/**
+ * Interface for warming up.
+ */
+public interface WarmUpStrategy<T extends WarmUpConfiguration> {
+ /**
+ * Returns configuration class for mapping to strategy.
+ *
+ * @return Configuration class.
+ */
+ Class<T> configClass();
+
+ /**
+ * Warm up.
+ *
+ * @param kernalCtx Kernal context.
+ * @param cfg Warm-up configuration.
+ * @param region Data region.
+ * @throws IgniteCheckedException if faild.
+ */
+ void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region) throws IgniteCheckedException;
Review comment:
I don't think that the strategy API should contain kernalCtx.
kernalCtx is internal part of the ignite but strategy can be implemented in plugins so it is not so internal.
kernalCtx binds components too tightly which is the opposite of our target - to make components such independent as possible.
coherence |
-- | --
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/warmup/NoOpWarmUpConfiguration.java
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.warmup;
Review comment:
Again, I don't sure about internal
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java
##########
@@ -1274,6 +1276,17 @@ public void start(
startTimer.finishGlobalStage("Init metastore");
+ if (!U.IGNITE_MBEANS_DISABLED) {
+ WarmUpMXBean warmUpMXBean = new WarmUpMXBeanImpl(ctx);
+
+ mBeansMgr.registerMBean(
+ "WarmUp",
+ warmUpMXBean.getClass().getSimpleName(),
+ warmUpMXBean,
+ WarmUpMXBean.class
+ );
+ }
Review comment:
In my opinion, it is better to use IgniteMBeansManager here. You can refactor it such that instead of one method registerAllBeans, it can contain two methods like registerDuringInitPhase and registerAfterNodeStarted
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/warmup/LoadAllWarmUpConfiguration.java
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.warmup;
Review comment:
I don't think that the internal package is the right place for this class because it is a public configuration.
##########
File path: modules/core/src/main/java/org/apache/ignite/configuration/DataRegionConfiguration.java
##########
@@ -142,6 +143,9 @@
*/
private boolean lazyMemoryAllocation = true;
+ /** Warm-up configuration. */
+ @Nullable private transient WarmUpConfiguration warmUpCfg;
Review comment:
Why is it transient?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/warmup/WarmUpMXBeanImpl.java
##########
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.warmup;
+
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteException;
+import org.apache.ignite.internal.GridKernalContext;
+import org.apache.ignite.internal.util.typedef.internal.S;
+import org.apache.ignite.mxbean.WarmUpMXBean;
+
+/**
+ * {@link WarmUpMXBean} implementation.
+ */
+public class WarmUpMXBeanImpl implements WarmUpMXBean {
+ /** Kernal context. */
+ private final GridKernalContext kernalCtx;
Review comment:
Please, avoid context here. This class requires the only cacheProcess so you can use it directly instead of getting it from context.
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java
##########
@@ -5477,6 +5494,96 @@ private void restorePartitionStates(
", partitionsProcessed=" + totalProcessed.get() +
", time=" + (U.currentTimeMillis() - startRestorePart) + "ms]");
}
+
+ /**
+ * Start warming up sequentially for each persist data region.
+ *
+ * @throws IgniteCheckedException If failed.
+ */
+ private void startWarmUp() throws IgniteCheckedException {
+ boolean start = false;
+
+ try {
+ // Collecting custom and default data regions.
+ DataStorageConfiguration dsCfg = ctx.config().getDataStorageConfiguration();
+
+ List<DataRegionConfiguration> regCfgs =
+ new ArrayList<>(asList(dsCfg.getDefaultDataRegionConfiguration()));
+
+ if (nonNull(dsCfg.getDataRegionConfigurations()))
+ regCfgs.addAll(asList(dsCfg.getDataRegionConfigurations()));
+
+ // Warm-up start.
+ Map<Class<? extends WarmUpConfiguration>, WarmUpStrategy> warmUpStrats = CU.warmUpStrategies(ctx);
+
+ WarmUpConfiguration dfltWarmUpCfg = dsCfg.getDefaultWarmUpConfiguration();
+
+ for (DataRegionConfiguration regCfg : regCfgs) {
+ if (stopWarmUp.get())
+ return;
+
+ if (!regCfg.isPersistenceEnabled())
+ continue;
+
+ WarmUpConfiguration warmUpCfg = nonNull(regCfg.getWarmUpConfiguration()) ?
+ regCfg.getWarmUpConfiguration() : dfltWarmUpCfg;
+
+ if (isNull(warmUpCfg))
+ continue;
+
+ WarmUpStrategy warmUpStrat = (curWarmUpStrat = warmUpStrats.get(warmUpCfg.getClass()));
+
+ DataRegion region = sharedCtx.database().dataRegion(regCfg.getName());
+
+ if (!stopWarmUp.get()) {
Review comment:
The current implementation doesn't allow stopping warm-up during the process on the one region which can be a problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] sergey-chugunov-1985 commented on a change in pull request #8148: IGNITE-13345
Posted by GitBox <gi...@apache.org>.
sergey-chugunov-1985 commented on a change in pull request #8148:
URL: https://github.com/apache/ignite/pull/8148#discussion_r476418131
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/warmup/LoadAllWarmUp.java
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.warmup;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Supplier;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.LoadAllWarmUpConfiguration;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.GridCacheProcessor;
+import org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition;
+import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
+import org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryEx;
+import org.apache.ignite.internal.util.tostring.GridToStringExclude;
+import org.apache.ignite.internal.util.typedef.internal.S;
+
+import static java.util.stream.Collectors.toList;
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+
+/**
+ * "Load all" warm-up strategy, which loads pages to persistent data region
+ * until it reaches {@link DataRegionConfiguration#getMaxSize} with priority
+ * to index partitions. Loading occurs sequentially for each cache group,
+ * starting with index partition, and then all others in ascending order.
+ */
+public class LoadAllWarmUp implements WarmUpStrategy<LoadAllWarmUpConfiguration> {
Review comment:
We should keep `Strategy` postfix on all classes implementing WarmUpStrategy: this will increase code readability.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] akalash commented on a change in pull request #8148: IGNITE-13345
Posted by GitBox <gi...@apache.org>.
akalash commented on a change in pull request #8148:
URL: https://github.com/apache/ignite/pull/8148#discussion_r474027432
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/warmup/LoadAllWarmUp.java
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.warmup;
+
+import java.util.Collection;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Supplier;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.LoadAllWarmUpConfiguration;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.GridCacheProcessor;
+import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
+import org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryEx;
+import org.apache.ignite.internal.util.typedef.T2;
+import org.apache.ignite.internal.util.typedef.internal.S;
+
+import static java.util.stream.Collectors.toList;
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+
+/**
+ * "Load all" warm-up strategy, which loads pages to persistent data region
+ * until it reaches {@link DataRegionConfiguration#getMaxSize} with priority
+ * to index partitions. Loading occurs sequentially for each cache group.
+ */
+public class LoadAllWarmUp implements WarmUpStrategy<LoadAllWarmUpConfiguration> {
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /**
+ * Cache group contexts supplier.
+ * Since {@link GridCacheProcessor} starts later.
+ */
+ private final Supplier<Collection<CacheGroupContext>> grpCtxSup;
+
+ /** Stop flag. */
+ private volatile boolean stop;
+
+ /**
+ * Constructor.
+ *
+ * @param log Logger.
+ * @param grpCtxSup Cache group contexts supplier. Since {@link GridCacheProcessor} starts later.
+ */
+ public LoadAllWarmUp(IgniteLogger log, Supplier<Collection<CacheGroupContext>> grpCtxSup) {
+ this.log = log;
+ this.grpCtxSup = grpCtxSup;
+ }
+
+ /** {@inheritDoc} */
+ @Override public Class<LoadAllWarmUpConfiguration> configClass() {
+ return LoadAllWarmUpConfiguration.class;
+ }
+
+ /** {@inheritDoc} */
+ @Override public void warmUp(
+ LoadAllWarmUpConfiguration cfg,
+ DataRegion region
+ ) throws IgniteCheckedException {
+ if (stop)
+ return;
+
+ assert region.config().isPersistenceEnabled();
+
+ Map<CacheGroupContext, T2<Integer, Long>> loadDataInfo = loadDataInfo(region);
+
+ if (log.isInfoEnabled()) {
+ log.info("Order of cache groups loaded into data region [name=" + region.config().getName()
+ + ", partCnt=" + loadDataInfo.values().stream().mapToLong(T2::get1).sum()
+ + ", pageCnt=" + loadDataInfo.values().stream().mapToLong(T2::get2).sum()
+ + ", availablePageCnt=" + availableLoadPageCount(region) + ", grpNames=" +
+ loadDataInfo.keySet().stream().map(CacheGroupContext::cacheOrGroupName).collect(toList()) + ']');
+ }
+
+ long allPartCnt = 0;
+ long allPageCnt = 0;
+
+ for (Map.Entry<CacheGroupContext, T2<Integer, Long>> e : loadDataInfo.entrySet()) {
+ CacheGroupContext grp = e.getKey();
+
+ // Including index partition and its pages.
+ final int partCnt = e.getValue().get1();
+ final long pageCnt = e.getValue().get2();
+
+ if (log.isInfoEnabled()) {
+ log.info("Start warm-up cache group, starting from index [name=" + grp.cacheOrGroupName()
+ + ", partCnt=" + partCnt + ", pageCnt=" + pageCnt + ']');
+ }
+
+ PageMemoryEx pageMemEx = (PageMemoryEx)region.pageMemory();
+
+ long grpPageCnt = 0;
+
+ // Starting with index.
+ for (int i = -1; i < partCnt - 1; i++, allPartCnt++) {
+ long pageId = pageMemEx.partitionMetaPageId(grp.groupId(), (i == -1 ? INDEX_PARTITION : i));
+ long pages = grp.shared().pageStore().pages(grp.groupId(), (i == -1 ? INDEX_PARTITION : i));
+
+ for (long j = 0; j < pages && grpPageCnt < pageCnt; j++, pageId++, grpPageCnt++, allPageCnt++) {
+ if (stop) {
+ if (log.isInfoEnabled()) {
+ log.info("Stop warm-up cache group with loading statistics [name="
+ + grp.cacheOrGroupName() + ", partCnt=" + (i + 1) + "partPageCnt=" + j +
+ ", grpPageCnt=" + grpPageCnt + ", allPartCnt=" + allPartCnt + ", allPageCnt=" +
+ allPageCnt + ']');
+ }
+
+ return;
+ }
+
+ long pagePtr = -1;
+
+ try {
+ pagePtr = pageMemEx.acquirePage(grp.groupId(), pageId);
+ }
+ finally {
+ if (pagePtr != -1)
+ pageMemEx.releasePage(grp.groupId(), pageId, pagePtr);
+ }
+ }
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override public void stop() throws IgniteCheckedException {
+ stop = true;
+ }
+
+ /** {@inheritDoc} */
+ @Override public String toString() {
+ return S.toString(LoadAllWarmUp.class, this);
+ }
+
+ /**
+ * Getting count of pages available for loading into data region.
+ *
+ * @param region Data region.
+ * @return Count(non-negative) of pages available for loading into data region.
+ */
+ protected long availableLoadPageCount(DataRegion region) {
+ long maxSize = region.config().getMaxSize();
+ long curSize = region.pageMemory().loadedPages() * region.pageMemory().systemPageSize();
+
+ return Math.max(0, (maxSize - curSize) / region.pageMemory().systemPageSize());
+ }
+
+ /**
+ * Calculation of cache groups, count of partitions and pages that can load
+ * into data region. Calculation starts and includes an index partition for
+ * each group.
+ *
+ * @param region Data region.
+ * @return Mapping: {cacheGrp -> {partCnt, pageCnt}}.
+ * @throws IgniteCheckedException – if faild.
+ */
+ protected Map<CacheGroupContext, T2<Integer, Long>> loadDataInfo(
+ DataRegion region
+ ) throws IgniteCheckedException {
+ // Get cache groups of data region.
+ List<CacheGroupContext> regionGrps = grpCtxSup.get().stream()
+ .filter(grpCtx -> region.equals(grpCtx.dataRegion())).collect(toList());
+
+ long availableLoadPageCnt = availableLoadPageCount(region);
+
+ // Computing groups, partitions, and pages to load into data region.
+ Map<CacheGroupContext, T2<Integer, Long>> loadableGrps = new LinkedHashMap<>();
+
+ for (int i = 0; i < regionGrps.size() && availableLoadPageCnt > 0; i++) {
+ CacheGroupContext grp = regionGrps.get(i);
+
+ int partCnt = 0;
+ long pageCnt = 0;
+
+ // Index partition in priority.
+ for (int j = -1; j < grp.topology().localPartitions().size() && availableLoadPageCnt > 0; j++, partCnt++) {
+ int partPageCnt = grp.shared().pageStore().pages(grp.groupId(), (j == -1 ? INDEX_PARTITION : j));
Review comment:
Do you sure we don't want to load partitions only after all indexes from all groups would be loaded?
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/warmup/LoadAllWarmUp.java
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.warmup;
+
+import java.util.Collection;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Supplier;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.LoadAllWarmUpConfiguration;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.GridCacheProcessor;
+import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
+import org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryEx;
+import org.apache.ignite.internal.util.typedef.T2;
+import org.apache.ignite.internal.util.typedef.internal.S;
+
+import static java.util.stream.Collectors.toList;
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+
+/**
+ * "Load all" warm-up strategy, which loads pages to persistent data region
+ * until it reaches {@link DataRegionConfiguration#getMaxSize} with priority
+ * to index partitions. Loading occurs sequentially for each cache group.
+ */
+public class LoadAllWarmUp implements WarmUpStrategy<LoadAllWarmUpConfiguration> {
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /**
+ * Cache group contexts supplier.
+ * Since {@link GridCacheProcessor} starts later.
+ */
+ private final Supplier<Collection<CacheGroupContext>> grpCtxSup;
+
+ /** Stop flag. */
+ private volatile boolean stop;
+
+ /**
+ * Constructor.
+ *
+ * @param log Logger.
+ * @param grpCtxSup Cache group contexts supplier. Since {@link GridCacheProcessor} starts later.
+ */
+ public LoadAllWarmUp(IgniteLogger log, Supplier<Collection<CacheGroupContext>> grpCtxSup) {
+ this.log = log;
+ this.grpCtxSup = grpCtxSup;
+ }
+
+ /** {@inheritDoc} */
+ @Override public Class<LoadAllWarmUpConfiguration> configClass() {
+ return LoadAllWarmUpConfiguration.class;
+ }
+
+ /** {@inheritDoc} */
+ @Override public void warmUp(
+ LoadAllWarmUpConfiguration cfg,
+ DataRegion region
+ ) throws IgniteCheckedException {
+ if (stop)
+ return;
+
+ assert region.config().isPersistenceEnabled();
+
+ Map<CacheGroupContext, T2<Integer, Long>> loadDataInfo = loadDataInfo(region);
+
+ if (log.isInfoEnabled()) {
+ log.info("Order of cache groups loaded into data region [name=" + region.config().getName()
+ + ", partCnt=" + loadDataInfo.values().stream().mapToLong(T2::get1).sum()
+ + ", pageCnt=" + loadDataInfo.values().stream().mapToLong(T2::get2).sum()
+ + ", availablePageCnt=" + availableLoadPageCount(region) + ", grpNames=" +
+ loadDataInfo.keySet().stream().map(CacheGroupContext::cacheOrGroupName).collect(toList()) + ']');
+ }
+
+ long allPartCnt = 0;
+ long allPageCnt = 0;
+
+ for (Map.Entry<CacheGroupContext, T2<Integer, Long>> e : loadDataInfo.entrySet()) {
+ CacheGroupContext grp = e.getKey();
+
+ // Including index partition and its pages.
+ final int partCnt = e.getValue().get1();
+ final long pageCnt = e.getValue().get2();
+
+ if (log.isInfoEnabled()) {
+ log.info("Start warm-up cache group, starting from index [name=" + grp.cacheOrGroupName()
+ + ", partCnt=" + partCnt + ", pageCnt=" + pageCnt + ']');
+ }
+
+ PageMemoryEx pageMemEx = (PageMemoryEx)region.pageMemory();
+
+ long grpPageCnt = 0;
+
+ // Starting with index.
+ for (int i = -1; i < partCnt - 1; i++, allPartCnt++) {
+ long pageId = pageMemEx.partitionMetaPageId(grp.groupId(), (i == -1 ? INDEX_PARTITION : i));
Review comment:
I didn't get how do you handle gaps in partitions? e.g. you have local partitions 1,3,5 - it is 3 in total but you can't iterate from 1 to 3
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/warmup/LoadAllWarmUp.java
##########
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.ignite.internal.processors.cache.warmup;
+
+import java.util.Collection;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Supplier;
+import org.apache.ignite.IgniteCheckedException;
+import org.apache.ignite.IgniteLogger;
+import org.apache.ignite.configuration.DataRegionConfiguration;
+import org.apache.ignite.configuration.LoadAllWarmUpConfiguration;
+import org.apache.ignite.internal.processors.cache.CacheGroupContext;
+import org.apache.ignite.internal.processors.cache.GridCacheProcessor;
+import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
+import org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryEx;
+import org.apache.ignite.internal.util.typedef.T2;
+import org.apache.ignite.internal.util.typedef.internal.S;
+
+import static java.util.stream.Collectors.toList;
+import static org.apache.ignite.internal.pagemem.PageIdAllocator.INDEX_PARTITION;
+
+/**
+ * "Load all" warm-up strategy, which loads pages to persistent data region
+ * until it reaches {@link DataRegionConfiguration#getMaxSize} with priority
+ * to index partitions. Loading occurs sequentially for each cache group.
+ */
+public class LoadAllWarmUp implements WarmUpStrategy<LoadAllWarmUpConfiguration> {
+ /** Logger. */
+ private final IgniteLogger log;
+
+ /**
+ * Cache group contexts supplier.
+ * Since {@link GridCacheProcessor} starts later.
+ */
+ private final Supplier<Collection<CacheGroupContext>> grpCtxSup;
+
+ /** Stop flag. */
+ private volatile boolean stop;
+
+ /**
+ * Constructor.
+ *
+ * @param log Logger.
+ * @param grpCtxSup Cache group contexts supplier. Since {@link GridCacheProcessor} starts later.
+ */
+ public LoadAllWarmUp(IgniteLogger log, Supplier<Collection<CacheGroupContext>> grpCtxSup) {
+ this.log = log;
+ this.grpCtxSup = grpCtxSup;
+ }
+
+ /** {@inheritDoc} */
+ @Override public Class<LoadAllWarmUpConfiguration> configClass() {
+ return LoadAllWarmUpConfiguration.class;
+ }
+
+ /** {@inheritDoc} */
+ @Override public void warmUp(
+ LoadAllWarmUpConfiguration cfg,
+ DataRegion region
+ ) throws IgniteCheckedException {
+ if (stop)
+ return;
+
+ assert region.config().isPersistenceEnabled();
+
+ Map<CacheGroupContext, T2<Integer, Long>> loadDataInfo = loadDataInfo(region);
+
+ if (log.isInfoEnabled()) {
+ log.info("Order of cache groups loaded into data region [name=" + region.config().getName()
+ + ", partCnt=" + loadDataInfo.values().stream().mapToLong(T2::get1).sum()
+ + ", pageCnt=" + loadDataInfo.values().stream().mapToLong(T2::get2).sum()
+ + ", availablePageCnt=" + availableLoadPageCount(region) + ", grpNames=" +
+ loadDataInfo.keySet().stream().map(CacheGroupContext::cacheOrGroupName).collect(toList()) + ']');
+ }
+
+ long allPartCnt = 0;
+ long allPageCnt = 0;
+
+ for (Map.Entry<CacheGroupContext, T2<Integer, Long>> e : loadDataInfo.entrySet()) {
+ CacheGroupContext grp = e.getKey();
+
+ // Including index partition and its pages.
+ final int partCnt = e.getValue().get1();
+ final long pageCnt = e.getValue().get2();
+
+ if (log.isInfoEnabled()) {
+ log.info("Start warm-up cache group, starting from index [name=" + grp.cacheOrGroupName()
+ + ", partCnt=" + partCnt + ", pageCnt=" + pageCnt + ']');
+ }
+
+ PageMemoryEx pageMemEx = (PageMemoryEx)region.pageMemory();
+
+ long grpPageCnt = 0;
+
+ // Starting with index.
+ for (int i = -1; i < partCnt - 1; i++, allPartCnt++) {
+ long pageId = pageMemEx.partitionMetaPageId(grp.groupId(), (i == -1 ? INDEX_PARTITION : i));
+ long pages = grp.shared().pageStore().pages(grp.groupId(), (i == -1 ? INDEX_PARTITION : i));
+
+ for (long j = 0; j < pages && grpPageCnt < pageCnt; j++, pageId++, grpPageCnt++, allPageCnt++) {
+ if (stop) {
+ if (log.isInfoEnabled()) {
+ log.info("Stop warm-up cache group with loading statistics [name="
+ + grp.cacheOrGroupName() + ", partCnt=" + (i + 1) + "partPageCnt=" + j +
+ ", grpPageCnt=" + grpPageCnt + ", allPartCnt=" + allPartCnt + ", allPageCnt=" +
+ allPageCnt + ']');
+ }
+
+ return;
+ }
+
+ long pagePtr = -1;
+
+ try {
+ pagePtr = pageMemEx.acquirePage(grp.groupId(), pageId);
+ }
+ finally {
+ if (pagePtr != -1)
+ pageMemEx.releasePage(grp.groupId(), pageId, pagePtr);
+ }
+ }
+ }
+ }
+ }
+
+ /** {@inheritDoc} */
+ @Override public void stop() throws IgniteCheckedException {
+ stop = true;
+ }
+
+ /** {@inheritDoc} */
+ @Override public String toString() {
+ return S.toString(LoadAllWarmUp.class, this);
+ }
+
+ /**
+ * Getting count of pages available for loading into data region.
+ *
+ * @param region Data region.
+ * @return Count(non-negative) of pages available for loading into data region.
+ */
+ protected long availableLoadPageCount(DataRegion region) {
+ long maxSize = region.config().getMaxSize();
+ long curSize = region.pageMemory().loadedPages() * region.pageMemory().systemPageSize();
+
+ return Math.max(0, (maxSize - curSize) / region.pageMemory().systemPageSize());
+ }
+
+ /**
+ * Calculation of cache groups, count of partitions and pages that can load
+ * into data region. Calculation starts and includes an index partition for
+ * each group.
+ *
+ * @param region Data region.
+ * @return Mapping: {cacheGrp -> {partCnt, pageCnt}}.
+ * @throws IgniteCheckedException – if faild.
+ */
+ protected Map<CacheGroupContext, T2<Integer, Long>> loadDataInfo(
Review comment:
Maybe is it better instead of this unspecified map to have some list of specific class like WarmUpPartitionInfo(cacheGroupContext, partitionId, MaxPagesToLoad)?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [ignite] tkalkirill commented on a change in pull request #8148: IGNITE-13345
Posted by GitBox <gi...@apache.org>.
tkalkirill commented on a change in pull request #8148:
URL: https://github.com/apache/ignite/pull/8148#discussion_r473621420
##########
File path: modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheProcessor.java
##########
@@ -5477,6 +5494,96 @@ private void restorePartitionStates(
", partitionsProcessed=" + totalProcessed.get() +
", time=" + (U.currentTimeMillis() - startRestorePart) + "ms]");
}
+
+ /**
+ * Start warming up sequentially for each persist data region.
+ *
+ * @throws IgniteCheckedException If failed.
+ */
+ private void startWarmUp() throws IgniteCheckedException {
+ boolean start = false;
+
+ try {
+ // Collecting custom and default data regions.
+ DataStorageConfiguration dsCfg = ctx.config().getDataStorageConfiguration();
+
+ List<DataRegionConfiguration> regCfgs =
+ new ArrayList<>(asList(dsCfg.getDefaultDataRegionConfiguration()));
+
+ if (nonNull(dsCfg.getDataRegionConfigurations()))
+ regCfgs.addAll(asList(dsCfg.getDataRegionConfigurations()));
+
+ // Warm-up start.
+ Map<Class<? extends WarmUpConfiguration>, WarmUpStrategy> warmUpStrats = CU.warmUpStrategies(ctx);
+
+ WarmUpConfiguration dfltWarmUpCfg = dsCfg.getDefaultWarmUpConfiguration();
+
+ for (DataRegionConfiguration regCfg : regCfgs) {
+ if (stopWarmUp.get())
+ return;
+
+ if (!regCfg.isPersistenceEnabled())
+ continue;
+
+ WarmUpConfiguration warmUpCfg = nonNull(regCfg.getWarmUpConfiguration()) ?
+ regCfg.getWarmUpConfiguration() : dfltWarmUpCfg;
+
+ if (isNull(warmUpCfg))
+ continue;
+
+ WarmUpStrategy warmUpStrat = (curWarmUpStrat = warmUpStrats.get(warmUpCfg.getClass()));
+
+ DataRegion region = sharedCtx.database().dataRegion(regCfg.getName());
+
+ if (!stopWarmUp.get()) {
Review comment:
Please look at **org.apache.ignite.internal.processors.cache.GridCacheProcessor#stopWarmUp**.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org