You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by ткаленко кирилл <tk...@yandex.ru> on 2020/08/04 10:28:57 UTC
Re: [DISCUSSION] Cache warmup
Hi, Slava!
Thank you for looking at the offer and making fair comments.
I personally discussed with Anton and Alexey because they are author and sponsor of "IEP-40" and we found out that point 2 in it is no longer relevant and it can be removed.
I suggest implementing point 3, since it may be independent of point 1. Also, the warm-up will always start after restore phase, without subscribing to events.
You are right this should be mentioned in the documentation and javadoc.
> This means that the user's thread, which starts Ignite via
> Ignition.start(), will wait for ana additional step - cache warm-up.
> I think this fact has to be clearly mentioned in our documentation (at
> Javadocat least) because this step can be time-consuming.
My suggestion for implementation:
1)Adding a marker interface "org.apache.ignite.configuration.WarmUpConfiguration" for configuring cache warming;
2)Set only one configuration via "org.apache.ignite.configuration.IgniteConfiguration#setWarmUpConfiguration";
3)Add an internal warm-up interface that will start in [1] after [2];
package org.apache.ignite.internal.processors.cache.warmup;
import org.apache.ignite.IgniteCheckedException;
import org.apache.ignite.configuration.WarmUpConfiguration;
import org.apache.ignite.internal.GridKernalContext;
/**
* Interface for warming up.
*/
public interface WarmUpStrategy<T extends WarmUpConfiguration> {
/**
* Returns configuration class for mapping to strategy.
*
* @return Configuration class.
*/
Class<T> configClass();
/**
* Warm up.
*
* @param kernalCtx Kernal context.
* @param cfg Warm-up configuration.
* @throws IgniteCheckedException if faild.
*/
void warmUp(GridKernalContext kernalCtx, T cfg) throws IgniteCheckedException;
}
4)Adding an internal plugin extension for add own strategies;
package org.apache.ignite.internal.processors.cache.warmup;
import java.util.Collection;
import org.apache.ignite.plugin.Extension;
/**
* Interface for getting warm-up strategies from plugins.
*/
public interface WarmUpStrategySupplier extends Extension {
/**
* Getting warm-up strategies.
*
* @return Warm-up strategies.
*/
Collection<WarmUpStrategy> strategies();
}
5)Add a "Load all" strategy that will load everything to memory as long as there is space in it. This strategy is suitable if the persistent storage is less than RAM.
Any objections or comments?
[1] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
[2] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
27.07.2020, 16:48, "Вячеслав Коптилин" <sl...@gmail.com>:
> Hello Kirill,
>
> Thanks a lot for driving this activity. If I am not mistaken, this
> discussion relates to IEP-40.
>
>> I suggest adding a warmup phase after recovery here [1] after [2], before
>
> discovery.
> This means that the user's thread, which starts Ignite via
> Ignition.start(), will wait for ana additional step - cache warm-up.
> I think this fact has to be clearly mentioned in our documentation (at
> Javadocat least) because this step can be time-consuming.
>
>> I suggest adding a new interface:
>
> I would change it a bit. First of all, it would be nice to place this
> interface to a public package and get rid of using GridCacheContext,
> which is an internal class and it should not leak to the public API in any
> case.
> Perhaps, this parameter is not needed at all or we should add some public
> abstraction instead of internal class.
>
> package org.apache.ignite.configuration;
>
> import org.apache.ignite.IgniteCheckedException;
> import org.apache.ignite.lang.IgniteFuture;
>
> public interface CacheWarmupper {
> /**
> * Warmup cache.
> *
> * @param cachename Cache name.
> * @return Future cache warmup.
> * @throws IgniteCheckedException If failed.
> */
> IgniteFuture<?> warmup(String cachename) throws IgniteCheckedException;
> }
>
> Thanks,
> S.
>
> пн, 27 июл. 2020 г. в 15:03, ткаленко кирилл <tk...@yandex.ru>:
>
>> Now, after restarting node, we have only cold caches, which at first
>> requests to them will gradually load data from disks, which can slow down
>> first calls to them.
>> If node has more RAM than data on disk, then they can be loaded at start
>> "warmup", thereby solving the issue of slowdowns during first calls to
>> caches.
>>
>> I suggest adding a warmup phase after recovery here [1] after [2], before
>> descovery.
>>
>> I suggest adding a new interface:
>>
>> package org.apache.ignite.internal.processors.cache;
>>
>> import org.apache.ignite.IgniteCheckedException;
>> import org.apache.ignite.internal.IgniteInternalFuture;
>> import org.jetbrains.annotations.Nullable;
>>
>> /**
>> * Interface for warming up cache.
>> */
>> public interface CacheWarmup {
>> /**
>> * Warmup cache.
>> *
>> * @param cacheCtx Cache context.
>> * @return Future cache warmup.
>> * @throws IgniteCheckedException if failed.
>> */
>> @Nullable IgniteInternalFuture<?> process(GridCacheContext cacheCtx)
>> throws IgniteCheckedException;
>> }
>>
>> Which will allow to warm up caches in parallel and asynchronously. Warmup
>> phase will end after all IgniteInternalFuture for all caches isDone.
>>
>> Also adding the ability to customize via methods:
>> org.apache.ignite.configuration.IgniteConfiguration#setDefaultCacheWarmup
>> org.apache.ignite.configuration.CacheConfiguration#setCacheWarmup
>>
>> Which will allow for each cache to set implementation of cache warming up,
>> both for a specific cache, and for all if necessary.
>>
>> I suggest adding an implementation of SequentialWarmup that will use [3].
>>
>> Questions, suggestions, comments?
>>
>> [1] -
>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>> [2] -
>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>> [3] -
>> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager.CacheDataStore#preload
Re: [DISCUSSION] Cache warmup
Posted by ткаленко кирилл <tk...@yandex.ru>.
Great! Then I proceed to ticket https://issues.apache.org/jira/browse/IGNITE-13345.
10.08.2020, 16:30, "Stanislav Lukyanov" <st...@gmail.com>:
> All of this looks awesome, covers the use cases I know about.
> Thanks!
>
> Stan
>
>> On 10 Aug 2020, at 15:39, ткаленко кирилл <tk...@yandex.ru> wrote:
>>
>> Hi, Stan again :-)
>>
>> I suggest adding a little more flexibility to configuration:
>> 1)Add default warm-up configuration for all regions into org.apache.ignite.configuration.DataStorageConfiguration#setDefaultWarmUpConfiguration
>> 2)Add a NoOp strategy for turning off heating of a specific region
>>
>> Thus, when starting warm-up, region configuration is taken at beginning, and if it is not present, it is taken from default. And if we don't want to warm up region, we set NoOp.
>>
>> 10.08.2020, 10:20, "ткаленко кирилл" <tk...@yandex.ru>:
>>> Hi, Stan!
>>>
>>> As a result of personal correspondence I realized that you are right about making changes:
>>> 1)Move warm-up configuration to org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration;
>>> 2)Start warming up for each region sequentially;
>>> 3)Improving warm-up interface:
>>>
>>> package org.apache.ignite.internal.processors.cache.warmup;
>>>
>>> import org.apache.ignite.IgniteCheckedException;
>>> import org.apache.ignite.configuration.WarmUpConfiguration;
>>> import org.apache.ignite.internal.GridKernalContext;
>>> import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
>>>
>>> /**
>>> * Interface for warming up.
>>> */
>>> public interface WarmUpStrategy<T extends WarmUpConfiguration> {
>>> /**
>>> * Returns configuration class for mapping to strategy.
>>> *
>>> * @return Configuration class.
>>> */
>>> Class<T> configClass();
>>>
>>> /**
>>> * Warm up.
>>> *
>>> * @param kernalCtx Kernal context.
>>> * @param cfg Warm-up configuration.
>>> * @param region Data region.
>>> * @throws IgniteCheckedException if faild.
>>> */
>>> void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region) throws IgniteCheckedException;
>>>
>>> /**
>>> * Closing warm up.
>>> *
>>> * @throws IgniteCheckedException if faild.
>>> */
>>> void close() throws IgniteCheckedException;
>>> }
>>>
>>> 4)Add a command to "control.sh", to stop current warm-up and cancel all others: --warm-up stop
>>> 5)The "load all" strategy will work as long as there is enough RAM and index pages will also take priority.
>>>
>>> 04.08.2020, 13:29, "ткаленко кирилл" <tk...@yandex.ru>:
>>>> Hi, Slava!
>>>>
>>>> Thank you for looking at the offer and making fair comments.
>>>>
>>>> I personally discussed with Anton and Alexey because they are author and sponsor of "IEP-40" and we found out that point 2 in it is no longer relevant and it can be removed.
>>>> I suggest implementing point 3, since it may be independent of point 1. Also, the warm-up will always start after restore phase, without subscribing to events.
>>>>
>>>> You are right this should be mentioned in the documentation and javadoc.
>>>>> This means that the user's thread, which starts Ignite via
>>>>> Ignition.start(), will wait for ana additional step - cache warm-up.
>>>>> I think this fact has to be clearly mentioned in our documentation (at
>>>>> Javadocat least) because this step can be time-consuming.
>>>>
>>>> My suggestion for implementation:
>>>> 1)Adding a marker interface "org.apache.ignite.configuration.WarmUpConfiguration" for configuring cache warming;
>>>> 2)Set only one configuration via "org.apache.ignite.configuration.IgniteConfiguration#setWarmUpConfiguration";
>>>> 3)Add an internal warm-up interface that will start in [1] after [2];
>>>>
>>>> package org.apache.ignite.internal.processors.cache.warmup;
>>>>
>>>> import org.apache.ignite.IgniteCheckedException;
>>>> import org.apache.ignite.configuration.WarmUpConfiguration;
>>>> import org.apache.ignite.internal.GridKernalContext;
>>>>
>>>> /**
>>>> * Interface for warming up.
>>>> */
>>>> public interface WarmUpStrategy<T extends WarmUpConfiguration> {
>>>> /**
>>>> * Returns configuration class for mapping to strategy.
>>>> *
>>>> * @return Configuration class.
>>>> */
>>>> Class<T> configClass();
>>>>
>>>> /**
>>>> * Warm up.
>>>> *
>>>> * @param kernalCtx Kernal context.
>>>> * @param cfg Warm-up configuration.
>>>> * @throws IgniteCheckedException if faild.
>>>> */
>>>> void warmUp(GridKernalContext kernalCtx, T cfg) throws IgniteCheckedException;
>>>> }
>>>>
>>>> 4)Adding an internal plugin extension for add own strategies;
>>>>
>>>> package org.apache.ignite.internal.processors.cache.warmup;
>>>>
>>>> import java.util.Collection;
>>>> import org.apache.ignite.plugin.Extension;
>>>>
>>>> /**
>>>> * Interface for getting warm-up strategies from plugins.
>>>> */
>>>> public interface WarmUpStrategySupplier extends Extension {
>>>> /**
>>>> * Getting warm-up strategies.
>>>> *
>>>> * @return Warm-up strategies.
>>>> */
>>>> Collection<WarmUpStrategy> strategies();
>>>> }
>>>>
>>>> 5)Add a "Load all" strategy that will load everything to memory as long as there is space in it. This strategy is suitable if the persistent storage is less than RAM.
>>>>
>>>> Any objections or comments?
>>>>
>>>> [1] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>>> [2] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>>>
>>>> 27.07.2020, 16:48, "Вячеслав Коптилин" <sl...@gmail.com>:
>>>>> Hello Kirill,
>>>>>
>>>>> Thanks a lot for driving this activity. If I am not mistaken, this
>>>>> discussion relates to IEP-40.
>>>>>
>>>>>> I suggest adding a warmup phase after recovery here [1] after [2], before
>>>>>
>>>>> discovery.
>>>>> This means that the user's thread, which starts Ignite via
>>>>> Ignition.start(), will wait for ana additional step - cache warm-up.
>>>>> I think this fact has to be clearly mentioned in our documentation (at
>>>>> Javadocat least) because this step can be time-consuming.
>>>>>
>>>>>> I suggest adding a new interface:
>>>>>
>>>>> I would change it a bit. First of all, it would be nice to place this
>>>>> interface to a public package and get rid of using GridCacheContext,
>>>>> which is an internal class and it should not leak to the public API in any
>>>>> case.
>>>>> Perhaps, this parameter is not needed at all or we should add some public
>>>>> abstraction instead of internal class.
>>>>>
>>>>> package org.apache.ignite.configuration;
>>>>>
>>>>> import org.apache.ignite.IgniteCheckedException;
>>>>> import org.apache.ignite.lang.IgniteFuture;
>>>>>
>>>>> public interface CacheWarmupper {
>>>>> /**
>>>>> * Warmup cache.
>>>>> *
>>>>> * @param cachename Cache name.
>>>>> * @return Future cache warmup.
>>>>> * @throws IgniteCheckedException If failed.
>>>>> */
>>>>> IgniteFuture<?> warmup(String cachename) throws IgniteCheckedException;
>>>>> }
>>>>>
>>>>> Thanks,
>>>>> S.
>>>>>
>>>>> пн, 27 июл. 2020 г. в 15:03, ткаленко кирилл <tk...@yandex.ru>:
>>>>>
>>>>>> Now, after restarting node, we have only cold caches, which at first
>>>>>> requests to them will gradually load data from disks, which can slow down
>>>>>> first calls to them.
>>>>>> If node has more RAM than data on disk, then they can be loaded at start
>>>>>> "warmup", thereby solving the issue of slowdowns during first calls to
>>>>>> caches.
>>>>>>
>>>>>> I suggest adding a warmup phase after recovery here [1] after [2], before
>>>>>> descovery.
>>>>>>
>>>>>> I suggest adding a new interface:
>>>>>>
>>>>>> package org.apache.ignite.internal.processors.cache;
>>>>>>
>>>>>> import org.apache.ignite.IgniteCheckedException;
>>>>>> import org.apache.ignite.internal.IgniteInternalFuture;
>>>>>> import org.jetbrains.annotations.Nullable;
>>>>>>
>>>>>> /**
>>>>>> * Interface for warming up cache.
>>>>>> */
>>>>>> public interface CacheWarmup {
>>>>>> /**
>>>>>> * Warmup cache.
>>>>>> *
>>>>>> * @param cacheCtx Cache context.
>>>>>> * @return Future cache warmup.
>>>>>> * @throws IgniteCheckedException if failed.
>>>>>> */
>>>>>> @Nullable IgniteInternalFuture<?> process(GridCacheContext cacheCtx)
>>>>>> throws IgniteCheckedException;
>>>>>> }
>>>>>>
>>>>>> Which will allow to warm up caches in parallel and asynchronously. Warmup
>>>>>> phase will end after all IgniteInternalFuture for all caches isDone.
>>>>>>
>>>>>> Also adding the ability to customize via methods:
>>>>>> org.apache.ignite.configuration.IgniteConfiguration#setDefaultCacheWarmup
>>>>>> org.apache.ignite.configuration.CacheConfiguration#setCacheWarmup
>>>>>>
>>>>>> Which will allow for each cache to set implementation of cache warming up,
>>>>>> both for a specific cache, and for all if necessary.
>>>>>>
>>>>>> I suggest adding an implementation of SequentialWarmup that will use [3].
>>>>>>
>>>>>> Questions, suggestions, comments?
>>>>>>
>>>>>> [1] -
>>>>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>>>>> [2] -
>>>>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>>>>> [3] -
>>>>>> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager.CacheDataStore#preload
Re: [DISCUSSION] Cache warmup
Posted by Stanislav Lukyanov <st...@gmail.com>.
All of this looks awesome, covers the use cases I know about.
Thanks!
Stan
> On 10 Aug 2020, at 15:39, ткаленко кирилл <tk...@yandex.ru> wrote:
>
> Hi, Stan again :-)
>
> I suggest adding a little more flexibility to configuration:
> 1)Add default warm-up configuration for all regions into org.apache.ignite.configuration.DataStorageConfiguration#setDefaultWarmUpConfiguration
> 2)Add a NoOp strategy for turning off heating of a specific region
>
> Thus, when starting warm-up, region configuration is taken at beginning, and if it is not present, it is taken from default. And if we don't want to warm up region, we set NoOp.
>
> 10.08.2020, 10:20, "ткаленко кирилл" <tk...@yandex.ru>:
>> Hi, Stan!
>>
>> As a result of personal correspondence I realized that you are right about making changes:
>> 1)Move warm-up configuration to org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration;
>> 2)Start warming up for each region sequentially;
>> 3)Improving warm-up interface:
>>
>> package org.apache.ignite.internal.processors.cache.warmup;
>>
>> import org.apache.ignite.IgniteCheckedException;
>> import org.apache.ignite.configuration.WarmUpConfiguration;
>> import org.apache.ignite.internal.GridKernalContext;
>> import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
>>
>> /**
>> * Interface for warming up.
>> */
>> public interface WarmUpStrategy<T extends WarmUpConfiguration> {
>> /**
>> * Returns configuration class for mapping to strategy.
>> *
>> * @return Configuration class.
>> */
>> Class<T> configClass();
>>
>> /**
>> * Warm up.
>> *
>> * @param kernalCtx Kernal context.
>> * @param cfg Warm-up configuration.
>> * @param region Data region.
>> * @throws IgniteCheckedException if faild.
>> */
>> void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region) throws IgniteCheckedException;
>>
>> /**
>> * Closing warm up.
>> *
>> * @throws IgniteCheckedException if faild.
>> */
>> void close() throws IgniteCheckedException;
>> }
>>
>> 4)Add a command to "control.sh", to stop current warm-up and cancel all others: --warm-up stop
>> 5)The "load all" strategy will work as long as there is enough RAM and index pages will also take priority.
>>
>> 04.08.2020, 13:29, "ткаленко кирилл" <tk...@yandex.ru>:
>>> Hi, Slava!
>>>
>>> Thank you for looking at the offer and making fair comments.
>>>
>>> I personally discussed with Anton and Alexey because they are author and sponsor of "IEP-40" and we found out that point 2 in it is no longer relevant and it can be removed.
>>> I suggest implementing point 3, since it may be independent of point 1. Also, the warm-up will always start after restore phase, without subscribing to events.
>>>
>>> You are right this should be mentioned in the documentation and javadoc.
>>>> This means that the user's thread, which starts Ignite via
>>>> Ignition.start(), will wait for ana additional step - cache warm-up.
>>>> I think this fact has to be clearly mentioned in our documentation (at
>>>> Javadocat least) because this step can be time-consuming.
>>>
>>> My suggestion for implementation:
>>> 1)Adding a marker interface "org.apache.ignite.configuration.WarmUpConfiguration" for configuring cache warming;
>>> 2)Set only one configuration via "org.apache.ignite.configuration.IgniteConfiguration#setWarmUpConfiguration";
>>> 3)Add an internal warm-up interface that will start in [1] after [2];
>>>
>>> package org.apache.ignite.internal.processors.cache.warmup;
>>>
>>> import org.apache.ignite.IgniteCheckedException;
>>> import org.apache.ignite.configuration.WarmUpConfiguration;
>>> import org.apache.ignite.internal.GridKernalContext;
>>>
>>> /**
>>> * Interface for warming up.
>>> */
>>> public interface WarmUpStrategy<T extends WarmUpConfiguration> {
>>> /**
>>> * Returns configuration class for mapping to strategy.
>>> *
>>> * @return Configuration class.
>>> */
>>> Class<T> configClass();
>>>
>>> /**
>>> * Warm up.
>>> *
>>> * @param kernalCtx Kernal context.
>>> * @param cfg Warm-up configuration.
>>> * @throws IgniteCheckedException if faild.
>>> */
>>> void warmUp(GridKernalContext kernalCtx, T cfg) throws IgniteCheckedException;
>>> }
>>>
>>> 4)Adding an internal plugin extension for add own strategies;
>>>
>>> package org.apache.ignite.internal.processors.cache.warmup;
>>>
>>> import java.util.Collection;
>>> import org.apache.ignite.plugin.Extension;
>>>
>>> /**
>>> * Interface for getting warm-up strategies from plugins.
>>> */
>>> public interface WarmUpStrategySupplier extends Extension {
>>> /**
>>> * Getting warm-up strategies.
>>> *
>>> * @return Warm-up strategies.
>>> */
>>> Collection<WarmUpStrategy> strategies();
>>> }
>>>
>>> 5)Add a "Load all" strategy that will load everything to memory as long as there is space in it. This strategy is suitable if the persistent storage is less than RAM.
>>>
>>> Any objections or comments?
>>>
>>> [1] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>> [2] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>>
>>> 27.07.2020, 16:48, "Вячеслав Коптилин" <sl...@gmail.com>:
>>>> Hello Kirill,
>>>>
>>>> Thanks a lot for driving this activity. If I am not mistaken, this
>>>> discussion relates to IEP-40.
>>>>
>>>>> I suggest adding a warmup phase after recovery here [1] after [2], before
>>>>
>>>> discovery.
>>>> This means that the user's thread, which starts Ignite via
>>>> Ignition.start(), will wait for ana additional step - cache warm-up.
>>>> I think this fact has to be clearly mentioned in our documentation (at
>>>> Javadocat least) because this step can be time-consuming.
>>>>
>>>>> I suggest adding a new interface:
>>>>
>>>> I would change it a bit. First of all, it would be nice to place this
>>>> interface to a public package and get rid of using GridCacheContext,
>>>> which is an internal class and it should not leak to the public API in any
>>>> case.
>>>> Perhaps, this parameter is not needed at all or we should add some public
>>>> abstraction instead of internal class.
>>>>
>>>> package org.apache.ignite.configuration;
>>>>
>>>> import org.apache.ignite.IgniteCheckedException;
>>>> import org.apache.ignite.lang.IgniteFuture;
>>>>
>>>> public interface CacheWarmupper {
>>>> /**
>>>> * Warmup cache.
>>>> *
>>>> * @param cachename Cache name.
>>>> * @return Future cache warmup.
>>>> * @throws IgniteCheckedException If failed.
>>>> */
>>>> IgniteFuture<?> warmup(String cachename) throws IgniteCheckedException;
>>>> }
>>>>
>>>> Thanks,
>>>> S.
>>>>
>>>> пн, 27 июл. 2020 г. в 15:03, ткаленко кирилл <tk...@yandex.ru>:
>>>>
>>>>> Now, after restarting node, we have only cold caches, which at first
>>>>> requests to them will gradually load data from disks, which can slow down
>>>>> first calls to them.
>>>>> If node has more RAM than data on disk, then they can be loaded at start
>>>>> "warmup", thereby solving the issue of slowdowns during first calls to
>>>>> caches.
>>>>>
>>>>> I suggest adding a warmup phase after recovery here [1] after [2], before
>>>>> descovery.
>>>>>
>>>>> I suggest adding a new interface:
>>>>>
>>>>> package org.apache.ignite.internal.processors.cache;
>>>>>
>>>>> import org.apache.ignite.IgniteCheckedException;
>>>>> import org.apache.ignite.internal.IgniteInternalFuture;
>>>>> import org.jetbrains.annotations.Nullable;
>>>>>
>>>>> /**
>>>>> * Interface for warming up cache.
>>>>> */
>>>>> public interface CacheWarmup {
>>>>> /**
>>>>> * Warmup cache.
>>>>> *
>>>>> * @param cacheCtx Cache context.
>>>>> * @return Future cache warmup.
>>>>> * @throws IgniteCheckedException if failed.
>>>>> */
>>>>> @Nullable IgniteInternalFuture<?> process(GridCacheContext cacheCtx)
>>>>> throws IgniteCheckedException;
>>>>> }
>>>>>
>>>>> Which will allow to warm up caches in parallel and asynchronously. Warmup
>>>>> phase will end after all IgniteInternalFuture for all caches isDone.
>>>>>
>>>>> Also adding the ability to customize via methods:
>>>>> org.apache.ignite.configuration.IgniteConfiguration#setDefaultCacheWarmup
>>>>> org.apache.ignite.configuration.CacheConfiguration#setCacheWarmup
>>>>>
>>>>> Which will allow for each cache to set implementation of cache warming up,
>>>>> both for a specific cache, and for all if necessary.
>>>>>
>>>>> I suggest adding an implementation of SequentialWarmup that will use [3].
>>>>>
>>>>> Questions, suggestions, comments?
>>>>>
>>>>> [1] -
>>>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>>>> [2] -
>>>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>>>> [3] -
>>>>> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager.CacheDataStore#preload
Re: [DISCUSSION] Cache warmup
Posted by ткаленко кирилл <tk...@yandex.ru>.
Hi, Stan again :-)
I suggest adding a little more flexibility to configuration:
1)Add default warm-up configuration for all regions into org.apache.ignite.configuration.DataStorageConfiguration#setDefaultWarmUpConfiguration
2)Add a NoOp strategy for turning off heating of a specific region
Thus, when starting warm-up, region configuration is taken at beginning, and if it is not present, it is taken from default. And if we don't want to warm up region, we set NoOp.
10.08.2020, 10:20, "ткаленко кирилл" <tk...@yandex.ru>:
> Hi, Stan!
>
> As a result of personal correspondence I realized that you are right about making changes:
> 1)Move warm-up configuration to org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration;
> 2)Start warming up for each region sequentially;
> 3)Improving warm-up interface:
>
> package org.apache.ignite.internal.processors.cache.warmup;
>
> import org.apache.ignite.IgniteCheckedException;
> import org.apache.ignite.configuration.WarmUpConfiguration;
> import org.apache.ignite.internal.GridKernalContext;
> import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
>
> /**
> * Interface for warming up.
> */
> public interface WarmUpStrategy<T extends WarmUpConfiguration> {
> /**
> * Returns configuration class for mapping to strategy.
> *
> * @return Configuration class.
> */
> Class<T> configClass();
>
> /**
> * Warm up.
> *
> * @param kernalCtx Kernal context.
> * @param cfg Warm-up configuration.
> * @param region Data region.
> * @throws IgniteCheckedException if faild.
> */
> void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region) throws IgniteCheckedException;
>
> /**
> * Closing warm up.
> *
> * @throws IgniteCheckedException if faild.
> */
> void close() throws IgniteCheckedException;
> }
>
> 4)Add a command to "control.sh", to stop current warm-up and cancel all others: --warm-up stop
> 5)The "load all" strategy will work as long as there is enough RAM and index pages will also take priority.
>
> 04.08.2020, 13:29, "ткаленко кирилл" <tk...@yandex.ru>:
>> Hi, Slava!
>>
>> Thank you for looking at the offer and making fair comments.
>>
>> I personally discussed with Anton and Alexey because they are author and sponsor of "IEP-40" and we found out that point 2 in it is no longer relevant and it can be removed.
>> I suggest implementing point 3, since it may be independent of point 1. Also, the warm-up will always start after restore phase, without subscribing to events.
>>
>> You are right this should be mentioned in the documentation and javadoc.
>>> This means that the user's thread, which starts Ignite via
>>> Ignition.start(), will wait for ana additional step - cache warm-up.
>>> I think this fact has to be clearly mentioned in our documentation (at
>>> Javadocat least) because this step can be time-consuming.
>>
>> My suggestion for implementation:
>> 1)Adding a marker interface "org.apache.ignite.configuration.WarmUpConfiguration" for configuring cache warming;
>> 2)Set only one configuration via "org.apache.ignite.configuration.IgniteConfiguration#setWarmUpConfiguration";
>> 3)Add an internal warm-up interface that will start in [1] after [2];
>>
>> package org.apache.ignite.internal.processors.cache.warmup;
>>
>> import org.apache.ignite.IgniteCheckedException;
>> import org.apache.ignite.configuration.WarmUpConfiguration;
>> import org.apache.ignite.internal.GridKernalContext;
>>
>> /**
>> * Interface for warming up.
>> */
>> public interface WarmUpStrategy<T extends WarmUpConfiguration> {
>> /**
>> * Returns configuration class for mapping to strategy.
>> *
>> * @return Configuration class.
>> */
>> Class<T> configClass();
>>
>> /**
>> * Warm up.
>> *
>> * @param kernalCtx Kernal context.
>> * @param cfg Warm-up configuration.
>> * @throws IgniteCheckedException if faild.
>> */
>> void warmUp(GridKernalContext kernalCtx, T cfg) throws IgniteCheckedException;
>> }
>>
>> 4)Adding an internal plugin extension for add own strategies;
>>
>> package org.apache.ignite.internal.processors.cache.warmup;
>>
>> import java.util.Collection;
>> import org.apache.ignite.plugin.Extension;
>>
>> /**
>> * Interface for getting warm-up strategies from plugins.
>> */
>> public interface WarmUpStrategySupplier extends Extension {
>> /**
>> * Getting warm-up strategies.
>> *
>> * @return Warm-up strategies.
>> */
>> Collection<WarmUpStrategy> strategies();
>> }
>>
>> 5)Add a "Load all" strategy that will load everything to memory as long as there is space in it. This strategy is suitable if the persistent storage is less than RAM.
>>
>> Any objections or comments?
>>
>> [1] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>> [2] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>
>> 27.07.2020, 16:48, "Вячеслав Коптилин" <sl...@gmail.com>:
>>> Hello Kirill,
>>>
>>> Thanks a lot for driving this activity. If I am not mistaken, this
>>> discussion relates to IEP-40.
>>>
>>>> I suggest adding a warmup phase after recovery here [1] after [2], before
>>>
>>> discovery.
>>> This means that the user's thread, which starts Ignite via
>>> Ignition.start(), will wait for ana additional step - cache warm-up.
>>> I think this fact has to be clearly mentioned in our documentation (at
>>> Javadocat least) because this step can be time-consuming.
>>>
>>>> I suggest adding a new interface:
>>>
>>> I would change it a bit. First of all, it would be nice to place this
>>> interface to a public package and get rid of using GridCacheContext,
>>> which is an internal class and it should not leak to the public API in any
>>> case.
>>> Perhaps, this parameter is not needed at all or we should add some public
>>> abstraction instead of internal class.
>>>
>>> package org.apache.ignite.configuration;
>>>
>>> import org.apache.ignite.IgniteCheckedException;
>>> import org.apache.ignite.lang.IgniteFuture;
>>>
>>> public interface CacheWarmupper {
>>> /**
>>> * Warmup cache.
>>> *
>>> * @param cachename Cache name.
>>> * @return Future cache warmup.
>>> * @throws IgniteCheckedException If failed.
>>> */
>>> IgniteFuture<?> warmup(String cachename) throws IgniteCheckedException;
>>> }
>>>
>>> Thanks,
>>> S.
>>>
>>> пн, 27 июл. 2020 г. в 15:03, ткаленко кирилл <tk...@yandex.ru>:
>>>
>>>> Now, after restarting node, we have only cold caches, which at first
>>>> requests to them will gradually load data from disks, which can slow down
>>>> first calls to them.
>>>> If node has more RAM than data on disk, then they can be loaded at start
>>>> "warmup", thereby solving the issue of slowdowns during first calls to
>>>> caches.
>>>>
>>>> I suggest adding a warmup phase after recovery here [1] after [2], before
>>>> descovery.
>>>>
>>>> I suggest adding a new interface:
>>>>
>>>> package org.apache.ignite.internal.processors.cache;
>>>>
>>>> import org.apache.ignite.IgniteCheckedException;
>>>> import org.apache.ignite.internal.IgniteInternalFuture;
>>>> import org.jetbrains.annotations.Nullable;
>>>>
>>>> /**
>>>> * Interface for warming up cache.
>>>> */
>>>> public interface CacheWarmup {
>>>> /**
>>>> * Warmup cache.
>>>> *
>>>> * @param cacheCtx Cache context.
>>>> * @return Future cache warmup.
>>>> * @throws IgniteCheckedException if failed.
>>>> */
>>>> @Nullable IgniteInternalFuture<?> process(GridCacheContext cacheCtx)
>>>> throws IgniteCheckedException;
>>>> }
>>>>
>>>> Which will allow to warm up caches in parallel and asynchronously. Warmup
>>>> phase will end after all IgniteInternalFuture for all caches isDone.
>>>>
>>>> Also adding the ability to customize via methods:
>>>> org.apache.ignite.configuration.IgniteConfiguration#setDefaultCacheWarmup
>>>> org.apache.ignite.configuration.CacheConfiguration#setCacheWarmup
>>>>
>>>> Which will allow for each cache to set implementation of cache warming up,
>>>> both for a specific cache, and for all if necessary.
>>>>
>>>> I suggest adding an implementation of SequentialWarmup that will use [3].
>>>>
>>>> Questions, suggestions, comments?
>>>>
>>>> [1] -
>>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>>> [2] -
>>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>>> [3] -
>>>> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager.CacheDataStore#preload
Re: [DISCUSSION] Cache warmup
Posted by ткаленко кирилл <tk...@yandex.ru>.
Hi, Stan!
As a result of personal correspondence I realized that you are right about making changes:
1)Move warm-up configuration to org.apache.ignite.configuration.DataRegionConfiguration#setWarmUpConfiguration;
2)Start warming up for each region sequentially;
3)Improving warm-up interface:
package org.apache.ignite.internal.processors.cache.warmup;
import org.apache.ignite.IgniteCheckedException;
import org.apache.ignite.configuration.WarmUpConfiguration;
import org.apache.ignite.internal.GridKernalContext;
import org.apache.ignite.internal.processors.cache.persistence.DataRegion;
/**
* Interface for warming up.
*/
public interface WarmUpStrategy<T extends WarmUpConfiguration> {
/**
* Returns configuration class for mapping to strategy.
*
* @return Configuration class.
*/
Class<T> configClass();
/**
* Warm up.
*
* @param kernalCtx Kernal context.
* @param cfg Warm-up configuration.
* @param region Data region.
* @throws IgniteCheckedException if faild.
*/
void warmUp(GridKernalContext kernalCtx, T cfg, DataRegion region) throws IgniteCheckedException;
/**
* Closing warm up.
*
* @throws IgniteCheckedException if faild.
*/
void close() throws IgniteCheckedException;
}
4)Add a command to "control.sh", to stop current warm-up and cancel all others: --warm-up stop
5)The "load all" strategy will work as long as there is enough RAM and index pages will also take priority.
04.08.2020, 13:29, "ткаленко кирилл" <tk...@yandex.ru>:
> Hi, Slava!
>
> Thank you for looking at the offer and making fair comments.
>
> I personally discussed with Anton and Alexey because they are author and sponsor of "IEP-40" and we found out that point 2 in it is no longer relevant and it can be removed.
> I suggest implementing point 3, since it may be independent of point 1. Also, the warm-up will always start after restore phase, without subscribing to events.
>
> You are right this should be mentioned in the documentation and javadoc.
>> This means that the user's thread, which starts Ignite via
>> Ignition.start(), will wait for ana additional step - cache warm-up.
>> I think this fact has to be clearly mentioned in our documentation (at
>> Javadocat least) because this step can be time-consuming.
>
> My suggestion for implementation:
> 1)Adding a marker interface "org.apache.ignite.configuration.WarmUpConfiguration" for configuring cache warming;
> 2)Set only one configuration via "org.apache.ignite.configuration.IgniteConfiguration#setWarmUpConfiguration";
> 3)Add an internal warm-up interface that will start in [1] after [2];
>
> package org.apache.ignite.internal.processors.cache.warmup;
>
> import org.apache.ignite.IgniteCheckedException;
> import org.apache.ignite.configuration.WarmUpConfiguration;
> import org.apache.ignite.internal.GridKernalContext;
>
> /**
> * Interface for warming up.
> */
> public interface WarmUpStrategy<T extends WarmUpConfiguration> {
> /**
> * Returns configuration class for mapping to strategy.
> *
> * @return Configuration class.
> */
> Class<T> configClass();
>
> /**
> * Warm up.
> *
> * @param kernalCtx Kernal context.
> * @param cfg Warm-up configuration.
> * @throws IgniteCheckedException if faild.
> */
> void warmUp(GridKernalContext kernalCtx, T cfg) throws IgniteCheckedException;
> }
>
> 4)Adding an internal plugin extension for add own strategies;
>
> package org.apache.ignite.internal.processors.cache.warmup;
>
> import java.util.Collection;
> import org.apache.ignite.plugin.Extension;
>
> /**
> * Interface for getting warm-up strategies from plugins.
> */
> public interface WarmUpStrategySupplier extends Extension {
> /**
> * Getting warm-up strategies.
> *
> * @return Warm-up strategies.
> */
> Collection<WarmUpStrategy> strategies();
> }
>
> 5)Add a "Load all" strategy that will load everything to memory as long as there is space in it. This strategy is suitable if the persistent storage is less than RAM.
>
> Any objections or comments?
>
> [1] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
> [2] - org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>
> 27.07.2020, 16:48, "Вячеслав Коптилин" <sl...@gmail.com>:
>> Hello Kirill,
>>
>> Thanks a lot for driving this activity. If I am not mistaken, this
>> discussion relates to IEP-40.
>>
>>> I suggest adding a warmup phase after recovery here [1] after [2], before
>>
>> discovery.
>> This means that the user's thread, which starts Ignite via
>> Ignition.start(), will wait for ana additional step - cache warm-up.
>> I think this fact has to be clearly mentioned in our documentation (at
>> Javadocat least) because this step can be time-consuming.
>>
>>> I suggest adding a new interface:
>>
>> I would change it a bit. First of all, it would be nice to place this
>> interface to a public package and get rid of using GridCacheContext,
>> which is an internal class and it should not leak to the public API in any
>> case.
>> Perhaps, this parameter is not needed at all or we should add some public
>> abstraction instead of internal class.
>>
>> package org.apache.ignite.configuration;
>>
>> import org.apache.ignite.IgniteCheckedException;
>> import org.apache.ignite.lang.IgniteFuture;
>>
>> public interface CacheWarmupper {
>> /**
>> * Warmup cache.
>> *
>> * @param cachename Cache name.
>> * @return Future cache warmup.
>> * @throws IgniteCheckedException If failed.
>> */
>> IgniteFuture<?> warmup(String cachename) throws IgniteCheckedException;
>> }
>>
>> Thanks,
>> S.
>>
>> пн, 27 июл. 2020 г. в 15:03, ткаленко кирилл <tk...@yandex.ru>:
>>
>>> Now, after restarting node, we have only cold caches, which at first
>>> requests to them will gradually load data from disks, which can slow down
>>> first calls to them.
>>> If node has more RAM than data on disk, then they can be loaded at start
>>> "warmup", thereby solving the issue of slowdowns during first calls to
>>> caches.
>>>
>>> I suggest adding a warmup phase after recovery here [1] after [2], before
>>> descovery.
>>>
>>> I suggest adding a new interface:
>>>
>>> package org.apache.ignite.internal.processors.cache;
>>>
>>> import org.apache.ignite.IgniteCheckedException;
>>> import org.apache.ignite.internal.IgniteInternalFuture;
>>> import org.jetbrains.annotations.Nullable;
>>>
>>> /**
>>> * Interface for warming up cache.
>>> */
>>> public interface CacheWarmup {
>>> /**
>>> * Warmup cache.
>>> *
>>> * @param cacheCtx Cache context.
>>> * @return Future cache warmup.
>>> * @throws IgniteCheckedException if failed.
>>> */
>>> @Nullable IgniteInternalFuture<?> process(GridCacheContext cacheCtx)
>>> throws IgniteCheckedException;
>>> }
>>>
>>> Which will allow to warm up caches in parallel and asynchronously. Warmup
>>> phase will end after all IgniteInternalFuture for all caches isDone.
>>>
>>> Also adding the ability to customize via methods:
>>> org.apache.ignite.configuration.IgniteConfiguration#setDefaultCacheWarmup
>>> org.apache.ignite.configuration.CacheConfiguration#setCacheWarmup
>>>
>>> Which will allow for each cache to set implementation of cache warming up,
>>> both for a specific cache, and for all if necessary.
>>>
>>> I suggest adding an implementation of SequentialWarmup that will use [3].
>>>
>>> Questions, suggestions, comments?
>>>
>>> [1] -
>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#afterLogicalUpdatesApplied
>>> [2] -
>>> org.apache.ignite.internal.processors.cache.GridCacheProcessor.CacheRecoveryLifecycle#restorePartitionStates
>>> [3] -
>>> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManager.CacheDataStore#preload