You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by mahesh76private <ma...@gmail.com> on 2019/06/13 04:44:34 UTC
loading a cache into memory after startup
Hi,
After Ignite cluster (w. data, assuming about 50GB of data) starts up we are
experiencing significant delays before data is accessible.
Understandably, ignite is bringing data from backup on disk into memory
based on query.
is there a way, where we as much data as possible from backup (disk) into
memory (allocated in config XML, see below) at one shot after a cluster
start up?
*
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true"/>
<property name="metricsEnabled" value="true"/>
<property name="maxSize" value="#{16L * 1024 * 1024
* 1024}"/>
</bean>
</property>
*
Right now, data seems to come into memory based on queries. So when we have
large tables with say 12 million records, some of the queries wait for about
3-4 minutes...
regards
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: loading a cache into memory after startup
Posted by mahesh76private <ma...@gmail.com>.
I don't know if I even want to call an api to warm up memory. I am thinking,
we could have a setting like the below in config xml, this is userfriendly
and intention of the config file is very clear.
Bottomline: if ignite is used more often from disk, then it loses its value
from a performance standpoint
So, this option (warmupOnStart) seems mandatory.
<property name="dataStorageConfiguration">
<bean
class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="persistenceEnabled" value="true"/>
<property name="metricsEnabled" value="true"/>
<property name="maxSize" value="#{16L * 1024 * 1024
* 1024}"/>
*<property name="warmupOnStart" value="true" />*
</bean>
</property>
</bean>
</property>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: loading a cache into memory after startup
Posted by Denis Magda <dm...@apache.org>.
Denis, thanks for pointing out to that ticket.
Reopened that ticket proposing to reconsider the public API:
https://issues.apache.org/jira/browse/IGNITE-8873
Generally, I'm in doubts that IgniteCache.preloadPartition is the right API
from the usability standpoint. For instance, Mahesh do you understand how
to use it for your case?
-
Denis
On Thu, Jun 13, 2019 at 3:23 AM Denis Mekhanikov <dm...@gmail.com>
wrote:
> Hi!
>
> You can use a scan query, that will iterate over the whole dataset and
> bring it to memory. You don't need to perform any specific processing. Just
> touching it is enough for it to appear in memory.
> In version 2.8 there will be a new API, allowing preloading partitions.
> IgniteCache.preloadPartition(...) will let you bring all data associated
> with the provided partition to memory.
> Here is the JIRA ticket, that introduces this ability:
> https://issues.apache.org/jira/browse/IGNITE-10019
>
> Denis
>
> чт, 13 июн. 2019 г. в 07:44, mahesh76private <ma...@gmail.com>:
>
>> Hi,
>>
>> After Ignite cluster (w. data, assuming about 50GB of data) starts up we
>> are
>> experiencing significant delays before data is accessible.
>>
>>
>> Understandably, ignite is bringing data from backup on disk into memory
>> based on query.
>>
>> is there a way, where we as much data as possible from backup (disk) into
>> memory (allocated in config XML, see below) at one shot after a cluster
>> start up?
>>
>> *
>> <bean
>> class="org.apache.ignite.configuration.DataRegionConfiguration">
>> <property name="persistenceEnabled" value="true"/>
>> <property name="metricsEnabled" value="true"/>
>> <property name="maxSize" value="#{16L * 1024 *
>> 1024
>> * 1024}"/>
>> </bean>
>> </property>
>> *
>>
>> Right now, data seems to come into memory based on queries. So when we
>> have
>> large tables with say 12 million records, some of the queries wait for
>> about
>> 3-4 minutes...
>>
>> regards
>>
>>
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
Re: loading a cache into memory after startup
Posted by Denis Mekhanikov <dm...@gmail.com>.
Hi!
You can use a scan query, that will iterate over the whole dataset and
bring it to memory. You don't need to perform any specific processing. Just
touching it is enough for it to appear in memory.
In version 2.8 there will be a new API, allowing preloading partitions.
IgniteCache.preloadPartition(...) will let you bring all data associated
with the provided partition to memory.
Here is the JIRA ticket, that introduces this ability:
https://issues.apache.org/jira/browse/IGNITE-10019
Denis
чт, 13 июн. 2019 г. в 07:44, mahesh76private <ma...@gmail.com>:
> Hi,
>
> After Ignite cluster (w. data, assuming about 50GB of data) starts up we
> are
> experiencing significant delays before data is accessible.
>
>
> Understandably, ignite is bringing data from backup on disk into memory
> based on query.
>
> is there a way, where we as much data as possible from backup (disk) into
> memory (allocated in config XML, see below) at one shot after a cluster
> start up?
>
> *
> <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property name="persistenceEnabled" value="true"/>
> <property name="metricsEnabled" value="true"/>
> <property name="maxSize" value="#{16L * 1024 * 1024
> * 1024}"/>
> </bean>
> </property>
> *
>
> Right now, data seems to come into memory based on queries. So when we have
> large tables with say 12 million records, some of the queries wait for
> about
> 3-4 minutes...
>
> regards
>
>
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>