You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by kane <lc...@hotmail.com> on 2017/08/18 08:51:38 UTC
2.1 new feature - warm up the cache / memory
How to config to warm up the cache automatically after restart the cluster
my case:
1, data loaded into IgniteCache
2, restart the cluster
3, expect data to be re-loaded into the IgniteCache, but it doesn't.
But I can see the wal log file and caches folder with specific cache name
has been created under &IGNITE_HOME/work
Has done following configurations in the ignite config xml.
<property name="persistentStoreConfiguration">
<bean
class="org.apache.ignite.configuration.PersistentStoreConfiguration"/>
</property>
<property name="cacheConfiguration">
<list>
<bean
class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value="cache name"/>
<property name="backups" value="1"/>
<property name="atomicityMode" value="TRANSACTIONAL"/>
<property name="writeSynchronizationMode"
value="FULL_SYNC"/>
<property name="indexedTypes">
<list>
<value>java.lang.String</value>
<value>xxx.xxx.xxx</value>
</list>
</property>
</bean>
</list>
</property>
<property name="binaryConfiguration">
<bean
class="org.apache.ignite.configuration.BinaryConfiguration">
<property name="compactFooter" value="false"/>
</bean>
</property>
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/2-1-new-feature-warm-up-the-cache-memory-tp16278.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: 2.1 new feature - warm up the cache / memory
Posted by Alexey Kukushkin <ku...@gmail.com>.
Hi,
The data is loaded by pages (somehow similar to file system's virtual
memory).
Persistence is transparent to all kinds of APIs - a page is loaded by
Ignite when you get data from that page.
If you really want to guarantee that the data your app is going to use are
already in memory you might want to "access" that data during your app
initialization, for example, run a SQL query for that data.
There is no housekeeping API in Apache Ignite but Ignite native persistence
exposes extensibility API. Implement
org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteCacheSnapshotManager
and add the implementation as Ignite plugin.
Some vendors might already provide snapshots implementations.
On Mon, Aug 21, 2017 at 11:12 AM, kane <lc...@hotmail.com> wrote:
> Hi Val,
>
> very appreciate you support, and one more question here.
>
> is there any data housekeeping for the file stored in on the disk ?
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/2-1-new-feature-warm-up-the-cache-
> memory-tp16278p16323.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Alexey
Re: 2.1 new feature - warm up the cache / memory
Posted by kane <lc...@hotmail.com>.
Hi Val,
very appreciate you support, and one more question here.
is there any data housekeeping for the file stored in on the disk ?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/2-1-new-feature-warm-up-the-cache-memory-tp16278p16323.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: 2.1 new feature - warm up the cache / memory
Posted by kane <lc...@hotmail.com>.
Hi Val,
Thanks for your clarification and I got two more questions.
1, How about the performance of the SQL query(for example) If the data is
not pre-loaded to the memory automatically after restart, as described
above, all the related data has to be loaded from the disk for the very
first time, will the query have a visible slow compare the the memory
calculation.
2, Is the data reloaded to the memory in page level or record level ?
e.g.
page A has
- record 1
- record 2
- record 3
If the query need record 1, will Page A(including record 1 , record 2 ,
record 3 ) be loaded or only recrod 1 will be loaded into the memory.
- Kane
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/2-1-new-feature-warm-up-the-cache-memory-tp16278p16316.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: 2.1 new feature - warm up the cache / memory
Posted by Marco <ma...@hotmail.com>.
Val,
Thanks for the explanation, that makes sense. Now I know you guys are making
a 'hybrid engine', the new feature is amazing, maybe it's time to request
new ssd drive.
Have a nice day.
-Marco
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/2-1-new-feature-warm-up-the-cache-memory-tp16278p16309.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: 2.1 new feature - warm up the cache / memory
Posted by vkulichenko <va...@gmail.com>.
Marco,
It's not a background warmup, there is no explicit warm-up at all. Ignite
starting with 2.0 uses page memory architecture, i.e. data is organized in
pages. Any page can be stored both in memory and on disk, and access to this
page is transparent to client, regardless of where it resides. This means
that when you access it (run a SQL query, for example), Ignite will detect
if it's in memory or not, and fetch into memory if it isn't. When if
everything is on disk (which is the case after cluster restart), you can
still access any data.
From architecture standpoint, there is indeed a lot in common with
databases. However Ignite follows memory-centric approach rather than
disk-centric, plus it's distributed and scalable.
Makes sense?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/2-1-new-feature-warm-up-the-cache-memory-tp16278p16303.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: 2.1 new feature - warm up the cache / memory
Posted by Denis Magda <dm...@apache.org>.
Hi Macro,
Val is trying to convey that the memory warmup is an optional operation if the Ignite Native Persistence is used. The data will be brought to memory automatically while you will be executing specific queries touching specific data on disk.
As for the explicit warmup it can be done this way:
https://apacheignite.readme.io/docs/data-loading#section-ignitecacheloadcache
—
Denis
> On Aug 18, 2017, at 11:39 AM, Marco <ma...@hotmail.com> wrote:
>
> Hi Val,
> According to your clarification, This new feature somehow looks like a
> background warm-up, if it's not blocking operation, how do we know
> initialization is done and data is ready? when should we start the new data
> process? to be frank, this design feels a bit like the transaction log as to
> RDBMS.
>
> -Marco
>
>
>
> --
> View this message in context: http://apache-ignite-users.70518.x6.nabble.com/2-1-new-feature-warm-up-the-cache-memory-tp16278p16298.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: 2.1 new feature - warm up the cache / memory
Posted by Marco <ma...@hotmail.com>.
Hi Val,
According to your clarification, This new feature somehow looks like a
background warm-up, if it's not blocking operation, how do we know
initialization is done and data is ready? when should we start the new data
process? to be frank, this design feels a bit like the transaction log as to
RDBMS.
-Marco
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/2-1-new-feature-warm-up-the-cache-memory-tp16278p16298.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: 2.1 new feature - warm up the cache / memory
Posted by vkulichenko <va...@gmail.com>.
Just to clarify: data will not be fetch into in-memory cache right away after
cluster restart. But it will be available for any IgniteCache operations,
including queries. Data will lazily loaded into memory while these
operations are executed.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/2-1-new-feature-warm-up-the-cache-memory-tp16278p16294.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: 2.1 new feature - warm up the cache / memory
Posted by Alexey Kukushkin <ku...@gmail.com>.
Hi,
With persistence enabled the data must be in the cache after the cluster
restart. Can you double check your application or may be share it for
review?
"Warming up" data with persistence enabled makes sense only if not all data
first into memory and you really want to force some specific data into
memory. It looks your problem is you do not see data at all and that is not
normal.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/2-1-new-feature-warm-up-the-cache-memory-tp16278p16291.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.