Best way to manage cache contents

[lmark58 <la...@principled.io>]

Within my application I have a concept of a gathering.  There can be many
gatherings, each of which have participants, activities and comments, which
are their own caches.  i am using cachestore and a database for the
persistent storage of all the gatherings.  At any given time there are a set
of active gatherings which I want to be in the cache.  I cannot just use the
cachestore to bring in the data I need because I use SQLFieldQuries on the
active gatherings and the cachestore does not work well with SQL queries.  I
need a way to make sure an entire gathering is in cache when I do the query.

Over time the number of gatherings will grow very large, but the amount of
active data is pretty small, so I don't want to just load all data.

I have looked at the persistent store which will allow me to have a small in
memory cache and the rest is on disk, but there seems to be a performance
penalty on writes as the data has to go to disk.  This also means that the
cache has to have all the gatherings even though very few will be active, so
I am pretty much doubling my disk needs to keep all the data in both the
database and the persistent store.  I am using the database for other
purposes and it will be managed, backed up, etc. so I cannot use just the
persistent store.  The advantage of this is of course that I don't have to
think about managing the transitions between memory and disk, ignite does it
all for me.


The alternative is to manage contents myself, and as a gathering becomes
active I load all the objects for that gathering into the cache ( from the
database ), and as the cache fills I evict gatherings that are no longer
active.  Ignite has no way to track which objects are part of which
gathering so this would all be logic I would implement.  In this mode I
would not have persistence on the cache, because I would manage the contents
to avoid out of memory errors, and having no persistence should give me the
best performance.  

Being new to Ignite, I wanted to reach out and make sure I am not missing
something.  Is this a common pattern for the application to control the
active subset of data that is in the cache when using the SQL query?  Is
there a better way to do this?  





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Best-way-to-manage-cache-contents-tp16239.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Best way to manage cache contents

[vkulichenko <va...@gmail.com>]

Hi,

You actually very well outlined your options. With third party store you
have to make sure that all data is in memory before running queries. This is
your responsibility to ensure that.

Of course there are use cases when whole data set does not fit in memory.
Ignite Persistence was implemented exactly for this and there is no better
and easier way to achieve this.

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Best-way-to-manage-cache-contents-tp16239p16269.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.