You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Ognen Duzlevski <og...@gmail.com> on 2015/06/29 21:38:18 UTC

CacheStore load/get and SQL

Hello all,

I am implementing a CacheStore interface to make use of the
read-through/write-through Ignite capabilities with a background database.

In this scenario a cache.get() method invokes the CacheStore.load()
overriden method. How about SQL queries on ignite? Do they end up being
translated into series of gets() and hence make use of the read-through
CacheStore implementation?

Thanks!

Re: CacheStore load/get and SQL

Posted by vkulichenko <va...@gmail.com>.
Kevin,

I just tried to simplify the task, but it looks like my suggestion will not
work. In your case I would recommend to avoid evictions if possible, because
they will slow you down. If there is no other way, you can enable disk-based
swap [1]. It will dump evicted entries into disk instead of completely
removing them, so they will be still available to the query engine.

[1]
https://ignite.apache.org/releases/1.3.0/javadoc/index.html?org/apache/ignite/spi/swapspace/file/FileSwapSpaceSpi.html

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/CacheStore-load-get-and-SQL-tp582p1488.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: CacheStore load/get and SQL

Posted by kevin <ke...@leonardo.com>.
Hi Val,

Sorry, I don't fully understand your suggestion. Are you suggesting I cache
an entry like:
   key=company/companyId, value=collection of persons
Then if I create/update a Person's company, I would need to update both that
Person, and this collection of persons entry. Correct?

This will not be the only query I'm going to execute. I'm sort of laying
down what the strategy is as I may have alot of similar queries later (eg.
get the collection of Addresses for a Person, etc). I'm also keeping in mind
how I would do the reverse lookup as well (eg. get the Company for a
Person), so I would like to keep companyId on the Person object.

Thanks,
Kevin



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/CacheStore-load-get-and-SQL-tp582p1486.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: CacheStore load/get and SQL

Posted by vkulichenko <va...@gmail.com>.
Kevin,

Is this the only query you're going to execute? If so, this is a simple
caching use case, i.e. you want to cache DB query results in memory. In this
case you can save this query result (e.g., collection of persons for a
particular organization) as one entry and use simple cache lookups and
updates with read/write-through. If evictions occur, you always have
consistent data. Does this make sense for you?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/CacheStore-load-get-and-SQL-tp582p1483.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: CacheStore load/get and SQL

Posted by kevin <ke...@leonardo.com>.
I'm trying to do something similar - run a query (ScanQuery or SqlQuery)
against the cache, and would like it to read-through.

My use case is this:
I store Person and Company objects in Ignite caches. Person objects have the
ID of the Company (which is also the key for the Company cache) they belong
to.
I would like to retrieve the Persons in a given Company.

If I run a query against the Person cache (where companyId = x), it would
only return the Persons that in the cache.
I'm thinking of calling cache.loadCache to read all the Persons for a given
Company and caching them, and then calling cache.query(). I would need some
strategy to know to only call cache.loadCache when necessary, otherwise
there would be a database hit every time I do the query.. which would defeat
the purpose of the cache. Unless I always keep every Person in the cache
(without evicting/expiring them), then I won't need to call loadCache before
the query.

An alternative is to have the Company in the cache store a list of IDs of
all the Persons belonging to that Company, then I can do a simple key lookup
for the Persons, but I would have to maintain that list of IDs in the cache
when creating/deleting Persons.

Any suggestions to what I'm trying to accomplish?

Thanks



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/CacheStore-load-get-and-SQL-tp582p1482.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: CacheStore load/get and SQL

Posted by Ognen Duzlevski <og...@gmail.com>.
And the code in
main/java/org/apache/ignite/examples/datagrid/store/jdbc/CacheJdbcStoreExample.java
explains it all. Sorry for the noise.

I am not sure if applying this to the sql query is a possibility or if it
even makes sense. I have to wrap my head around the way to implement it.

Thanks,
Ognen

On Mon, Jun 29, 2015 at 5:09 PM, Ognen Duzlevski <og...@gmail.com>
wrote:

> Ok, reading the javax spec - it would appear that CacheLoader load() only
> gets invoked (automatically by the cache engine such as Ignite) only if the
> data is not available in cache.
>
> "
> * If the cache is configured to use read-through, and get would return null
> * because the entry is missing from the cache, the Cache's {@link
> CacheLoader}
> * is called in an attempt to load the entry.
> "
>
> Thanks!
>
> On Mon, Jun 29, 2015 at 3:32 PM, Ognen Duzlevski <
> ognen.duzlevski@gmail.com> wrote:
>
>> Dmitriy,
>>
>> Thanks. Just so I understand: I am looking at the CacheStore example and
>> the whole discussion on:
>> http://apacheignite.readme.io/v1.1/docs/persistent-store
>>
>> In that example the load() method takes a key which is then used
>> automatically to query the underlying SQL persistent store and the results
>> are served back to client.
>>
>> How about this scenario? I get the key in the load method and I first
>> check to see if my data is in cache. If it is, I serve the result from
>> cache, otherwise I go to persistent storage.
>>
>> Is this handled automatically by Ignite in the background and that's why
>> the load() method in the example pays no attention to the possibility of
>> the key/value already existing in memory? Or is that something I have to
>> explicitly check for in the load() implementation?
>>
>> Thanks!
>>
>> On Mon, Jun 29, 2015 at 2:53 PM, Dmitriy Setrakyan <dsetrakyan@apache.org
>> > wrote:
>>
>>>
>>> On Mon, Jun 29, 2015 at 12:38 PM, Ognen Duzlevski <
>>> ognen.duzlevski@gmail.com> wrote:
>>>
>>>> How about SQL queries on ignite? Do they end up being translated into
>>>> series of gets() and hence make use of the read-through CacheStore
>>>> implementation?
>>>>
>>>
>>> The answer is No. Queries are executed only on the in-memory data. To
>>> support what you are suggesting, we would have to execute a query on the
>>> database, get all the keys back, and then execute a bunch of cache.get(..)
>>> calls for each returned key. Overall, this would be very slow and that's
>>> why was not done for Ignite.
>>>
>>> Having said that, we can enhance our Hibernate integration to support
>>> this use case. I will file a ticket.
>>>
>>> D.
>>>
>>>
>>
>

Re: CacheStore load/get and SQL

Posted by Ognen Duzlevski <og...@gmail.com>.
Ok, reading the javax spec - it would appear that CacheLoader load() only
gets invoked (automatically by the cache engine such as Ignite) only if the
data is not available in cache.

"
* If the cache is configured to use read-through, and get would return null
* because the entry is missing from the cache, the Cache's {@link
CacheLoader}
* is called in an attempt to load the entry.
"

Thanks!

On Mon, Jun 29, 2015 at 3:32 PM, Ognen Duzlevski <og...@gmail.com>
wrote:

> Dmitriy,
>
> Thanks. Just so I understand: I am looking at the CacheStore example and
> the whole discussion on:
> http://apacheignite.readme.io/v1.1/docs/persistent-store
>
> In that example the load() method takes a key which is then used
> automatically to query the underlying SQL persistent store and the results
> are served back to client.
>
> How about this scenario? I get the key in the load method and I first
> check to see if my data is in cache. If it is, I serve the result from
> cache, otherwise I go to persistent storage.
>
> Is this handled automatically by Ignite in the background and that's why
> the load() method in the example pays no attention to the possibility of
> the key/value already existing in memory? Or is that something I have to
> explicitly check for in the load() implementation?
>
> Thanks!
>
> On Mon, Jun 29, 2015 at 2:53 PM, Dmitriy Setrakyan <ds...@apache.org>
> wrote:
>
>>
>> On Mon, Jun 29, 2015 at 12:38 PM, Ognen Duzlevski <
>> ognen.duzlevski@gmail.com> wrote:
>>
>>> How about SQL queries on ignite? Do they end up being translated into
>>> series of gets() and hence make use of the read-through CacheStore
>>> implementation?
>>>
>>
>> The answer is No. Queries are executed only on the in-memory data. To
>> support what you are suggesting, we would have to execute a query on the
>> database, get all the keys back, and then execute a bunch of cache.get(..)
>> calls for each returned key. Overall, this would be very slow and that's
>> why was not done for Ignite.
>>
>> Having said that, we can enhance our Hibernate integration to support
>> this use case. I will file a ticket.
>>
>> D.
>>
>>
>

Re: CacheStore load/get and SQL

Posted by Ognen Duzlevski <og...@gmail.com>.
Dmitriy,

Thanks. Just so I understand: I am looking at the CacheStore example and
the whole discussion on:
http://apacheignite.readme.io/v1.1/docs/persistent-store

In that example the load() method takes a key which is then used
automatically to query the underlying SQL persistent store and the results
are served back to client.

How about this scenario? I get the key in the load method and I first check
to see if my data is in cache. If it is, I serve the result from cache,
otherwise I go to persistent storage.

Is this handled automatically by Ignite in the background and that's why
the load() method in the example pays no attention to the possibility of
the key/value already existing in memory? Or is that something I have to
explicitly check for in the load() implementation?

Thanks!

On Mon, Jun 29, 2015 at 2:53 PM, Dmitriy Setrakyan <ds...@apache.org>
wrote:

>
> On Mon, Jun 29, 2015 at 12:38 PM, Ognen Duzlevski <
> ognen.duzlevski@gmail.com> wrote:
>
>> How about SQL queries on ignite? Do they end up being translated into
>> series of gets() and hence make use of the read-through CacheStore
>> implementation?
>>
>
> The answer is No. Queries are executed only on the in-memory data. To
> support what you are suggesting, we would have to execute a query on the
> database, get all the keys back, and then execute a bunch of cache.get(..)
> calls for each returned key. Overall, this would be very slow and that's
> why was not done for Ignite.
>
> Having said that, we can enhance our Hibernate integration to support this
> use case. I will file a ticket.
>
> D.
>
>

Re: CacheStore load/get and SQL

Posted by kevin <ke...@leonardo.com>.
dsetrakyan wrote
> Having said that, we can enhance our Hibernate integration to support this
> use case. I will file a ticket.
> 
> D.


Hi,

Is there a link to that ticket?

Thanks



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/CacheStore-load-get-and-SQL-tp582p1502.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: CacheStore load/get and SQL

Posted by Dmitriy Setrakyan <ds...@apache.org>.
On Mon, Jun 29, 2015 at 12:38 PM, Ognen Duzlevski <ognen.duzlevski@gmail.com
> wrote:

> How about SQL queries on ignite? Do they end up being translated into
> series of gets() and hence make use of the read-through CacheStore
> implementation?
>

The answer is No. Queries are executed only on the in-memory data. To
support what you are suggesting, we would have to execute a query on the
database, get all the keys back, and then execute a bunch of cache.get(..)
calls for each returned key. Overall, this would be very slow and that's
why was not done for Ignite.

Having said that, we can enhance our Hibernate integration to support this
use case. I will file a ticket.

D.