Cache pre-loading question

[franck102 <fr...@yahoo.com>]

Hi all, I am trying to figure out the best approach for pre-loading cache
entries from a SQL DB on startup.

I have read the recommendations to use a client node and a data streamer to
pre-load cache data on startup.

One issue I see with the approach however, compared with cache.loadCache(),
is that I can't seem to easily reuse the CacheStore.loadCache
implementations - is there any way to do so?

My (JDBC) cache store configuration contains logic to map the DB schema to
the cache schema; it seems that I would need to replicate that logic in the
loader client in order to call streamer.addData()?

Thanks!
Franck



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Cache pre-loading question

[franck102 <fr...@yahoo.com>]

IgniteCache has a loadCache method and that is what I am using currently, but
I have seen many recommendations here and in the docs to a streamer rather
than loadCache (because loadCache is not transactional, can lead to
out-of-sync data if the topology changes...), e.g.:

https://apacheignite.readme.io/docs/data-loading 
"Data streamers should be used to load large amount of data into caches at
any time, including pre-loading on startup."

I am currently using loadCache on a client node, which if I understand
correctly simply dispatches a loadCache task to all nodes in the cluster.
What I am trying to achieve is pretty simple - assuming for example a
10-node cluster with a partitioned cache with backups=1:
I want a client to send an instruction to the 2 relevant nodes (primary and
backup) to pre-load the cache, using logic & mappings defined in the cache
configuration; and I need some way to detect/recover from nodes failing in
the process.

I isn't very clear to me how I can achieve that though...

Franck



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Cache pre-loading question

[afedotov <al...@gmail.com>]

Hi,

By "custom" CacheStore I meant that you could implenent a CacheStore with
your custom read/write logic, but since you are using CacheJdbcPojoStore it
should be OK.

IgniteCache does have loadCache method.
Please take a look at
https://apacheignite.readme.io/docs/data-loading#section-ignitecacheloadcache

Kind regards,
Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Cache pre-loading question

[franck102 <fr...@yahoo.com>]

Hi Alex, what do you mean by custom? 

I am indeed using a CacheJdbcPojoStore, it has the method I need 
(loadCache(final IgniteBiInClosure<K, V> clo, @Nullable Object... args)) 

however I can't obtain a reference to the store from my ignite instance. 

I can try to build an instance of the store outside Ignite however that 
means (at best) maintaining its configuration separately from the rest of 
the Ignite config... 

What I'd like to do instead is: 


... however that loadCache method exists on the cache store, but not on the
cache :( 

Franck 




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Cache pre-loading question

[franck102 <fr...@yahoo.com>]

Hi Alex, what do you mean by custom?

I am indeed using a CacheJdbcPojoStore, it has the method I need
(loadCache(final IgniteBiInClosure<K, V> clo, @Nullable Object... args)) 

however I can't obtain a reference to the store from my ignite instance.

I can try to build an instance of the store outside Ignite however that
means (at best) maintaining its configuration separately from the rest of
the Ignite config... 

What I'd like to do instead is:



... however there is no IgniteCache.getStore API :(

Franck




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Cache pre-loading question

[afedotov <al...@gmail.com>]

Hi Franck,

Yes. You would need to implement the same mapping logic in your client
populating a cache via data streamer.
A custom CacheStore or CacheJdbcPojoStore should be an easier approach in
case of an underlying DB, especially if you need read-through/write-through
semantics as well.

Kind regards,
Alex



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/