Re: Increase the indexing speed while loading the cache from an RDBMS

[Michael Cherkasov <mi...@gmail.com>]

>I've checked the RDBMS perf metrics. I do not see any spike in the CPU or
memory usage.
>Is there anything in particular that could cause a bottleneck from the DB
perspective ? The DB in use is a Postgres DB.

if you use cache store, each node will read the full table you want to
load, but only records which belong to a node will be saved(based on data
affinity), so Ignite will do N parallel read of the same table, where N is
number of nodes.
So if you have 50 nodes, then you should use data streamer, so one node
will read data and stream them to the cluster.
However if you have relatively small amount of nodes, it's fine to use
CacheStore to load data.

>Is there a sample/best practice reference code for Using Datastreamers to
directly query the DB to preload the cache ?
> My intention is to use Spring Data via IgniteRepositories. In order to
use an Ignite repository, I would have to preload the cache.
>And now, to preload the cache, I would have to query the DB. A bit of a
chicken and egg problem.
I would say, use JDBC and Ignite Data Streamer, it's one time action that
you don't need to repeat often, you can even java separate app to do this.

Thanks,
Mike.

чт, 27 авг. 2020 г. в 14:01, Srikanta Patanjali <p....@gmail.com>:

> Thanks Anton & Mikhail for your responses.
>
> I'll try to disable the WAL and test the cache preload performance. Also
> will execute a JFR to capture some info.
>
> @Mikhail I've checked the RDBMS perf metrics. I do not see any spike in
> the CPU or memory usage. Is there anything in particular that could cause a
> bottleneck from the DB perspective ? The DB in use is a Postgres DB.
>
> @Anton Is there a sample/best practice reference code for Using
> Datastreamers to directly query the DB to preload the cache ? My intention
> is to use Spring Data via IgniteRepositories. In order to use an Ignite
> repository, I would have to preload the cache. And now, to preload the
> cache, I would have to query the DB. A bit of a chicken and egg problem.
>
> Regards,
> Srikanta
>
> On Thu, Aug 27, 2020 at 7:54 PM Mikhail Cherkasov <mc...@gridgain.com>
> wrote:
>
>> btw, might be bottleneck is your RDBMS, it can just stream data slowly,
>> slower then ignite can save, if make sense to check this version too.
>>
>> On Thu, Aug 27, 2020 at 10:46 AM akurbanov <an...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Which API are you using to load the entries and what is the node
>>> configuration? I would recommend to share the configs and try utilizing
>>> the
>>> data streamer.
>>>
>>> https://apacheignite.readme.io/docs/data-streamers
>>>
>>> I would recommend recording a JFR to find where the VM spends most time.
>>>
>>> Best regards,
>>> Anton
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>>
>>
>> --
>> Thanks,
>> Mikhail.
>>
>