Cache loading performance

[Prasad Bhalerao <pr...@gmail.com>]

Hi,

I am loading the data from oracle table to ignite cache using
CaceStore.loadCache. I am loading the data in multithreaded way. I am using
16 threads to fetch the data from table and push it to cache and the select
SQL is simple query with where in clause.

To load 100K record it is taking 45 seconds. It is taking most of the time
for fetching the data from db. it is dev env and db server is not that fast
and this issue can be addressed.

I have more than 300 million rows in my table. Now I am wondering if this
db issue gets resolved by using powerful hardware and 10 gbps link, how
long will it take to load 300 million records in grid. The table size is 5
GB.

Has anyone in this group done benchmarking to load hundreds of millions of
rows from oracle or any 3rd party db to ignite grid?
Can you guys share your experience?

Thanks,
Prasad

Re: Cache loading performance

[Evgenii Zhuravlev <e....@gmail.com>]

Hi,

I'm not sure that such kind of benchmarking make sense. I've loaded much
more than 300 million rows, but my numbers won't give anything to you
because it depends only on objects size, nodes count, etc.

From my experience, I would say that the bottleneck in your case will be
the CPU or I/O on your store(DB)'s machine.

Regards,
Evgenii

2018-03-07 19:25 GMT+03:00 Prasad Bhalerao <pr...@gmail.com>:

> Hi,
>
> I am loading the data from oracle table to ignite cache using
> CaceStore.loadCache. I am loading the data in multithreaded way. I am using
> 16 threads to fetch the data from table and push it to cache and the select
> SQL is simple query with where in clause.
>
> To load 100K record it is taking 45 seconds. It is taking most of the time
> for fetching the data from db. it is dev env and db server is not that fast
> and this issue can be addressed.
>
> I have more than 300 million rows in my table. Now I am wondering if this
> db issue gets resolved by using powerful hardware and 10 gbps link, how
> long will it take to load 300 million records in grid. The table size is 5
> GB.
>
> Has anyone in this group done benchmarking to load hundreds of millions of
> rows from oracle or any 3rd party db to ignite grid?
> Can you guys share your experience?
>
> Thanks,
> Prasad
>