What are typical load time for millions of records

["steve.hostettler" <st...@gmail.com>]

I'm currently trying to improve the load time. I would like to have some
feedbacks from other Ignite users as for the actual load performances.

In my setup, I have a DB in the same subnet with an Ignite node with 8
cores. It loads 20,000,000 records in 1h with the standard loadCache
implementation. Which amounts to 5,555 rows per seconds = 694 rows per
seconds and per threads.

I know it is very dependent upon the actual topology, size of the objects,
etc but I like to understand whether   my performances are reasonable. I'm
asking that because I must load 180,000,000 every day and this sounds not
feasible with 5,555 rows per node on the grid.





--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/What-are-typical-load-time-for-millions-of-records-tp9648.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: What are typical load time for millions of records

["steve.hostettler" <st...@gmail.com>]

Hi,

I am using the partition capability with multiple threads to improve the
data loading, but referring to my other post : I see that the performance
are not constant with respect to the size of the cache. 

Regards



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/What-are-typical-load-time-for-millions-of-records-tp9648p9682.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: What are typical load time for millions of records

[vkulichenko <va...@gmail.com>]

Hi,

You're right, this heavily depends on a lot of factors and it's almost
impossible to tell if these numbers are good or bad. For large data sets you
can try partition aware data loading [1]. It can give performance
improvement and also is pretty scalable. I.e. you can make it even better by
adding nodes and/or loading in multiple parallel threads.

[1]
https://apacheignite.readme.io/docs/data-loading#section-partition-aware-data-loading

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/What-are-typical-load-time-for-millions-of-records-tp9648p9654.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.