Re: Loading cache from Oracle Table

[nithin91 <ni...@franklintempleton.com>]

Hi Prasad,

Is there a improvement in performance. If so can you please share the sample
code as i am also looking for a similar solution.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Loading cache from Oracle Table

[yossi <yo...@gmail.com>]

Hi Prasad, 

Any update on this? I have a similar use case when I have to load 10million
of records  from a mysql database table but I keep getting the error 
"(jvm-pause-detector-worker) Possible too long JVM pause" even ignite
persistence is enabled (which should write in disk the data when the data
does not fit in RAM).
 
I followed  https://apacheignite.readme.io/docs/jvm-and-system-tuning
<https://apacheignite.readme.io/docs/jvm-and-system-tuning>   tips for
tuning GC and I have also increased the heap size.

For testing purposes I am using a single node with the following JVM
settings:

-server -Xms8g -Xmx8g -XX:+AlwaysPreTouch -XX:+UseG1GC
-XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC 

So, I would like to know how you manage to use multiple threads to retrieve
rows by batches and how to use partition aware for loading big tables
because it is not clear at all to me. It would be easier If I see an
example.

Thank you!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Loading cache from Oracle Table

[Ilya Kasnacheev <il...@gmail.com>]

Hello!

20 minutes for 40 mln entries seems in reasonable ballpark to me. What's
the CPU usage on Ignite nodes at that time?

Regards,
-- 
Ilya Kasnacheev


ср, 19 февр. 2020 г. в 19:43, Prasad Bhalerao <prasadbhalerao1983@gmail.com
>:

> Hi,
> I am using partition aware loading to load big tables using 4-8 threads on
> each nodes.
> I have around 16 caches. 2 caches have 38-40 million entries. It takes
> around approx 21 minutes to load the caches on 6 nodes.
> I have kept the backup count to 2.
>
> Will share the code tomorrow.
>
> Regards,
> Prasad
>
>
> On Tue 18 Feb, 2020, 6:08 PM nithin91 <
> nithinbharadwaj.govindaraju@franklintempleton.com wrote:
>
>> Hi Prasad,
>>
>> Is there a improvement in performance. If so can you please share the
>> sample
>> code as i am also looking for a similar solution.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Re: Loading cache from Oracle Table

[Prasad Bhalerao <pr...@gmail.com>]

Hi,
I am using partition aware loading to load big tables using 4-8 threads on
each nodes.
I have around 16 caches. 2 caches have 38-40 million entries. It takes
around approx 21 minutes to load the caches on 6 nodes.
I have kept the backup count to 2.

Will share the code tomorrow.

Regards,
Prasad


On Tue 18 Feb, 2020, 6:08 PM nithin91 <
nithinbharadwaj.govindaraju@franklintempleton.com wrote:

> Hi Prasad,
>
> Is there a improvement in performance. If so can you please share the
> sample
> code as i am also looking for a similar solution.
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>