Unstable cluster with high load

[kevin <ke...@leonardo.com>]

Hi,

We've noticed that while we are writing many records into the datagrid
(total is about 4 million), our cluster of 2 nodes becomes unstable.
Any general tips on what would be good things to try? Eg. try to reduce size
of cache objects, tune some Ignite settings? What appears to be the cause of
this?



Thanks,
Kevin



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Unstable-cluster-with-high-load-tp4289.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Unstable cluster with high load

[vkulichenko <va...@gmail.com>]

Kevin,

You should avoid having that large heaps. If your heap space is more than
10-12G, you will very likely get long GC pauses. Consider using off-heap
memory for your data: https://apacheignite.readme.io/docs/off-heap-memory

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Unstable-cluster-with-high-load-tp4289p4343.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Unstable cluster with high load

[kevin <ke...@leonardo.com>]

We also tried a quick hack with 1 node, to store the objects in a
ConcurrentHashMap instead of the datagrid and the peak memory usage was
about 18gb, which is a significant difference from 45gb when using the
datagrid. Does this sound normal? In both cases, there's plenty of free
memory, so there probably wasn't a lot of GC going on though.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Unstable-cluster-with-high-load-tp4289p4331.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Unstable cluster with high load

[kevin <ke...@leonardo.com>]

We tried to rule out memory issues by running the following tests:
Ran with 1 node with 128gb heap memory successfully. Peak memory usage was
about 45gb.
Ran with 2 nodes with 128gb each. Memory usage was about 20gb for each node
when they disconnected with above error.



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Unstable-cluster-with-high-load-tp4289p4330.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Unstable cluster with high load

[vkulichenko <va...@gmail.com>]

Hi Kevin,

One of the most common reasons for this is memory issues. Can you check if
you're running out of memory or having long GC pauses on either of nodes?

-Val



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Unstable-cluster-with-high-load-tp4289p4300.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.