You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Naveen <na...@gmail.com> on 2021/04/19 11:46:32 UTC
Ignite node heap gradually increasing and not getting stabilized
Hello All
we are using Ignite 2.8.1 with native persistence enabled.
We have cluster with 5 nodes and did a cluster restart recently. Few days
after the cluster restart, we are seeing node heap memory is constantly
increasing and not getting stabilized.
<http://apache-ignite-users.70518.x6.nabble.com/file/t1478/heap_memory.gif>
And not much increase in offheap memory though
<http://apache-ignite-users.70518.x6.nabble.com/file/t1478/offheap-memory.gif>
And, nothing has been changed, no change in the load like writes and reads
are pretty much the same.
We have not pre-loaded (or warming-up memory) the data after the cluster
restart, it does increase the latency and disk IO etc, but does it increase
the heap utilization so much.
what else I can verify to know the root cause of this heap increase and how
can we make this stable, not seeing any other side affects of this heap
utilization though. Its just that its almost nearing the max heap set for
the node
Thanks
Naveen
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite node heap gradually increasing and not getting
stabilized
Posted by Naveen <na...@gmail.com>.
Hi
Can someone confirm what I am doing this correct for forcing the GC.
I did this on one of our lower environments,
$ sudo -u ignite jcmd 90341 GC.heap_info
90341:
garbage-first heap total 2101248K, *used 1261559K* [0x00000005c0000000,
0x0000000800000000)
region size 2048K, 172 young (352256K), 3 survivors (6144K)
Metaspace used 77269K, capacity 79504K, committed 80892K, reserved
1122304K
class space used 7363K, capacity 8207K, committed 8576K, reserved
1048576K
******************************
then ran the below command
******************************
$ sudo -u ignite jcmd 90341 GC.run
90341:
Command executed successfully
********************************
$ sudo -u ignite jcmd 90341 GC.heap_info
90341:
garbage-first heap total 2097152K, *used 530847K* [0x00000005c0000000,
0x0000000800000000)
region size 2048K, 9 young (18432K), 0 survivors (0K)
Metaspace used 77195K, capacity 79390K, committed 80892K, reserved
1122304K
class space used 7352K, capacity 8187K, committed 8576K, reserved
1048576K
The only difference I see here is
used 1261559K (before)
used 530847K (after)
does that mean, it did collect some objects and heap memory freed up ?
or any better ways of forcing GC ?
Thanks
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite node heap gradually increasing and not getting stabilized
Posted by Stephen Darlington <st...@gridgain.com>.
Not a direct answer, but there were a lot of improvements to thin clients in Ignite 2.10, specifically in cases where there are many connections.
> On 26 May 2021, at 09:44, Naveen <na...@gmail.com> wrote:
>
> Just reopening this issue.
> Unfortunately I could not analyze the heap dump which is of 25GB, so I could
> not do any analysis with this.
>
> What else can go wrong, however I could see 500 to 800 active thin client
> connections on each node, can this cause this
>
> I see our heap memory utilization is more than the off-heap memory and its
> hosted on VMs, anything I can verify on the underlying hardware ?
>
> Thanks
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite node heap gradually increasing and not getting
stabilized
Posted by Naveen <na...@gmail.com>.
Just reopening this issue.
Unfortunately I could not analyze the heap dump which is of 25GB, so I could
not do any analysis with this.
What else can go wrong, however I could see 500 to 800 active thin client
connections on each node, can this cause this
I see our heap memory utilization is more than the off-heap memory and its
hosted on VMs, anything I can verify on the underlying hardware ?
Thanks
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite node heap gradually increasing and not getting stabilized
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
You need to find out which objects keep references to these, eventually
tracing their usage to some large component of Ignite or your application.
Regards,
--
Ilya Kasnacheev
ср, 21 апр. 2021 г. в 13:37, Naveen <na...@gmail.com>:
> HI Ilya
>
> We hardly have 20 tables, data around 10 to 15 millions
>
> Thin clients also around 50 to 100 on each node, not many and load is also
> not so high.
>
> Here are the details about CipherSuite's objects
>
> <http://apache-ignite-users.70518.x6.nabble.com/file/t1478/ssl_objects.png>
>
>
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1478/GridH2Processors.png>
>
>
> what can go wrong after looking at this
>
> Thanks
>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Ignite node heap gradually increasing and not getting
stabilized
Posted by Naveen <na...@gmail.com>.
HI Ilya
We hardly have 20 tables, data around 10 to 15 millions
Thin clients also around 50 to 100 on each node, not many and load is also
not so high.
Here are the details about CipherSuite's objects
<http://apache-ignite-users.70518.x6.nabble.com/file/t1478/ssl_objects.png>
<http://apache-ignite-users.70518.x6.nabble.com/file/t1478/GridH2Processors.png>
what can go wrong after looking at this
Thanks
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite node heap gradually increasing and not getting stabilized
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
How many tables do you have? I don't see why you there are so many.
I'm not sure who holds all those CipherSuite's. Can you check?
If you have a ton of thin client connections with SSL which are doing MERGE
on many tables, I guess this may be OKish though.
Regards,
--
Ilya Kasnacheev
вт, 20 апр. 2021 г. в 16:11, Naveen <na...@gmail.com>:
> HI All
>
> We have analyzed the heap dump, some of the observations are
>
> what can we change for this, is there any configurations we can set to get
> rid of these leaks
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1478/tool_analysis1.gif>
>
>
> We do use SQL API, especially MERGE for upserts, anything we can do for h3
> engine
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1478/tool_h2_report.gif>
>
>
> We see 12% for H2GridTable and 14% for security/ssl/CipherSuite. any
> pointers here, are these expected or can we tuned further
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1478/tool_overall.gif>
>
>
> Thanks
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: Ignite node heap gradually increasing and not getting
stabilized
Posted by Naveen <na...@gmail.com>.
HI All
We have analyzed the heap dump, some of the observations are
what can we change for this, is there any configurations we can set to get
rid of these leaks
<http://apache-ignite-users.70518.x6.nabble.com/file/t1478/tool_analysis1.gif>
We do use SQL API, especially MERGE for upserts, anything we can do for h3
engine
<http://apache-ignite-users.70518.x6.nabble.com/file/t1478/tool_h2_report.gif>
We see 12% for H2GridTable and 14% for security/ssl/CipherSuite. any
pointers here, are these expected or can we tuned further
<http://apache-ignite-users.70518.x6.nabble.com/file/t1478/tool_overall.gif>
Thanks
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite node heap gradually increasing and not getting stabilized
Posted by Ilya Kasnacheev <il...@gmail.com>.
Hello!
Did you try to trigger a full GC and see what is the low level of heap
usage?
Please try collecting a heap dump from the node after some heap growth, and
checking it with Eclipse MAT for example.
Regards,
--
Ilya Kasnacheev
пн, 19 апр. 2021 г. в 14:46, Naveen <na...@gmail.com>:
> Hello All
>
> we are using Ignite 2.8.1 with native persistence enabled.
>
> We have cluster with 5 nodes and did a cluster restart recently. Few days
> after the cluster restart, we are seeing node heap memory is constantly
> increasing and not getting stabilized.
>
> <http://apache-ignite-users.70518.x6.nabble.com/file/t1478/heap_memory.gif>
>
>
> And not much increase in offheap memory though
>
> <
> http://apache-ignite-users.70518.x6.nabble.com/file/t1478/offheap-memory.gif>
>
>
> And, nothing has been changed, no change in the load like writes and reads
> are pretty much the same.
> We have not pre-loaded (or warming-up memory) the data after the cluster
> restart, it does increase the latency and disk IO etc, but does it increase
> the heap utilization so much.
>
> what else I can verify to know the root cause of this heap increase and how
> can we make this stable, not seeing any other side affects of this heap
> utilization though. Its just that its almost nearing the max heap set for
> the node
>
> Thanks
> Naveen
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>