You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by tysli2016 <To...@cityline.com.hk> on 2016/12/08 03:47:45 UTC
Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
We have 2 machines (M1, M2)
In M1 running a Ignite server (I1) and a Tomcat server (T1), which hosted 2
Ignite clients (C1a, C1b),
similarly in M2 running a Ignite server (I2) and a Tomcat server (T2), which
hosted 2 Ignite clients (C2a, C2b).
OutOfMemoryError were found in both T1 and T2 yesterday, we have a heap dump
from T1, but failed to get a heap dump from T2.
After the error, by using visor we found only T1, C1a, and C1b were still in
the cluster, all Ignite nodes on M2 were disconnected.
We've lost the Tomcat server log due to some reason.
From the heap dump of T1 we found 6
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor holding
total 1.2GB memory:
Class Name
| Shallow Heap | Retained Heap | Percentage
------------------------------------------------------------------------------------------------------------------------------------------
| | |
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81a78dc8 | 56 | 227,306,848 | 14.05%
'- java.util.HashMap @ 0x8423c790
| 48 | 227,306,792 | 14.05%
'- java.util.HashMap$Node[65536] @ 0xcd657f20
| 262,160 | 227,306,744 | 14.05%
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x8449bed0 | 56 | 227,299,144 | 14.05%
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81ae5c10 | 56 | 227,071,080 | 14.04%
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x8449f158 | 56 | 227,063,384 | 14.04%
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81adfa38 | 56 | 226,599,544 | 14.01%
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x8449f090 | 56 | 226,591,864 | 14.01%
org.apache.catalina.loader.WebappClassLoader @ 0x80167cd8
| 136 | 29,521,360 | 1.83%
org.apache.catalina.loader.WebappClassLoader @ 0x8363a078
| 136 | 28,471,928 | 1.76%
com.cityline.cps.admin.api.controller.LoginController @ 0x819c54d0
| 48 | 21,042,952 | 1.30%
org.apache.ignite.spi.discovery.tcp.ClientImpl @ 0x84477b20
| 96 | 14,456,360 | 0.89%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0xade21488 | 152 | 10,462,696 | 0.65%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0xae8ec420 | 152 | 9,800,264 | 0.61%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0x86aa66a0 | 152 | 9,677,848 | 0.60%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0x86ac6b58 | 152 | 9,634,288 | 0.60%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0x83013720 | 152 | 6,075,032 | 0.38%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0x86008f08 | 152 | 5,967,304 | 0.37%
class java.beans.ThreadGroupContext @ 0x81070c30 System Class
| 8 | 2,634,888 | 0.16%
org.hibernate.internal.SessionFactoryImpl @ 0x811ec9f0
| 136 | 2,406,176 | 0.15%
org.hibernate.internal.SessionFactoryImpl @ 0x85de4c98
| 136 | 2,082,128 | 0.13%
org.hibernate.internal.SessionFactoryImpl @ 0x83b30890
| 136 | 2,036,976 | 0.13%
org.apache.ignite.internal.GridKernalContextImpl @ 0x841d4090
| 248 | 1,599,680 | 0.10%
org.apache.ignite.internal.GridKernalContextImpl @ 0x818f6900
| 248 | 1,596,328 | 0.10%
java.net.URLClassLoader @ 0x800fe890
| 80 | 1,451,640 | 0.09%
org.apache.catalina.webresources.JarResourceSet @ 0x8019d4d0
| 72 | 1,302,400 | 0.08%
org.apache.ignite.internal.processors.cache.GridCacheSharedContext @
0x8427bbc0 | 72 | 1,242,936 | 0.08%
org.apache.ignite.internal.processors.cache.GridCacheSharedContext @
0x819c5e18 | 72 | 1,239,464 | 0.08%
org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager
@ 0x819d54e8 | 88 | 1,052,272 | 0.07%
org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager
@ 0x842cc828 | 88 | 1,052,272 | 0.07%
org.apache.tomcat.util.net.NioEndpoint @ 0x80780e38
| 288 | 1,052,136 | 0.07%
org.apache.tomcat.util.net.NioEndpoint @ 0x80666fb0
| 288 | 1,051,584 | 0.07%
org.springframework.beans.factory.support.DefaultListableBeanFactory @
0x80fe9e50 | 208 | 882,424 | 0.05%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0x82fb0d28 | 152 | 847,280 | 0.05%
org.apache.ignite.internal.processors.cache.GridCacheContext @
0x85fa7cd0 | 152 | 834,120 | 0.05%
org.springframework.beans.factory.support.DefaultListableBeanFactory @
0x857e33d0 | 208 | 725,632 | 0.04%
org.springframework.beans.factory.support.DefaultListableBeanFactory @
0x83b2fbb8 | 208 | 695,544 | 0.04%
org.springframework.security.access.method.DelegatingMethodSecurityMetadataSource
@ 0x81367d80| 24 | 583,376 | 0.04%
Total: 36 of 252,650 entries; 252,614 more
| | |
------------------------------------------------------------------------------------------------------------------------------------------
drilling into one of DynamicCacheDescriptor, found the memory was held by a
HashMap (java.util.HashMap @ 0x8423c790, java.util.HashMap$Node[65536] @
0xcd657f20), which seems to contain a bunch of CacheConfiguration objects:
Class Name
| Shallow Heap | Retained Heap | Percentage
-----------------------------------------------------------------------------------------------------------------------------------------------------------
| | |
java.util.HashMap$Node[65536] @ 0xcd657f20
| 262,160 | 227,306,744 | 14.05%
|- java.util.HashMap$Node @ 0xc559b860
| 32 | 38,520 | 0.00%
| |- java.util.HashMap$Node @ 0xcad62418
| 32 | 30,816 | 0.00%
| | |- java.util.HashMap$Node @ 0xcb6d56b8
| 32 | 23,112 | 0.00%
| | | |- java.util.HashMap$Node @ 0xce1be088
| 32 | 15,408 | 0.00%
| | | | |- java.util.HashMap$Node @ 0xd4e3dbe8
| 32 | 7,704 | 0.00%
| | | | | '- org.apache.ignite.configuration.CacheConfiguration @
0xd4e3be10 | 288 | 7,672 |
0.00%
| | | | | |-
org.apache.ignite.internal.processors.cache.GridCacheDefaultAffinityKeyMapper
@ 0xd4e3bfe0| 24 | 7,096 | 0.00%
| | | | | |- java.lang.String @ 0xd4e38260
ignite-marshaller-sys-cache | 24 |
96 | 0.00%
| | | | | |-
org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction @
0xd4e3bf90 | 40 | 80 | 0.00%
| | | | | |- java.util.HashSet @ 0xd4e3bf50
| 16 | 64 | 0.00%
| | | | | |-
javax.cache.configuration.FactoryBuilder$SingletonFactory @ 0xd4e3bf30
| 16 | 32 | 0.00%
| | | | | |-
org.apache.ignite.plugin.CachePluginConfiguration[0] @ 0xd4e3db98
| 16 | 16 | 0.00%
| | | | | '- Total: 6 entries
| | |
| | | | |- org.apache.ignite.configuration.CacheConfiguration @
0xce1be0a8 | 288 | 7,672 |
0.00%
| | | | | |-
org.apache.ignite.internal.processors.cache.GridCacheDefaultAffinityKeyMapper
@ 0xce1be2e8 | 24 | 7,096 | 0.00%
| | | | | |- java.lang.String @ 0xce1be228
ignite-marshaller-sys-cache | 24
| 96 | 0.00%
| | | | | |-
org.apache.ignite.cache.affinity.rendezvous.RendezvousAffinityFunction @
0xce1be298 | 40 | 80 | 0.00%
| | | | | |- java.util.HashSet @ 0xce1be1c8
| 16 | 64 | 0.00%
| | | | | |-
javax.cache.configuration.FactoryBuilder$SingletonFactory @ 0xce1be208
| 16 | 32 | 0.00%
| | | | | |- org.apache.ignite.plugin.CachePluginConfiguration[0] @
0xce1c2098 | 16 | 16 |
0.00%
| | | | | '- Total: 6 entries
| | |
| | | | '- Total: 2 entries
| | |
| | | |- org.apache.ignite.configuration.CacheConfiguration @
0xcb6d56d8 | 288 | 7,672
| 0.00%
| | | '- Total: 2 entries
| | |
| | |- org.apache.ignite.configuration.CacheConfiguration @ 0xcad62438
| 288 | 7,672 | 0.00%
| | '- Total: 2 entries
| | |
| |- org.apache.ignite.configuration.CacheConfiguration @ 0xc559b880
| 288 | 7,672 | 0.00%
| '- Total: 2 entries
| | |
|- java.util.HashMap$Node @ 0x8e45adf0
| 32 | 38,520 | 0.00%
|- java.util.HashMap$Node @ 0x983d3e70
| 32 | 38,520 | 0.00%
|- java.util.HashMap$Node @ 0x87b1c1c8
| 32 | 38,520 | 0.00%
|- java.util.HashMap$Node @ 0x8a0d6980
| 32 | 38,520 | 0.00%
|- java.util.HashMap$Node @ 0x97143f50
| 32 | 38,520 | 0.00%
|- java.util.HashMap$Node @ 0x9731b858
| 32 | 38,520 | 0.00%
|- java.util.HashMap$Node @ 0x906efbd8
| 32 | 38,520 | 0.00%
|- java.util.HashMap$Node @ 0x9735be90
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x8ebdf958
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x951d6be0
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x9061a5a8
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x97b80fa0
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x877bad58
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x873bb2d0
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x8d47e168
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x89e19d58
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x8711c5f0
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x980292b0
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0xb25e7ac8
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x9d3ddbe0
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x948dd4d0
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x8d7eb430
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0xc60fa120
| 32 | 30,816 | 0.00%
|- java.util.HashMap$Node @ 0x9cd18138
| 32 | 30,816 | 0.00%
'- Total: 25 of 23,850 entries; 23,825 more
| | |
-----------------------------------------------------------------------------------------------------------------------------------------------------------
and this is the shortest paths to the accumulation point for
java.util.HashMap @ 0x8423c790
Class Name
| Shallow Heap | Retained Heap
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| |
java.util.HashMap$Node[65536] @ 0xcd657f20
| 262,160 | 227,306,744
'- table java.util.HashMap @ 0x8423c790
| 48 | 227,306,792
'- rmtCfgs
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81a78dc8
| 56 | 227,306,848
'- value java.util.HashMap$Node @ 0xe066c018
| 32 | 32
'- [11] java.util.HashMap$Node[16] @ 0xe066bf48
| 80 | 272
'- table java.util.HashMap @ 0xe066bf08
| 48 | 320
'- cachesOnDisconnect
org.apache.ignite.internal.processors.cache.GridCacheProcessor @ 0x81bacb08
| 80 | 2,640
'- cacheProc
org.apache.ignite.internal.GridKernalContextImpl @ 0x818f6900
| 248 | 1,596,328
|- ctx org.apache.ignite.internal.IgniteKernal @
0x818f68a0
| 96 | 200
| |- ignite
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi @ 0x819c19c0
| 240 | 2,000
| | |- this$0
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker
@ 0x81b97128 tcp-comm-worker-#1%null% Thread| 144 | 2,640
| | |- this$0
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$3 @ 0x81b97098
| 16 | 16
| | '- Total: 2 entries
| |
| |- ignite
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi @ 0x81968500
| 256 | 104,176
| |- grid
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance @ 0x81967d28
| 88 | 344
| '- Total: 3 entries
| |
|- ctx
org.apache.ignite.internal.processors.clock.GridClockServer @ 0x819680e0
| 32 | 32
|- ctx
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager @
0x81968468 |
128 | 45,944
|- ctx
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor @
0x81b97c98 |
32 | 512
|- ctx
org.apache.ignite.internal.processors.rest.GridRestProcessor @ 0x81bad1a0
| 64 | 1,056
|- ctx
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor @
0x83009a50 | 48
| 2,008
|- ctx
org.apache.ignite.internal.managers.deployment.GridDeploymentManager @
0x818f69f8 |
48 | 2,496
'- Total: 7 entries
| |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by Rishi Yagnik <ri...@gmail.com>.
Hello Val,
We figure out the issue with the help of ignite engineers that is as
follows -
-DIGNITE_AFFINITY_HISTORY_SIZE=500
Thank you for all your help..
Rishi
On Fri, Jul 7, 2017 at 1:23 PM, ignite_user2016 <ri...@gmail.com>
wrote:
> Hello Val..
>
> Thanks, will try out and let you know ...
>
> On Fri, Jul 7, 2017 at 12:25 PM, vkulichenko [via Apache Ignite Users] <[hidden
> email] <http:///user/SendEmail.jtp?type=node&node=14495&i=0>> wrote:
>
>> Hi Rishi,
>>
>> ignite.cluster().nodes() will give the list of nodes and full information
>> about topology. Will this work for you?
>>
>> -Val
>>
>> ------------------------------
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-
>> suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p14492.html
>> To start a new topic under Apache Ignite Users, email [hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=14495&i=1>
>> To unsubscribe from Apache Ignite Users, click here.
>> NAML
>> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>
>
>
>
> --
> Rishi Yagnik
>
> ------------------------------
> View this message in context: Re: Ignite 1.6.0 suspected memory leak from
> DynamicCacheDescriptor
> <http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p14495.html>
> Sent from the Apache Ignite Users mailing list archive
> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
>
--
Rishi Yagnik
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by ignite_user2016 <ri...@gmail.com>.
Hello Val..
Thanks, will try out and let you know ...
On Fri, Jul 7, 2017 at 12:25 PM, vkulichenko [via Apache Ignite Users] <
ml+s70518n14492h6@n6.nabble.com> wrote:
> Hi Rishi,
>
> ignite.cluster().nodes() will give the list of nodes and full information
> about topology. Will this work for you?
>
> -Val
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-
> 6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p14492.html
> To start a new topic under Apache Ignite Users, email
> ml+s70518n1h85@n6.nabble.com
> To unsubscribe from Apache Ignite Users, click here
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=cmlzaGl5YWduaWtAZ21haWwuY29tfDF8MTMwNTI4OTg1Mw==>
> .
> NAML
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
Rishi Yagnik
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p14495.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by vkulichenko <va...@gmail.com>.
Hi Rishi,
ignite.cluster().nodes() will give the list of nodes and full information
about topology. Will this work for you?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p14492.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by ignite_user2016 <ri...@gmail.com>.
Hello Val,
Do you have any example on it ? would like to see how can we leverage that
piece of code for monitoring ?
Thanks for all your help..
On Thu, Jul 6, 2017 at 5:46 PM, vkulichenko [via Apache Ignite Users] <
ml+s70518n14430h84@n6.nabble.com> wrote:
> Tom,
>
> It sounds like these jobs are triggered pretty frequently. So why not
> create a small application that will create an embedded client once and
> also have an internal timer or cron based scheduler to execute jobs
> periodically. You can do the topology check within this application as
> well, instead of using Visor.
>
> -Val
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-
> 6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p14430.html
> To start a new topic under Apache Ignite Users, email
> ml+s70518n1h85@n6.nabble.com
> To unsubscribe from Apache Ignite Users, click here
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=cmlzaGl5YWduaWtAZ21haWwuY29tfDF8MTMwNTI4OTg1Mw==>
> .
> NAML
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
Rishi Yagnik
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p14491.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by vkulichenko <va...@gmail.com>.
Tom,
It sounds like these jobs are triggered pretty frequently. So why not create
a small application that will create an embedded client once and also have
an internal timer or cron based scheduler to execute jobs periodically. You
can do the topology check within this application as well, instead of using
Visor.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p14430.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by tysli2016 <To...@cityline.com.hk>.
Hi Val,
sorry for this late reply.
yes, we have a client node constantly joining and leaving topology.
does the new versions of Ignite have improvement on this issue?
the reason client node constantly joining and leaving topology is twofold:
1. we used a visor cli called by cron job to check there should be more than
2 server nodes, it send out an email alert otherwise. it runs every 10
minutes.
it collects the server count like this:
IGNITE_COUNT=`/usr/local/apache-ignite-fabric-1.6.0-bin/bin/ignitevisorcmd.sh
-e="'open
-cpath=/usr/local/apache-ignite-fabric-1.6.0-bin/config/default-config.xml;node;c'"
|grep -e "Server " |wc -l`
2. we have another cron job to do some cleanup for the application, which
also connects to the topology and leave after done, also runs every few
minutes.
many thanks
Tom
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p14370.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by vkulichenko <va...@gmail.com>.
Hi,
It looks like there is client node constantly joining and leaving topology.
What is the reason for this?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p12894.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by tysli2016 <To...@cityline.com.hk>.
reproduced the OOME, the heap dump here
https://drive.google.com/drive/folders/0BwY2dxDlRYhBMEhmckpWeHg1bjg?usp=sharing
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p12529.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by tysli2016 <To...@cityline.com.hk>.
Thanks for your effort in patiently and promptly replies, we are trying to
reproduce the issue, will keep you posted.
Tom
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9669.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by vkulichenko <va...@gmail.com>.
Tom,
I am not sure there is a memory leak at all because I have no idea what you
code is doing. I'm just guessing here, but without success so far. Looking
through code didn't give anything useful as well, it looks correct. So if
you need my help further, please provide something that will help me to
reproduce the issue. That's the only way for me to investigate this further.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9637.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by tysli2016 <To...@cityline.com.hk>.
How about this OOME?
http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-GridDhtPartitionMap2-td9504.html
We found it on another client node in the same cluster, however it seems to
exhibit a different pattern of memory leak.
- Tom
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9636.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by vkulichenko <va...@gmail.com>.
Well, I need to reproduce it to investigate further. I tried to dig up in the
code, but do not see anything that can lead to memory leak, and even to so
many instances of CacheConfiguration in this map. Is it possible for you to
create a reproducer for this? That would be really helpful.
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9633.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by tysli2016 <To...@cityline.com.hk>.
> Are you sure that all of them are in this map? What is the size of rmtCfgs
map?
This map contains about 1/6 of the total CacheConfigurations, the size of
rmtCfg map is 227,306,792:
Class Name
| Shallow Heap | Retained Heap
-----------------------------------------------------------------------------------------------------------------------------
| |
java.util.HashMap$Node[65536] @ 0xcd657f20
| 262,160 | 227,306,744
'- table java.util.HashMap @ 0x8423c790
| 48 | 227,306,792
'- rmtCfgs
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81a78dc8 | 56 | 227,306,848
The Leak Suspects report shows there are 6 DynamicCacheDescriptors, each
size about 216MB:
https://drive.google.com/open?id=0BwY2dxDlRYhBbllWZ3pEMW1Tc00
All 6 DynamicCacheDescriptors exhibit the same pattern, they all contains a
whole lot of CacheConfigurations in a map, the numbers does add up:
Class Name
| Objects | Shallow Heap | Retained Heap
-----------------------------------------------------------------------------------------------------------------------------
| | |
java.util.HashMap$Node[]
| 33,252 | 10,886,584 | >= 1,446,877,296
java.util.HashMap
| 205,140 | 9,846,720 | >= 1,446,860,856
java.util.HashMap$Node
| 1,116,555 | 35,729,760 | >= 1,441,725,448
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor
| 6 | 336 | >= 681,000,984
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor
| 6 | 336 | >= 680,977,904
org.apache.ignite.configuration.CacheConfiguration
| 88,421 | 25,465,248 | >= 677,386,632
org.apache.ignite.configuration.CacheConfiguration
| 88,418 | 25,464,384 | >= 677,363,648
> Actually this map can be non-empty only on a node which is not fully
> started yet. Basically, when a new node joins a topology, it collects the
> configuration from all nodes for validation check, does the check and
> cleans the map. All this is part of the join process. Is this the case?
> How many nodes do you have?
There are 2 client nodes associated with the heap dump.
But they should already joined the cluster, cause these 2 client nodes could
be found in the topology with the visor cli. Would these CacheConfiguration
objects created for disconnect process as well? because I found they lead to
a GridCacheProcessor.cachesOnDisconnect:
Class Name
| Shallow Heap | Retained Heap
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| |
java.util.HashMap$Node[65536] @ 0xcd657f20
| 262,160 | 227,306,744
'- table java.util.HashMap @ 0x8423c790
| 48 | 227,306,792
'- rmtCfgs
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81a78dc8
| 56 | 227,306,848
'- value java.util.HashMap$Node @ 0xe066c018
| 32 | 32
'- [11] java.util.HashMap$Node[16] @ 0xe066bf48
| 80 | 272
'- table java.util.HashMap @ 0xe066bf08
| 48 | 320
'- cachesOnDisconnect
org.apache.ignite.internal.processors.cache.GridCacheProcessor @ 0x81bacb08
| 80 | 2,640
> To be honest, it's very hard to get to the bottom without a heap dump...
> Is it possible to reproduce in a standalone test without sensitive data?
Yes, we understand that fixes cannot be completely concluded without a
faithfully bug reproduction, we are trying to reproduce the symptom on
another machines. You know some OOMEs are so intermittent and quite time
consuming to reproduce them.
And we might also approach the problem the other way round, try to find what
might cause the symptom so that we might be able to reproduce it easier.
How does Ignite handle multiple connection event?
Does Ignite also create CacheConfiguration on disconnect?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9623.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by vkulichenko <va...@gmail.com>.
Are you sure that all of them are in this map? What is the size of rmtCfgs
map?
Actually this map can be non-empty only on a node which is not fully started
yet. Basically, when a new node joins a topology, it collects the
configuration from all nodes for validation check, does the check and cleans
the map. All this is part of the join process. Is this the case? How many
nodes do you have?
To be honest, it's very hard to get to the bottom without a heap dump... Is
it possible to reproduce in a standalone test without sensitive data?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9615.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by tysli2016 <To...@cityline.com.hk>.
ic, so the embedded instance is the IgniteKernal object which implement the
Ignite interface.
> So these are instances are not really used by Ignite, but are saved
> somewhere, most likely in your code. Can you use heap dump to trace it?
Yes, there are a whole lot of CacheConfiguration objects created, but it's
not from my code.
Frankly I think Ignite is having a very well designed API with with I don't
have to code much to use it, and I don't have a good reason to create
CacheConfiguration objects. What my code do is call Ignition.start() with a
IgniteConfiguration object like this:
IgniteConfiguration configuration = new IgniteConfiguration();
configuration.setGridLogger(new Slf4jLogger());
configuration.setDiscoverySpi(discoverySpi);
configuration.setCommunicationSpi(commumicationSpi);
Ignition.setClientMode(true);
Ignite ignite = Ignition.start(configuration);
And what I found from the heap dump is those CacheConfiguration objects are
not saved "somewhere", but in a java.util.HashMap @ 0x8423c790, which is
ultimately held by a GridKernalContextImpl:
Class Name
| Shallow Heap | Retained Heap
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| |
java.util.HashMap$Node[65536] @ 0xcd657f20
| 262,160 | 227,306,744
'- table java.util.HashMap @ 0x8423c790
| 48 | 227,306,792
'- rmtCfgs
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81a78dc8
| 56 | 227,306,848
'- value java.util.HashMap$Node @ 0xe066c018
| 32 | 32
'- [11] java.util.HashMap$Node[16] @ 0xe066bf48
| 80 | 272
'- table java.util.HashMap @ 0xe066bf08
| 48 | 320
'- cachesOnDisconnect
org.apache.ignite.internal.processors.cache.GridCacheProcessor @ 0x81bacb08
| 80 | 2,640
'- cacheProc
org.apache.ignite.internal.GridKernalContextImpl @ 0x818f6900
| 248 | 1,596,328
|- ctx org.apache.ignite.internal.IgniteKernal @
0x818f68a0
| 96 | 200
| |- ignite
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi @ 0x819c19c0
| 240 | 2,000
| | |- this$0
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker
@ 0x81b97128 tcp-comm-worker-#1%null% Thread| 144 | 2,640
| | |- this$0
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$3 @ 0x81b97098
| 16 | 16
| | '- Total: 2 entries
| |
| |- ignite
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi @ 0x81968500
| 256 | 104,176
| |- grid
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance @ 0x81967d28
| 88 | 344
| '- Total: 3 entries
| |
|- ctx
org.apache.ignite.internal.processors.clock.GridClockServer @ 0x819680e0
| 32 | 32
|- ctx
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager @
0x81968468 |
128 | 45,944
|- ctx
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor @
0x81b97c98 |
32 | 512
|- ctx
org.apache.ignite.internal.processors.rest.GridRestProcessor @ 0x81bad1a0
| 64 | 1,056
|- ctx
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor @
0x83009a50 | 48
| 2,008
|- ctx
org.apache.ignite.internal.managers.deployment.GridDeploymentManager @
0x818f69f8 |
48 | 2,496
'- Total: 7 entries
| |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> May be you somehow start and stop your caches periodically?
No, and I don't have a good reason to do so. The Ignite object once created
from Ignition.start() is able to (re)connect to servers in the cluster in
case of temporary disconnection. So my code only create one Ignite object
from Ignition.start() once, and only call Ignite.close() when the
application server is about to stop.
And just out of curiosity, what if one actually did start and stop caches
periodically? If they are legitimate operations then should not lead to
memory leak.
Tom
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9582.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by vkulichenko <va...@gmail.com>.
Embedded instance is the one started with Ignition.start().
I look at the histogram and see that there are more than 88000 instances of
CacheConfiguration in each application, while the number of caches is only
6. So these are instances are not really used by Ignite, but are saved
somewhere, most likely in your code. Can you use heap dump to trace it?
May be you somehow start and stop your caches periodically?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9575.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by tysli2016 <To...@cityline.com.hk>.
What do you mean by "embedded instances"? if it's org.apache.ignite.Ignite
object then we have two, one for each .war application deployed.
And what do you mean by "not properly stopped and / or disconnected"? The
application call org.apache.ignite.Ignite.close() only when Tomcat shutdown.
What if there are intermittent network failures?
There are 2 GridKernalContextImpl found in the heap:
Class Name | Objects | Shallow
Heap | Retained Heap
-------------------------------------------------------------------------------------------
.*GridKernalContextImpl.* | |
|
org.apache.ignite.internal.GridKernalContextImpl| 1 |
248 | >= 1,596,360
org.apache.ignite.internal.GridKernalContextImpl| 1 |
248 | >= 1,599,712
Total: 2 entries (23,992 filtered) | 2 |
496 |
-------------------------------------------------------------------------------------------
you might find the fully expanded histogram here:
https://drive.google.com/open?id=0BwY2dxDlRYhBR3pnYUM1cjZiNDg
thx, Tom
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9551.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by vkulichenko <va...@gmail.com>.
It seems to me you have a lot of embedded instances that are not properly
stopped and/or disconnected. Can this be the case. How many instances of
GridKernalContextImpl do you have in heap?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9543.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by tysli2016 <To...@cityline.com.hk>.
and we have the iptables open on ports 30000-50000, wondering if it's related
to the symtom.
because it shows `cachesOnDisconnect
org.apache.ignite.internal.processors.cache.GridCacheProcessor` holds the
objects.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9520.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by tysli2016 <To...@cityline.com.hk>.
> How many caches do you have?
there are only 2 caches
> Any idea why you have so many CacheConfiguration objects? Who holds
> references to them?
the shortest paths to the accumulation point for java.util.HashMap @
0x8423c790 shows who holds the references, as you can see below, it's an
org.apache.ignite.internal.GridKernalContextImpl @ 0x818f6900, which is held
by some other ignite spi / internal objects
Class Name
| Shallow Heap | Retained Heap
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
| |
java.util.HashMap$Node[65536] @ 0xcd657f20
| 262,160 | 227,306,744
'- table java.util.HashMap @ 0x8423c790
| 48 | 227,306,792
'- rmtCfgs
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor @
0x81a78dc8
| 56 | 227,306,848
'- value java.util.HashMap$Node @ 0xe066c018
| 32 | 32
'- [11] java.util.HashMap$Node[16] @ 0xe066bf48
| 80 | 272
'- table java.util.HashMap @ 0xe066bf08
| 48 | 320
'- cachesOnDisconnect
org.apache.ignite.internal.processors.cache.GridCacheProcessor @ 0x81bacb08
| 80 | 2,640
'- cacheProc
org.apache.ignite.internal.GridKernalContextImpl @ 0x818f6900
| 248 | 1,596,328
|- ctx org.apache.ignite.internal.IgniteKernal @
0x818f68a0
| 96 | 200
| |- ignite
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi @ 0x819c19c0
| 240 | 2,000
| | |- this$0
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker
@ 0x81b97128 tcp-comm-worker-#1%null% Thread| 144 | 2,640
| | |- this$0
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$3 @ 0x81b97098
| 16 | 16
| | '- Total: 2 entries
| |
| |- ignite
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi @ 0x81968500
| 256 | 104,176
| |- grid
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance @ 0x81967d28
| 88 | 344
| '- Total: 3 entries
| |
|- ctx
org.apache.ignite.internal.processors.clock.GridClockServer @ 0x819680e0
| 32 | 32
|- ctx
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager @
0x81968468 |
128 | 45,944
|- ctx
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor @
0x81b97c98 |
32 | 512
|- ctx
org.apache.ignite.internal.processors.rest.GridRestProcessor @ 0x81bad1a0
| 64 | 1,056
|- ctx
org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor @
0x83009a50 | 48
| 2,008
|- ctx
org.apache.ignite.internal.managers.deployment.GridDeploymentManager @
0x818f69f8 |
48 | 2,496
'- Total: 7 entries
| |
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9519.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by vkulichenko <va...@gmail.com>.
How many caches do you have?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9517.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by vkulichenko <va...@gmail.com>.
Any idea why you have so many CacheConfiguration objects? Who holds
references to them?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9515.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by tysli2016 <To...@cityline.com.hk>.
Hi Val, sorry I m afraid I cannot provide the heap dump because it might
contain some sensitive data.
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9507.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by tysli2016 <To...@cityline.com.hk>.
Thx Val for your reply, let me check if I can give you the .hprof file later.
This is the dominator tree showing all 23,850 items from
java.util.HashMap$Node[65535] @ 0xcd657f20, It's too many of them so I just
expand some of them and they are all holding CacheConfiguration objects.
https://drive.google.com/file/d/0BwY2dxDlRYhBWXR1NHdWelMtY0E/view
We actually found that there are 88,421 + 88418 CacheConfiguration objects
holding 677,386,632 + 677,363,648 retained heap
Class Name
| Objects | Shallow Heap | Retained Heap
-----------------------------------------------------------------------------------------------------------------------------
| | |
java.util.HashMap$Node[]
| 33,252 | 10,886,584 | >= 1,446,877,296
java.util.HashMap
| 205,140 | 9,846,720 | >= 1,446,860,856
java.util.HashMap$Node
| 1,116,555 | 35,729,760 | >= 1,441,725,448
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor
| 6 | 336 | >= 681,000,984
org.apache.ignite.internal.processors.cache.DynamicCacheDescriptor
| 6 | 336 | >= 680,977,904
org.apache.ignite.configuration.CacheConfiguration
| 88,421 | 25,465,248 | >= 677,386,632
org.apache.ignite.configuration.CacheConfiguration
| 88,418 | 25,464,384 | >= 677,363,648
org.apache.ignite.internal.processors.cache.GridCacheDefaultAffinityKeyMapper|
88,419 | 2,122,056 | >= 627,421,920
org.apache.ignite.internal.processors.cache.GridCacheDefaultAffinityKeyMapper|
88,416 | 2,121,984 | >= 627,400,632
org.apache.ignite.internal.util.GridReflectionCache
| 88,426 | 2,829,632 | >= 625,349,432
org.apache.ignite.internal.util.GridReflectionCache
| 88,423 | 2,829,536 | >= 625,328,216
org.apache.ignite.internal.util.GridBoundedConcurrentLinkedHashMap
| 176,854 | 14,148,320 | >= 620,153,264
org.apache.ignite.internal.util.GridBoundedConcurrentLinkedHashMap
| 176,860 | 14,148,800 | >= 620,098,736
org.jsr166.ConcurrentLinkedHashMap$Segment[]
| 176,862 | 14,151,184 | >= 581,610,816
org.jsr166.ConcurrentLinkedHashMap$Segment[]
| 176,868 | 14,151,664 | >= 581,554,984
org.jsr166.ConcurrentLinkedHashMap$Segment
| 2,830,348 | 158,499,488 | >= 567,459,640
org.jsr166.ConcurrentLinkedHashMap$Segment
| 2,830,444 | 158,504,864 | >= 567,403,328
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync
| 5,661,690 | 271,761,120 | >= 362,348,560
org.jsr166.ConcurrentLinkedHashMap$HashEntry[]
| 2,830,444 | 137,029,536 | >= 137,037,776
org.jsr166.ConcurrentLinkedHashMap$HashEntry[]
| 2,830,348 | 137,024,840 | >= 137,033,280
java.lang.String
| 979,752 | 23,514,048 | >= 92,539,136
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock
| 5,661,691 | 90,587,056 | >= 90,587,064
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock
| 5,661,691 | 90,587,056 | >= 90,587,064
java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter|
5,661,691 | 90,587,056 | >= 90,587,056
char[]
| 984,330 | 80,884,824 | >= 80,884,824
org.apache.catalina.loader.WebappClassLoader
| 8 | 1,088 | >= 58,022,056
java.util.concurrent.ConcurrentSkipListMap$HeadIndex
| 171 | 5,472 | >= 52,983,616
java.util.concurrent.ConcurrentSkipListMap
| 58 | 2,784 | >= 52,871,176
java.lang.Class
| 24,003 | 213,864 | >= 48,210,712
java.util.HashSet
| 187,879 | 3,006,064 | >= 46,999,120
java.util.concurrent.ConcurrentSkipListMap$Node
| 10,534 | 252,816 | >= 37,540,176
java.util.Collections$UnmodifiableSet
| 2,857 | 45,712 | >= 33,773,632
long[]
| 3,094 | 33,291,192 | >= 33,291,192
java.lang.Object[]
| 398,466 | 11,845,896 | >= 32,896,712
java.util.concurrent.ConcurrentLinkedQueue
| 5,956 | 142,944 | >= 31,574,704
java.util.concurrent.ConcurrentLinkedQueue$Node
| 12,248 | 293,952 | >= 31,431,752
Total: 36 of 23,994 entries; 23,958 more
| 44,511,835 | 1,617,452,472 |
-----------------------------------------------------------------------------------------------------------------------------
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9501.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: Ignite 1.6.0 suspected memory leak from DynamicCacheDescriptor
Posted by vkulichenko <va...@gmail.com>.
CacheConfiguration objects consume 0% though, so there should be something
else. Is there any understanding what is actually consuming the space? Can
you upload the whole .hprof file somewhere?
-Val
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-1-6-0-suspected-memory-leak-from-DynamicCacheDescriptor-tp9443p9496.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.