You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by ignite_user2016 <ri...@gmail.com> on 2017/07/26 04:30:25 UTC
heap dump on ignite servers
Hello Igniters / Val,
Since last few weeks, we are seeing out of memory issues on ignite servers,
today I was able to capture the heapdumps and that is pointing to following
classes -
Our version of Ignite is 1.7.0.
15 instances of
"org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache",
loaded by "sun.misc.Launcher$AppClassLoader @ 0xc0029998" occupy 453,929,512
(76.41%) bytes.
Biggest instances:
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc426b868 - 34,931,744 (5.88%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc2f94e28 - 34,745,904 (5.85%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc4a0ffc8 - 34,745,896 (5.85%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc2e41e10 - 34,642,904 (5.83%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc100a648 - 34,376,920 (5.79%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc3da2ac0 - 34,376,288 (5.79%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc32991b0 - 34,376,208 (5.79%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc427d798 - 34,376,008 (5.79%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc57cbd28 - 34,270,952 (5.77%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc0a22ae8 - 34,191,040 (5.76%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc0d8ade8 - 34,190,960 (5.76%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc38a3618 - 34,190,920 (5.76%) bytes.
•org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
@ 0xc50d0ca8 - 33,900,352 (5.71%) bytes.
Keywords
sun.misc.Launcher$AppClassLoader @ 0xc0029998
org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
we are running ignite on 1G memory and dont use ignite for any computing.
Our configuration is based on ip cluster that is as follows -
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:util="http://www.springframework.org/schema/util"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/util
http://www.springframework.org/schema/util/spring-util.xsd">
<bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="gridName" value="GRID_NAME"/>
<property name="peerClassLoadingEnabled" value="true"/>
<property name="clientMode" value="false"/>
<property name="gridLogger">
<bean class="org.apache.ignite.logger.log4j2.Log4J2Logger">
<constructor-arg type="java.lang.String"
value="config/log4j2.xml"/>
</bean>
</property>
<property name="includeEventTypes">
<list>
<util:constant
static-field="org.apache.ignite.events.EventType.EVT_CLIENT_NODE_RECONNECTED"/>
</list>
</property>
<property name="cacheConfiguration">
<list>
</list>
</property>
<property name="discoverySpi">
<bean
class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
<property name="ipFinder">
<bean
class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
<property name="addresses">
<list>
<value>HOST_IP1:47500..47509</value>
<value>HOST_IP2:47500..47509</value>
</list>
</property>
</bean>
</property>
</bean>
</property>
</bean>
</beans>
what am I missing here ?
thanks for all your help..
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/heap-dump-on-ignite-servers-tp15679.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: heap dump on ignite servers
Posted by ignite_user2016 <ri...@gmail.com>.
any help here ?
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/heap-dump-on-ignite-servers-tp15679p16089.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: heap dump on ignite servers
Posted by ignite_user2016 <ri...@gmail.com>.
I also applied the IGNITE_AFFINITY_HISTORY_SIZE= 500 on Ignite 1.7 version
but ignite heap keeps rising and eventually runs out of memory.
our use case is more with spring caching and we use ignite for spring 2nd
level cache.
Is there any setting on cache that can reduce the memory footprint ? OR is
there some kind of leak here ?
Please provide your guidance here.
Thanks..
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/heap-dump-on-ignite-servers-tp15679p16060.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: heap dump on ignite servers
Posted by ignite_user2016 <ri...@gmail.com>.
Thank you Andrew.. we are upgrading Ignite to 2.0 so hopefully we wont have
such issues.
On Wed, Jul 26, 2017 at 10:35 AM, Andrew Mashenkov [via Apache Ignite
Users] <ml...@n6.nabble.com> wrote:
> Hi,
>
> Looks like we have a more optimal code in latest versions. See IGNITE-4154
> [1]
> Try to update to >= 1.8 version.
>
> As a workaround you can try to reduce discovery history size with JVM
> option IGNITE_AFFINITY_HISTORY_SIZE, which is 500 by default.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-4154
>
> On Wed, Jul 26, 2017 at 7:30 AM, ignite_user2016 <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=15695&i=0>> wrote:
>
>> Hello Igniters / Val,
>>
>> Since last few weeks, we are seeing out of memory issues on ignite
>> servers,
>> today I was able to capture the heapdumps and that is pointing to
>> following
>> classes -
>>
>> Our version of Ignite is 1.7.0.
>>
>> 15 instances of
>> "org.apache.ignite.internal.processors.affinity.GridAffinity
>> AssignmentCache",
>> loaded by "sun.misc.Launcher$AppClassLoader @ 0xc0029998" occupy
>> 453,929,512
>> (76.41%) bytes.
>>
>> Biggest instances:
>> •org.apache.ignite.internal.processors.affinity.GridAffinity
>> AssignmentCache
>> @ 0xc426b868 - 34,931,744 (5.88%) bytes.
>> •org.apache.ignite.internal.processors.affinity.GridAffinity
>> AssignmentCache
>> @ 0xc2f94e28 - 34,745,904 (5.85%) bytes.
>> •org.apache.ignite.internal.processors.affinity.GridAffinity
>> AssignmentCache
>> @ 0xc4a0ffc8 - 34,745,896 (5.85%) bytes.
>> •org.apache.ignite.internal.processors.affinity.GridAffinity
>> AssignmentCache
>> @ 0xc2e41e10 - 34,642,904 (5.83%) bytes.
>> •org.apache.ignite.internal.processors.affinity.GridAffinity
>> AssignmentCache
>> @ 0xc100a648 - 34,376,920 (5.79%) bytes.
>> •org.apache.ignite.internal.processors.affinity.GridAffinity
>> AssignmentCache
>> @ 0xc3da2ac0 - 34,376,288 (5.79%) bytes.
>> •org.apache.ignite.internal.processors.affinity.GridAffinity
>> AssignmentCache
>> @ 0xc32991b0 - 34,376,208 (5.79%) bytes.
>> •org.apache.ignite.internal.processors.affinity.GridAffinity
>> AssignmentCache
>> @ 0xc427d798 - 34,376,008 (5.79%) bytes.
>> •org.apache.ignite.internal.processors.affinity.GridAffinity
>> AssignmentCache
>> @ 0xc57cbd28 - 34,270,952 (5.77%) bytes.
>> •org.apache.ignite.internal.processors.affinity.GridAffinity
>> AssignmentCache
>> @ 0xc0a22ae8 - 34,191,040 (5.76%) bytes.
>> •org.apache.ignite.internal.processors.affinity.GridAffinity
>> AssignmentCache
>> @ 0xc0d8ade8 - 34,190,960 (5.76%) bytes.
>> •org.apache.ignite.internal.processors.affinity.GridAffinity
>> AssignmentCache
>> @ 0xc38a3618 - 34,190,920 (5.76%) bytes.
>> •org.apache.ignite.internal.processors.affinity.GridAffinity
>> AssignmentCache
>> @ 0xc50d0ca8 - 33,900,352 (5.71%) bytes.
>>
>>
>> Keywords
>> sun.misc.Launcher$AppClassLoader @ 0xc0029998
>> org.apache.ignite.internal.processors.affinity.GridAffinityA
>> ssignmentCache
>>
>> we are running ignite on 1G memory and dont use ignite for any computing.
>>
>> Our configuration is based on ip cluster that is as follows -
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>>
>>
>>
>>
>> <beans xmlns="http://www.springframework.org/schema/beans"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> xmlns:util="http://www.springframework.org/schema/util"
>> xsi:schemaLocation="
>> http://www.springframework.org/schema/beans
>> http://www.springframework.org/schema/beans/spring-beans.xsd
>> http://www.springframework.org/schema/util
>> http://www.springframework.org/schema/util/spring-util.xsd">
>>
>>
>>
>>
>> <bean id="ignite.cfg"
>> class="org.apache.ignite.configuration.IgniteConfiguration">
>>
>> <property name="gridName" value="GRID_NAME"/>
>>
>>
>> <property name="peerClassLoadingEnabled" value="true"/>
>>
>> <property name="clientMode" value="false"/>
>>
>>
>> <property name="gridLogger">
>> <bean class="org.apache.ignite.logger.log4j2.Log4J2Logger">
>> <constructor-arg type="java.lang.String"
>> value="config/log4j2.xml"/>
>> </bean>
>> </property>
>>
>>
>> <property name="includeEventTypes">
>> <list>
>> <util:constant
>> static-field="org.apache.ignite.events.EventType.EVT_CLIENT_
>> NODE_RECONNECTED"/>
>> </list>
>> </property>
>>
>>
>> <property name="cacheConfiguration">
>> <list>
>>
>>
>>
>>
>>
>>
>>
>>
>> </list>
>> </property>
>>
>>
>>
>> <property name="discoverySpi">
>> <bean
>> class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
>> <property name="ipFinder">
>>
>>
>> <bean
>> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDi
>> scoveryVmIpFinder">
>>
>> <property name="addresses">
>> <list>
>>
>> <value>HOST_IP1:47500..47509</value>
>> <value>HOST_IP2:47500..47509</value>
>> </list>
>> </property>
>> </bean>
>> </property>
>> </bean>
>> </property>
>>
>> </bean>
>> </beans>
>>
>>
>>
>> what am I missing here ?
>>
>> thanks for all your help..
>>
>>
>>
>>
>> --
>> View this message in context: http://apache-ignite-users.705
>> 18.x6.nabble.com/heap-dump-on-ignite-servers-tp15679.html
>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>
>
>
>
> --
> Best regards,
> Andrey V. Mashenkov
> Regards, Andrew.
>
>
> ------------------------------
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-users.70518.x6.nabble.com/heap-dump-
> on-ignite-servers-tp15679p15695.html
> To start a new topic under Apache Ignite Users, email
> ml+s70518n1h85@n6.nabble.com
> To unsubscribe from Apache Ignite Users, click here
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=cmlzaGl5YWduaWtAZ21haWwuY29tfDF8MTMwNTI4OTg1Mw==>
> .
> NAML
> <http://apache-ignite-users.70518.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
--
Rishi Yagnik
--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/heap-dump-on-ignite-servers-tp15679p15701.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
Re: heap dump on ignite servers
Posted by Andrey Mashenkov <an...@gmail.com>.
Hi,
Looks like we have a more optimal code in latest versions. See IGNITE-4154
[1]
Try to update to >= 1.8 version.
As a workaround you can try to reduce discovery history size with JVM
option IGNITE_AFFINITY_HISTORY_SIZE, which is 500 by default.
[1] https://issues.apache.org/jira/browse/IGNITE-4154
On Wed, Jul 26, 2017 at 7:30 AM, ignite_user2016 <ri...@gmail.com>
wrote:
> Hello Igniters / Val,
>
> Since last few weeks, we are seeing out of memory issues on ignite servers,
> today I was able to capture the heapdumps and that is pointing to following
> classes -
>
> Our version of Ignite is 1.7.0.
>
> 15 instances of
> "org.apache.ignite.internal.processors.affinity.
> GridAffinityAssignmentCache",
> loaded by "sun.misc.Launcher$AppClassLoader @ 0xc0029998" occupy
> 453,929,512
> (76.41%) bytes.
>
> Biggest instances:
> •org.apache.ignite.internal.processors.affinity.
> GridAffinityAssignmentCache
> @ 0xc426b868 - 34,931,744 (5.88%) bytes.
> •org.apache.ignite.internal.processors.affinity.
> GridAffinityAssignmentCache
> @ 0xc2f94e28 - 34,745,904 (5.85%) bytes.
> •org.apache.ignite.internal.processors.affinity.
> GridAffinityAssignmentCache
> @ 0xc4a0ffc8 - 34,745,896 (5.85%) bytes.
> •org.apache.ignite.internal.processors.affinity.
> GridAffinityAssignmentCache
> @ 0xc2e41e10 - 34,642,904 (5.83%) bytes.
> •org.apache.ignite.internal.processors.affinity.
> GridAffinityAssignmentCache
> @ 0xc100a648 - 34,376,920 (5.79%) bytes.
> •org.apache.ignite.internal.processors.affinity.
> GridAffinityAssignmentCache
> @ 0xc3da2ac0 - 34,376,288 (5.79%) bytes.
> •org.apache.ignite.internal.processors.affinity.
> GridAffinityAssignmentCache
> @ 0xc32991b0 - 34,376,208 (5.79%) bytes.
> •org.apache.ignite.internal.processors.affinity.
> GridAffinityAssignmentCache
> @ 0xc427d798 - 34,376,008 (5.79%) bytes.
> •org.apache.ignite.internal.processors.affinity.
> GridAffinityAssignmentCache
> @ 0xc57cbd28 - 34,270,952 (5.77%) bytes.
> •org.apache.ignite.internal.processors.affinity.
> GridAffinityAssignmentCache
> @ 0xc0a22ae8 - 34,191,040 (5.76%) bytes.
> •org.apache.ignite.internal.processors.affinity.
> GridAffinityAssignmentCache
> @ 0xc0d8ade8 - 34,190,960 (5.76%) bytes.
> •org.apache.ignite.internal.processors.affinity.
> GridAffinityAssignmentCache
> @ 0xc38a3618 - 34,190,920 (5.76%) bytes.
> •org.apache.ignite.internal.processors.affinity.
> GridAffinityAssignmentCache
> @ 0xc50d0ca8 - 33,900,352 (5.71%) bytes.
>
>
> Keywords
> sun.misc.Launcher$AppClassLoader @ 0xc0029998
> org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache
>
> we are running ignite on 1G memory and dont use ignite for any computing.
>
> Our configuration is based on ip cluster that is as follows -
>
> <?xml version="1.0" encoding="UTF-8"?>
>
>
>
>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xmlns:util="http://www.springframework.org/schema/util"
> xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
> http://www.springframework.org/schema/util
> http://www.springframework.org/schema/util/spring-util.xsd">
>
>
>
>
> <bean id="ignite.cfg"
> class="org.apache.ignite.configuration.IgniteConfiguration">
>
> <property name="gridName" value="GRID_NAME"/>
>
>
> <property name="peerClassLoadingEnabled" value="true"/>
>
> <property name="clientMode" value="false"/>
>
>
> <property name="gridLogger">
> <bean class="org.apache.ignite.logger.log4j2.Log4J2Logger">
> <constructor-arg type="java.lang.String"
> value="config/log4j2.xml"/>
> </bean>
> </property>
>
>
> <property name="includeEventTypes">
> <list>
> <util:constant
> static-field="org.apache.ignite.events.EventType.EVT_
> CLIENT_NODE_RECONNECTED"/>
> </list>
> </property>
>
>
> <property name="cacheConfiguration">
> <list>
>
>
>
>
>
>
>
>
> </list>
> </property>
>
>
>
> <property name="discoverySpi">
> <bean
> class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="ipFinder">
>
>
> <bean
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.
> TcpDiscoveryVmIpFinder">
>
> <property name="addresses">
> <list>
>
> <value>HOST_IP1:47500..47509</value>
> <value>HOST_IP2:47500..47509</value>
> </list>
> </property>
> </bean>
> </property>
> </bean>
> </property>
>
> </bean>
> </beans>
>
>
>
> what am I missing here ?
>
> thanks for all your help..
>
>
>
>
> --
> View this message in context: http://apache-ignite-users.
> 70518.x6.nabble.com/heap-dump-on-ignite-servers-tp15679.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>
--
Best regards,
Andrey V. Mashenkov