Ignite 2.7.6: Memory Leak with Direct buffers in TCPCommunication SPI

[Mahesh Renduchintala <ma...@aline-consulting.com>]

Hi,

We have been searching for a cause of memory leak in ignite server nodes for many weeks now.
The memory leak exists and below is the scenario.

Scenario

  *   Scenario
     *   Our ignite servers have about 50GB of data. Two servers were baselined
     *   There are about 10 client nodes connected
     *   OffHeap memory - 64GB per node, 48GB heap (XMS, XMS parameters)
  *   Test
     *   start all 10 nodes
     *   wait for 10 nodes to be connected
     *   sleep for 30 mins
     *   Stop all 10 nodes (docker rm -f X)
     *   Loop 1 to 4 steps infinitely

    Observations

     *   With every iteration, about 1GB of memory disappears from the heap f servers
     *   After about 100 iterations, server nodes crash reporting OOM

Workaround

     *   The memory leak does not occur when directBuffer is made false as below. Meaning TCP buffers are only onHeap
        *

<property name="communicationSpi">
    <bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi">
        <property name="connectTimeout" value="90000"/>
        <property name="directBuffer" value="false"/>
        <property name="directSendBuffer" value="false"/>
        <property name="messageQueueLimit" value="10000"/>
        <property name="idleConnectionTimeout" value="#{315360000000L}"/>
        <property name="selectorSpins" value="2"/>
        <property name="usePairedConnections" value="true"/>
        <property name="filterReachableAddresses" value="true"/>
        <property name="socketWriteTimeout" value="20000"/>
    </bean>
</property>

Hope you can reproduce this problem at your end.
This occurs with any default configuration of ignite servers. This is why, I am not sending mine.

regards
Mahesh



                             *   The only work around we found is to use on

Re: Ignite 2.7.6: Memory Leak with Direct buffers in TCPCommunication SPI

[Ilya Kasnacheev <il...@gmail.com>]

Hello!

I can see that you have 19G of Object[] in heap. I'm not sure if it has any
relation to direct buffers.

Can you debug where these Object[]'s are referenced from?

As a side note, our test suites bring tens of thousands of nodes up and
down in one JVM and not face this issue.

Regards,
-- 
Ilya Kasnacheev


вс, 1 мар. 2020 г. в 08:12, Mahesh Renduchintala <
mahesh.renduchintala@aline-consulting.com>:

> Please see the attached jhist.
> In this condition one of the node consumed about 18 GB.
>

Re: Ignite 2.7.6: Memory Leak with Direct buffers in TCPCommunication SPI

[Mahesh Renduchintala <ma...@aline-consulting.com>]

Please see the attached jhist.
In this condition one of the node consumed about 18 GB.

Re: Ignite 2.7.6: Memory Leak with Direct buffers in TCPCommunication SPI

[Vladimir Pligin <vo...@yandex.ru>]

By the way it would be great if you share a couple of heap histograms between
restarts. Something like jmap -histo <pid> > histo. Is it possible?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Ignite 2.7.6: Memory Leak with Direct buffers in TCPCommunication SPI

[Vladimir Pligin <vo...@yandex.ru>]

Hi Manesh,

Thanks for sharing your case. We'll try to reproduce it.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/