Offheap memory consumption + enabled persistence

[mikle-a <an...@inbox.ru>]

Hi!

Problem

I've noticed some weird behavior while working with enabled persistence
(native storage): I put some amount of entries to the cluster and delete
them afterwards. When I use the same set of keys each time, I see that
*DataRegionMetrics.getOffheapUsedSize* reaches some limit and don't grow
anymore, i.e. memory is reused. When I do the same action, but with random
keys, I see that mentioned above offheap memory metric always grows and
never stops.

In pictures, test with static keys:
<http://apache-ignite-users.70518.x6.nabble.com/file/t2766/static-keys.png> 

test with random keys (running ~ 10 hours):
<http://apache-ignite-users.70518.x6.nabble.com/file/t2766/random-keys.png> 

As I can see, if I disable persistence, everything works as with static
keys.

I really wonder why enabling persistence causes such excessive offheap
consumption, could you explain please?

Code to reproduce

Here are 3 scenarios: 
1) TestPersistenceStaticKeys: persistence enabled, static keys - offheap
reaches some limit and stops growing, as expected
2) TestPersistenceRandomKeys: persistence enabled, random keys - offheap
always grow, why?
3) TestRandomKeysWithoutPersistence: persistence disabled, random keys -
offheap reaches limit and stops growing, as expected

package com.test.ignite.memory;

import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.Ignition;
import org.apache.ignite.configuration.CacheConfiguration;
import org.apache.ignite.configuration.DataRegionConfiguration;
import org.apache.ignite.configuration.DataStorageConfiguration;
import org.apache.ignite.configuration.IgniteConfiguration;
import org.bson.types.ObjectId;

import java.util.concurrent.ThreadLocalRandom;
import java.util.function.Function;
import java.util.stream.IntStream;

public class OffheapConsumption {

    public static class TestPersistenceStaticKeys {
        public static void main(String[] args) {
            runTest(true, i -> i);
        }
    }

    public static class TestPersistenceRandomKeys {
        public static void main(String[] args) {
            runTest(true, i -> ThreadLocalRandom.current().nextInt());
        }
    }

    public static class TestRandomKeysWithoutPersistence {
        public static void main(String[] args) {
            runTest(false, i -> ThreadLocalRandom.current().nextInt());
        }
    }

    private static void runTest(boolean persistenceEnabled,
Function<Integer, Integer> idProvider) {
        final DataRegionConfiguration defaultDataRegionConfig = new
DataRegionConfiguration();
        defaultDataRegionConfig.setMetricsEnabled(true);
        defaultDataRegionConfig.setPersistenceEnabled(persistenceEnabled);

        final DataStorageConfiguration storageConfiguration = new
DataStorageConfiguration();
       
storageConfiguration.setDefaultDataRegionConfiguration(defaultDataRegionConfig);

        final CacheConfiguration<ObjectId, String> cacheConfig = new
CacheConfiguration<>();
        cacheConfig.setName("cache");

        final Ignite server1 = Ignition.start(new IgniteConfiguration()
                .setDataStorageConfiguration(storageConfiguration)
                .setCacheConfiguration(cacheConfig)
                .setIgniteInstanceName("server1"));

        final Ignite server2 = Ignition.start(new IgniteConfiguration()
                .setDataStorageConfiguration(storageConfiguration)
                .setCacheConfiguration(cacheConfig)
                .setIgniteInstanceName("server2"));

        server1.cluster().active(true);

        final Ignite client = Ignition.start(new
IgniteConfiguration().setClientMode(true));
        final IgniteCache<Integer, String> cache = client.cache("cache");


        for (int i = 0; i < 100; i++) {
            //print data regions used offheap size
            server1.dataRegionMetrics().stream()
                    .filter(drm -> drm.getName().equals("default"))
                    .findFirst()
                    .ifPresent(dataRegionMetrics ->
                            System.out.println(String.format("Data region
'%s' used off heap size = %s",
                            dataRegionMetrics.getName(),
dataRegionMetrics.getOffheapUsedSize())));

            //put entries
            System.out.println("put entries");
            IntStream.range(0, 1000)
                    .forEach(n -> cache.put(idProvider.apply(n), new
String(new byte[1024])));

            //clean all
            System.out.println("clear entries");
            cache.clear();
        }

        client.close();
        server1.close();
        server2.close();
    }
}



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Offheap memory consumption + enabled persistence

[Dmitriy Govorukhin <dm...@gmail.com>]

Mikle,

A pure data is a size of(keyByte+ValueBytes), as I can see you have Key=int
and Value=String(1024)
4+1024 bytes this is your pure data size which will be stored in the cache
(of course Ignite has overhead for save data ~45 bytes ) 

In internal Ignite has a structure named "FreeList" for tracking free space,
the in-memory region has only one FreeList for the region, the persistence
region has FreeList per partition in the cache, this means persistence case
have move overhead for internal structure. This structure has 256 buckets in
each bucket we have a page, so this means we may have 256*pageSize(4kb) -
1mb overhead. As default value cache has 1024 partition, therefore we may
have 1GB  overhead if we putting data in each partition repeatedly.
in custody, persistence mode may have more overhead for work with data, it
related to the internal structure. 
As a check, you can persistence create a cache with 1 partition and the
result will be the same as in-memory.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Offheap memory consumption + enabled persistence

[mikle-a <an...@inbox.ru>]

Hi Dmitry!

I didn't understand about "pure data size" :( Could you please specify how
to calculate it?

Despite this, I've added monitoring for mentioned cache folders and retested
case with random keys and enabled persistence.

Overall test chart:
<http://apache-ignite-users.70518.x6.nabble.com/file/t2766/large-frame.png> 

Last 30 min:
<http://apache-ignite-users.70518.x6.nabble.com/file/t2766/small-frame.png> 

As I can see, cache directory grows along with "offheap memory used".

Thanks in advance for your help!



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Offheap memory consumption + enabled persistence

[Dmitriy Govorukhin <dm...@gmail.com>]

Hi Mikle,

Could you please check what happening with files, they too grow as a metric?
You need to check the size of files in
{IGNITE_HOME}/work/db/cache-{cacheName}/* during random key upload and clean
up. Also, calculate data size on each iteration and print to log.
My assumption in that we have some leak in tracing free space in partition
and each iteration tries to allocate new pages.
In the end, please provide the case with random keys.
- Pure data size on each iteration
- Folder cache size for each iteration
- Chart for DataRegionMetrics.getOffheapUsedSize



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Offheap memory consumption + enabled persistence

[ezhuravlev <e....@gmail.com>]

>But I am still confused. Did I understood properly that each node should 
have enough memory to store full data set? 
No, it's not the case. It's just preallocating this memory with internal
structures and shows it in the metrics.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Offheap memory consumption + enabled persistence

[mikle-a <an...@inbox.ru>]

First, thanks a lot for your reply!

But I am still confused. Did I understood properly that each node should
have enough memory to store full data set?

Previously I thought that the main idea of partitioning is to distribute
data among servers. For example, to distribute 300GB of data among 3 servers
having 100GB each. Now it turns that each server should have 300GB, or I've
understood it wrong?



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Offheap memory consumption + enabled persistence

[ezhuravlev <e....@gmail.com>]

Hi,

Growth of this metric will be stopped after all partitions(1024 by default)
allocate memory for maximum count of entries that can be stored in it
simultaneously. It will take some time for 1024 partitions(not 100
iterations). You can check it by setting partitions count to smaller size:
        cacheConfig.setAffinity(new RendezvousAffinityFunction(false, 8));

Evgenii



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/