You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Neil Wightman <ne...@intergral.com> on 2016/03/09 11:16:24 UTC
Off Heap cache using lots of heap memory
Hi All,
I have been trying out ignite for the past few weeks but I am hitting a
strange problem. I dont know if this problem is in my code or ignite.
Currently I have an OFFHEAP_TIERED cache with ~3 million entries, max
off heap size of 4Gb, but this is showing some very high heap memory
usage. The cache stores data for 1 hour then expires it and constantly
streams data into this cache via the dataStreamer.addData API.
The problem seems to be with this specific expiry policy configuration :
cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new
CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );
- with this expiry policy the heap memory usage for the ignite process
is 976Mb, without this expiry policy set its just 156Mb.
The actual off heap memory usage is only 557Mb (assuming the cache
metrics are correct).
I have generated two heap dumps and the memory difference is all in the
GridCacheTtlManager which is retaining a heap of ~889Mb according to
Eclipses MAT.
I have ran my tests with Ignite 1.5.0 Final and 1.6.0 SNAPSHOT from git
(Monday 7th March) and both show the same issue.
The problem with this large heap usage is that I am now getting very
large GC pauses which I cant stop without disabling the expiry policy.
My main question is am I doing something wrong? Is the Expiry Policy
really supposed to use so much heap memory compared to the off heap
cache size?
Thanks in advance,
Neil
---- config definition ----
final CacheConfiguration<MetricKey, AggregatePack> cacheConfiguration =
new CacheConfiguration<>( "cacheName" );
cacheConfiguration.setBackups( 1 );
cacheConfiguration.setCacheMode( CacheMode.PARTITIONED );
cacheConfiguration.setIndexedTypes( CustomKey.class, CustomData.class );
cacheConfiguration.setMemoryMode( CacheMemoryMode.OFFHEAP_TIERED );
cacheConfiguration.setOffHeapMaxMemory( 4L * 1024L * 1024L * 1024L );
cacheConfiguration.setEvictionPolicy( new LruEvictionPolicy<>( 6_000_000
) );
cacheConfiguration.setSwapEnabled( true );
cacheConfiguration.setStatisticsEnabled( true );
// This causes large memory usage
cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new
CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );
Re: Off Heap cache using lots of heap memory
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi Neil,
Thank you for code snippet. I was able to reproduce the problem and created
a ticket - https://issues.apache.org/jira/browse/IGNITE-2833
Hope we will be able to fix it soon.
Vladimir.
On Fri, Mar 11, 2016 at 10:13 AM, Neil Wightman <neil_wightman@intergral.com
> wrote:
> Hi Vladimir,
>
> It took me a while to make a small test case but I now have one (see below)
>
> With the setExpiryPolicyFactory method enabled the heap after operation
> 983Mb, with the setExpiryPolicyFactory commented out it is just 85Mb.
>
> Simple run this class file and once you see the console message attach
> JVisualVM, then spam the "Perform GC" button.
>
> Also without the expiry time set the test runs with -Xm512m and with
> expiry the test needs -Xmx2024m.
>
> Thanks
> Neil
>
> -- IgniteExpiryIssue.java --
> package de.wightman.neil.apache.ignite.test;
>
> import javax.cache.configuration.FactoryBuilder;
> import javax.cache.expiry.CreatedExpiryPolicy;
> import javax.cache.expiry.Duration;
> import org.apache.ignite.Ignite;
> import org.apache.ignite.IgniteCache;
> import org.apache.ignite.IgniteDataStreamer;
> import org.apache.ignite.Ignition;
> import org.apache.ignite.cache.CacheMemoryMode;
> import org.apache.ignite.cache.CacheMode;
> import org.apache.ignite.cache.eviction.lru.LruEvictionPolicy;
> import org.apache.ignite.cache.query.annotations.QuerySqlField;
> import org.apache.ignite.configuration.CacheConfiguration;
>
> public class IgniteExpiryIssue
> {
>
> public static void main( final String[] args ) throws
> InterruptedException
> {
> final long start = System.currentTimeMillis();
> // configures Tcp discovery for local host only
> try (final Ignite ignite = Ignition.start( "LocalConfig.xml" ))
> {
>
> final IgniteCache<CustomKey, CustomValue> cache =
> ignite.getOrCreateCache( cacheConfig() );
> final IgniteDataStreamer<Object, Object> metricStreamer =
> ignite.dataStreamer( "TestCacheName" );
>
> final long counter = 1000;
> final long counter2 = 49;
> final long minutes = 60;
>
> final long fakeBaseTime = System.currentTimeMillis();
>
> for( long k = 0; k < minutes; k++ )
> {
> final long ts = fakeBaseTime + (minutes * 60 * 1000);
> for( long i = 0; i < counter; i++ )
> {
> for( long j = 0; j < counter2; j++ )
> {
> final CustomKey key = new CustomKey( ts, i, j );
>
> final CustomValue value = new CustomValue( 1.0,
> 1.0, 1.0, 1.0, 10 );
> metricStreamer.addData( key, value );
> }
> }
> }
>
> // debug memory issue here
> System.out.println("Duration = " + (System.currentTimeMillis()
> - start));
> System.out.println("Cache filled attach JVisualVM to view
> memory and generate a heap dump.");
> Thread.sleep(Long.MAX_VALUE);
> }
> }
>
>
> public static CacheConfiguration<CustomKey, CustomValue> cacheConfig()
> {
> final CacheConfiguration<CustomKey, CustomValue>
> cacheConfiguration = new CacheConfiguration<>( "TestCacheName" );
>
> cacheConfiguration.setBackups( 1 );
> cacheConfiguration.setCacheMode( CacheMode.PARTITIONED );
> cacheConfiguration.setIndexedTypes( CustomKey.class,
> CustomValue.class );
> cacheConfiguration.setMemoryMode( CacheMemoryMode.OFFHEAP_TIERED );
> cacheConfiguration.setOffHeapMaxMemory( 4L * 1024L * 1024L * 1024L
> );
> cacheConfiguration.setEvictionPolicy( new LruEvictionPolicy(
> 6_000_000 ) );
> cacheConfiguration.setSwapEnabled( true );
> cacheConfiguration.setStatisticsEnabled( true );
>
> // Causes heap usage to be high
> cacheConfiguration.setExpiryPolicyFactory(
> FactoryBuilder.factoryOf( new CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );
>
> return cacheConfiguration;
> }
>
> public static class CustomKey
> {
>
> @QuerySqlField(index = true, descending = false, name = "ts")
> private long timeStamp;
>
> @QuerySqlField(index = true, name = "id1")
> private long id1;
>
> @QuerySqlField(index = true, name = "id2")
> private long id2;
>
>
> public CustomKey( long timeStamp,
> long id1,
> long id2 )
> {
> this.timeStamp = timeStamp;
> this.id1 = id1;
> this.id2 = id2;
> }
> }
>
> public static class CustomValue
> {
>
> @QuerySqlField(name = "_d1")
> private double d1;
> @QuerySqlField()
> private double d2;
> @QuerySqlField()
> private double d3;
> @QuerySqlField()
> private double d4;
> @QuerySqlField(name = "_count")
> private long count;
>
>
> public CustomValue( double d1,
> double d2,
> double d3,
> double d4,
> long count )
> {
> this.d1 = d1;
> this.d2 = d2;
> this.d3 = d3;
> this.d4 = d4;
> this.count = count;
> }
> }
> }
> ----
>
>
> On 10/03/16 13:04, Vladimir Ozerov wrote:
>
> Hi Neil,
>
> Could you please attach the code reproducing the problem?
>
> Vladimir.
>
> On Wed, Mar 9, 2016 at 1:16 PM, Neil Wightman <neil_wightman@intergral.com
> > wrote:
>
>> Hi All,
>>
>> I have been trying out ignite for the past few weeks but I am hitting a
>> strange problem. I dont know if this problem is in my code or ignite.
>>
>> Currently I have an OFFHEAP_TIERED cache with ~3 million entries, max off
>> heap size of 4Gb, but this is showing some very high heap memory usage.
>> The cache stores data for 1 hour then expires it and constantly streams
>> data into this cache via the dataStreamer.addData API.
>>
>> The problem seems to be with this specific expiry policy configuration :
>>
>> cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new
>> CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );
>>
>> - with this expiry policy the heap memory usage for the ignite process
>> is 976Mb, without this expiry policy set its just 156Mb.
>>
>> The actual off heap memory usage is only 557Mb (assuming the cache
>> metrics are correct).
>>
>> I have generated two heap dumps and the memory difference is all in the
>> GridCacheTtlManager which is retaining a heap of ~889Mb according to
>> Eclipses MAT.
>>
>> I have ran my tests with Ignite 1.5.0 Final and 1.6.0 SNAPSHOT from git
>> (Monday 7th March) and both show the same issue.
>>
>> The problem with this large heap usage is that I am now getting very
>> large GC pauses which I cant stop without disabling the expiry policy.
>>
>> My main question is am I doing something wrong? Is the Expiry Policy
>> really supposed to use so much heap memory compared to the off heap cache
>> size?
>>
>> Thanks in advance,
>> Neil
>>
>> ---- config definition ----
>>
>> final CacheConfiguration<MetricKey, AggregatePack> cacheConfiguration =
>> new CacheConfiguration<>( "cacheName" );
>> cacheConfiguration.setBackups( 1 );
>> cacheConfiguration.setCacheMode( CacheMode.PARTITIONED );
>> cacheConfiguration.setIndexedTypes( CustomKey.class, CustomData.class );
>> cacheConfiguration.setMemoryMode( CacheMemoryMode.OFFHEAP_TIERED );
>> cacheConfiguration.setOffHeapMaxMemory( 4L * 1024L * 1024L * 1024L );
>> cacheConfiguration.setEvictionPolicy( new LruEvictionPolicy<>( 6_000_000
>> ) );
>> cacheConfiguration.setSwapEnabled( true );
>> cacheConfiguration.setStatisticsEnabled( true );
>> // This causes large memory usage
>> cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new
>> CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );
>>
>>
>
>
Re: Off Heap cache using lots of heap memory
Posted by Neil Wightman <ne...@intergral.com>.
Hi Vladimir,
It took me a while to make a small test case but I now have one (see below)
With the setExpiryPolicyFactory method enabled the heap after operation
983Mb, with the setExpiryPolicyFactory commented out it is just 85Mb.
Simple run this class file and once you see the console message attach
JVisualVM, then spam the "Perform GC" button.
Also without the expiry time set the test runs with -Xm512m and with
expiry the test needs -Xmx2024m.
Thanks
Neil
-- IgniteExpiryIssue.java --
package de.wightman.neil.apache.ignite.test;
import javax.cache.configuration.FactoryBuilder;
import javax.cache.expiry.CreatedExpiryPolicy;
import javax.cache.expiry.Duration;
import org.apache.ignite.Ignite;
import org.apache.ignite.IgniteCache;
import org.apache.ignite.IgniteDataStreamer;
import org.apache.ignite.Ignition;
import org.apache.ignite.cache.CacheMemoryMode;
import org.apache.ignite.cache.CacheMode;
import org.apache.ignite.cache.eviction.lru.LruEvictionPolicy;
import org.apache.ignite.cache.query.annotations.QuerySqlField;
import org.apache.ignite.configuration.CacheConfiguration;
public class IgniteExpiryIssue
{
public static void main( final String[] args ) throws
InterruptedException
{
final long start = System.currentTimeMillis();
// configures Tcp discovery for local host only
try (final Ignite ignite = Ignition.start( "LocalConfig.xml" ))
{
final IgniteCache<CustomKey, CustomValue> cache =
ignite.getOrCreateCache( cacheConfig() );
final IgniteDataStreamer<Object, Object> metricStreamer =
ignite.dataStreamer( "TestCacheName" );
final long counter = 1000;
final long counter2 = 49;
final long minutes = 60;
final long fakeBaseTime = System.currentTimeMillis();
for( long k = 0; k < minutes; k++ )
{
final long ts = fakeBaseTime + (minutes * 60 * 1000);
for( long i = 0; i < counter; i++ )
{
for( long j = 0; j < counter2; j++ )
{
final CustomKey key = new CustomKey( ts, i, j );
final CustomValue value = new CustomValue( 1.0,
1.0, 1.0, 1.0, 10 );
metricStreamer.addData( key, value );
}
}
}
// debug memory issue here
System.out.println("Duration = " +
(System.currentTimeMillis() - start));
System.out.println("Cache filled attach JVisualVM to view
memory and generate a heap dump.");
Thread.sleep(Long.MAX_VALUE);
}
}
public static CacheConfiguration<CustomKey, CustomValue> cacheConfig()
{
final CacheConfiguration<CustomKey, CustomValue>
cacheConfiguration = new CacheConfiguration<>( "TestCacheName" );
cacheConfiguration.setBackups( 1 );
cacheConfiguration.setCacheMode( CacheMode.PARTITIONED );
cacheConfiguration.setIndexedTypes( CustomKey.class,
CustomValue.class );
cacheConfiguration.setMemoryMode( CacheMemoryMode.OFFHEAP_TIERED );
cacheConfiguration.setOffHeapMaxMemory( 4L * 1024L * 1024L *
1024L );
cacheConfiguration.setEvictionPolicy( new LruEvictionPolicy(
6_000_000 ) );
cacheConfiguration.setSwapEnabled( true );
cacheConfiguration.setStatisticsEnabled( true );
// Causes heap usage to be high
cacheConfiguration.setExpiryPolicyFactory(
FactoryBuilder.factoryOf( new CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );
return cacheConfiguration;
}
public static class CustomKey
{
@QuerySqlField(index = true, descending = false, name = "ts")
private long timeStamp;
@QuerySqlField(index = true, name = "id1")
private long id1;
@QuerySqlField(index = true, name = "id2")
private long id2;
public CustomKey( long timeStamp,
long id1,
long id2 )
{
this.timeStamp = timeStamp;
this.id1 = id1;
this.id2 = id2;
}
}
public static class CustomValue
{
@QuerySqlField(name = "_d1")
private double d1;
@QuerySqlField()
private double d2;
@QuerySqlField()
private double d3;
@QuerySqlField()
private double d4;
@QuerySqlField(name = "_count")
private long count;
public CustomValue( double d1,
double d2,
double d3,
double d4,
long count )
{
this.d1 = d1;
this.d2 = d2;
this.d3 = d3;
this.d4 = d4;
this.count = count;
}
}
}
----
On 10/03/16 13:04, Vladimir Ozerov wrote:
> Hi Neil,
>
> Could you please attach the code reproducing the problem?
>
> Vladimir.
>
> On Wed, Mar 9, 2016 at 1:16 PM, Neil Wightman
> <neil_wightman@intergral.com <ma...@intergral.com>> wrote:
>
> Hi All,
>
> I have been trying out ignite for the past few weeks but I am
> hitting a strange problem. I dont know if this problem is in my
> code or ignite.
>
> Currently I have an OFFHEAP_TIERED cache with ~3 million entries,
> max off heap size of 4Gb, but this is showing some very high heap
> memory usage. The cache stores data for 1 hour then expires it
> and constantly streams data into this cache via the
> dataStreamer.addData API.
>
> The problem seems to be with this specific expiry policy
> configuration :
>
> cacheConfiguration.setExpiryPolicyFactory(
> FactoryBuilder.factoryOf( new CreatedExpiryPolicy(
> Duration.ONE_HOUR ) ) );
>
> - with this expiry policy the heap memory usage for the ignite
> process is 976Mb, without this expiry policy set its just 156Mb.
>
> The actual off heap memory usage is only 557Mb (assuming the cache
> metrics are correct).
>
> I have generated two heap dumps and the memory difference is all
> in the GridCacheTtlManager which is retaining a heap of ~889Mb
> according to Eclipses MAT.
>
> I have ran my tests with Ignite 1.5.0 Final and 1.6.0 SNAPSHOT
> from git (Monday 7th March) and both show the same issue.
>
> The problem with this large heap usage is that I am now getting
> very large GC pauses which I cant stop without disabling the
> expiry policy.
>
> My main question is am I doing something wrong? Is the Expiry
> Policy really supposed to use so much heap memory compared to the
> off heap cache size?
>
> Thanks in advance,
> Neil
>
> ---- config definition ----
>
> final CacheConfiguration<MetricKey, AggregatePack>
> cacheConfiguration = new CacheConfiguration<>( "cacheName" );
> cacheConfiguration.setBackups( 1 );
> cacheConfiguration.setCacheMode( CacheMode.PARTITIONED );
> cacheConfiguration.setIndexedTypes( CustomKey.class,
> CustomData.class );
> cacheConfiguration.setMemoryMode( CacheMemoryMode.OFFHEAP_TIERED );
> cacheConfiguration.setOffHeapMaxMemory( 4L * 1024L * 1024L * 1024L );
> cacheConfiguration.setEvictionPolicy( new LruEvictionPolicy<>(
> 6_000_000 ) );
> cacheConfiguration.setSwapEnabled( true );
> cacheConfiguration.setStatisticsEnabled( true );
> // This causes large memory usage
> cacheConfiguration.setExpiryPolicyFactory(
> FactoryBuilder.factoryOf( new CreatedExpiryPolicy(
> Duration.ONE_HOUR ) ) );
>
>
Re: Off Heap cache using lots of heap memory
Posted by Vladimir Ozerov <vo...@gridgain.com>.
Hi Neil,
Could you please attach the code reproducing the problem?
Vladimir.
On Wed, Mar 9, 2016 at 1:16 PM, Neil Wightman <ne...@intergral.com>
wrote:
> Hi All,
>
> I have been trying out ignite for the past few weeks but I am hitting a
> strange problem. I dont know if this problem is in my code or ignite.
>
> Currently I have an OFFHEAP_TIERED cache with ~3 million entries, max off
> heap size of 4Gb, but this is showing some very high heap memory usage.
> The cache stores data for 1 hour then expires it and constantly streams
> data into this cache via the dataStreamer.addData API.
>
> The problem seems to be with this specific expiry policy configuration :
>
> cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new
> CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );
>
> - with this expiry policy the heap memory usage for the ignite process is
> 976Mb, without this expiry policy set its just 156Mb.
>
> The actual off heap memory usage is only 557Mb (assuming the cache metrics
> are correct).
>
> I have generated two heap dumps and the memory difference is all in the
> GridCacheTtlManager which is retaining a heap of ~889Mb according to
> Eclipses MAT.
>
> I have ran my tests with Ignite 1.5.0 Final and 1.6.0 SNAPSHOT from git
> (Monday 7th March) and both show the same issue.
>
> The problem with this large heap usage is that I am now getting very large
> GC pauses which I cant stop without disabling the expiry policy.
>
> My main question is am I doing something wrong? Is the Expiry Policy
> really supposed to use so much heap memory compared to the off heap cache
> size?
>
> Thanks in advance,
> Neil
>
> ---- config definition ----
>
> final CacheConfiguration<MetricKey, AggregatePack> cacheConfiguration =
> new CacheConfiguration<>( "cacheName" );
> cacheConfiguration.setBackups( 1 );
> cacheConfiguration.setCacheMode( CacheMode.PARTITIONED );
> cacheConfiguration.setIndexedTypes( CustomKey.class, CustomData.class );
> cacheConfiguration.setMemoryMode( CacheMemoryMode.OFFHEAP_TIERED );
> cacheConfiguration.setOffHeapMaxMemory( 4L * 1024L * 1024L * 1024L );
> cacheConfiguration.setEvictionPolicy( new LruEvictionPolicy<>( 6_000_000 )
> );
> cacheConfiguration.setSwapEnabled( true );
> cacheConfiguration.setStatisticsEnabled( true );
> // This causes large memory usage
> cacheConfiguration.setExpiryPolicyFactory( FactoryBuilder.factoryOf( new
> CreatedExpiryPolicy( Duration.ONE_HOUR ) ) );
>
>