You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by scottmf <sc...@gmail.com> on 2018/03/09 03:51:51 UTC
eviction performance
Hi,
I am prototyping using ignite to ingest lots of short lived events in a
system. In order to ingest the data i'm using the kafka streamer mechanism.
I'm pushing 20k events / second into the system into a partitioned off heap
cache. I've run into a few performance issues but I've been able to wiggle
my way out of it using information from the forums and ignite docs. But the
latest issue I can't seem to find any information about.
My cache is setup to evict the data after one hour. The insertion
performance degrades substantially when the eviction mechanism kicks in
where the latency of the insertion continues to degrade over time.
WRT to the OS, memory and disk none of them are saturated. The load avg is
low, the gc is not under pressure and the disk is not fully utilized.
I've experimented around with different eviction times to make sure that it
is indeed the eviction that is causing this. After some idle time I startup
our event simulator and wait for the evictions to start and sure enough the
latency of insertions increases immediately when evictions start.
Each node is 32 GB / 8 cpus.
Here are my jvm opts: "-Xmx12g -Xms12g -XX:MaxDirectMemorySize=10g
-XX:PermSize=192m -XX:MaxPermSize=192m -XX:+UseG1GC -XX:ConcGCThreads=12
-XX:ParallelGCThreads=22 -XX:MaxGCPauseMillis=1000 -XX:G1HeapWastePercent=2
-XX:G1ReservePercent=15 -XX:+UnlockExperimentalVMOptions
-XX:G1OldCSetRegionThresholdPercent=15 -XX:G1MixedGCLiveThresholdPercent=90
-XX:G1MaxNewSizePercent=25 -Dcom.sun.management.jmxremote
-XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError"
I was wondering if there is no tuning I can do to avoid this overhead, I am
partitioning my data by time - is there a performant way to clear a
partition in a cache one at a time? (i was thinking of something similar to
a truncate command in a rdbms)
Here is my schema:
CachePojo {
@QuerySqlField(index=false)
private String buffer0;
@QuerySqlField(index=true)
private String buffer1;
@QuerySqlField
private String buffer2;
@QuerySqlField(index=true, inlineSize=8, descending=true)
private Timestamp timestamp;
@QuerySqlField(index=false)
private Map<String, String> fields = new HashMap<>();
}
Ignite config:
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setIgniteHome(igniteHome);
// this is probably not needed
igniteConfiguration.setTransactionConfiguration(new
TransactionConfiguration());
igniteConfiguration.setPeerClassLoadingEnabled(true);
igniteConfiguration.setIncludeEventTypes(new int[0]);
igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi());
igniteConfiguration.setDataStreamerThreadPoolSize(8);
igniteConfiguration.setPublicThreadPoolSize(16);
igniteConfiguration.setSystemThreadPoolSize(16);
DataStorageConfiguration dsc = dataStorageConfiguration();
igniteConfiguration.setDataStorageConfiguration(dsc);
dataStorageConfig:
DataStorageConfiguration dataStorageConfiguration = new
DataStorageConfiguration();
DataRegionConfiguration dataRegionConfiguration =
dataRegionConfiguration();
dataStorageConfiguration.setDefaultDataRegionConfiguration(dataRegionConfiguration);
dataStorageConfiguration.setDataRegionConfigurations(largeDataRegionConfiguration());
dataStorageConfiguration.setPageSize(8192);
dataStorageConfiguration.setMetricsEnabled(true);
dataStorageConfiguration.setWriteThrottlingEnabled(true);
dataStorageConfiguration.setStoragePath("/var/lib/ignite/persistence");
dataStorageConfiguration.setWalMode(WALMode.NONE);
dataStorageConfiguration.setWalPath("/var/lib/ignite/wal");
dataStorageConfiguration.setWalArchivePath("/var/lib/ignite/wal/archive");
cacheDataRegionConfig:
DataRegionConfiguration dataRegionConfiguration = new
DataRegionConfiguration();
dataRegionConfiguration.setPersistenceEnabled(true);
dataRegionConfiguration.setName("dataRegion");
dataRegionConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
dataRegionConfiguration.setInitialSize(500l * 1024 * 1024);
dataRegionConfiguration.setMaxSize(8l * 1024 * 1024 * 1024);
dataRegionConfiguration.setMetricsEnabled(true);
dataRegionConfiguration.setEmptyPagesPoolSize(1024);
cacheConfig:
CacheConfiguration<Key, CachePojo> cacheConfig = new
CacheConfiguration<>();
cacheConfig.setSqlFunctionClasses(SearchFunctions.class);
cacheConfig.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(new
Duration(TimeUnit.MINUTES, 60)));
cacheConfig.setName("mycache");
cacheConfig.setTypes(Key.class, CachePojo.class);
cacheConfig.setIndexedTypes(Key.class, CachePojo.class);
cacheConfig.setCacheMode(CacheMode.PARTITIONED);
cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheConfig.setQueryParallelism(64);
cacheConfig.setStatisticsEnabled(true);
cacheConfig.setGroupName("cache");
cacheConfig.setDataRegionName("dataRegion");
// partitions in one minute chunks evenly across all nodes
cacheConfig.setAffinity(new CacheAffinityFunction());
cacheConfig.setRebalanceBatchSize(1024 * 1024 * 4);
cacheConfig.setRebalanceThreadPoolSize(4);
cacheConfig.setRebalanceTimeout(TimeUnit.MINUTES.toMillis(10));
cacheConfig.setRebalanceDelay(TimeUnit.MINUTES.toMillis(2));
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: eviction performance
Posted by Scott Feldstein <sc...@gmail.com>.
Hi Stan,
Thanks for the follow up. The part that I'm confused about here is when an
expired entry isn't accessed it would simply linger on disk forever.
In my prototype I want to be able to store one continuous hour of hot data
to be queryable at all times. The volume of data currently sits at 20k
entries / second but I'd like to increase this substantially.
Unfortunately the cleanup issue getting in the way of this.
To load up the data I'm using the ignite streamer, this makes it more
difficult to create a new cache in order to clean up old data (not
impossible but awkward). I'm using the streamer since in the docs it says
that is the most efficient way to load data into the system. Is there a
better way to load up the data for my use case other than using a streamer?
In my scenario I would like to be able to have a light weight way of
cleaning up the data. While the data may not be cleaned up immediately I'd
like to have it eventually cleaned up since it is taking up a lot of disk.
Is this possible without using a separate cache? Or is using multiple
caches the only way you can think of to achieve this?
Do you think that a "truncate partition" feature is something that would be
useful / doable in Ignite? Would it be worth a feature request?
thanks!
On Sun, Mar 25, 2018 at 1:59 PM Stanislav Lukyanov <st...@gmail.com>
wrote:
> Hi Scott,
>
> With eagetTTL=false, each time you access an entry its TTL is automatically
> checked - if it is expired, the entry will be removed. It means that
> instead
> of having a separate thread waking up and removing all of the expired
> entries at the same time (and taking a chunk of CPU and IO for a noticeable
> period of time), you'll be removing them over time. On average, individual
> accesses may become a bit more expensive (because some of them will trigger
> removal) but that will allow you to avoid latency spikes.
>
> About the truncate - I think there is no API to do that based on
> partitions.
> I guess you could try creating a new cache for each batch and destroying it
> after you don't need it anymore, but it's hard to say whether it will be
> more efficient.
>
> Stan
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
Re: eviction performance
Posted by Stanislav Lukyanov <st...@gmail.com>.
Hi Scott,
With eagetTTL=false, each time you access an entry its TTL is automatically
checked - if it is expired, the entry will be removed. It means that instead
of having a separate thread waking up and removing all of the expired
entries at the same time (and taking a chunk of CPU and IO for a noticeable
period of time), you'll be removing them over time. On average, individual
accesses may become a bit more expensive (because some of them will trigger
removal) but that will allow you to avoid latency spikes.
About the truncate - I think there is no API to do that based on partitions.
I guess you could try creating a new cache for each batch and destroying it
after you don't need it anymore, but it's hard to say whether it will be
more efficient.
Stan
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: eviction performance
Posted by Scott Feldstein <sc...@gmail.com>.
Hi Stanislav,
Thanks for info and the note on the terminology.
So in my setup im partitioning by chunks of time. If I turn eagerttl to false how would the cleanup look? Would I periodically scan the partitions that i want to be expired? Are there any best practices to that end?
Additionally are there any truncate-like commands I can use to achieve a lightweight cleanup of my data?
Thanks,
Scott
> On Mar 9, 2018, at 1:41 AM, Stanislav Lukyanov <st...@gmail.com> wrote:
>
> Hi,
>
> A terminology nitpicking: it seems that you’re talking about expiry; eviction is a similar mechanism but is based on the data size, not time.
>
> Have you tried setting CacheConfiguration.eagerTtl = false (https://apacheignite.readme.io/v2.3/docs/expiry-policies#section-eager-ttl)?
> With that setting the expired entries will be removed on the next access instead of doing that in the background.
> If you add entries in large batches but access them more uniformly over time, it could redistribute the cost of expiring the data more evenly.
>
> Thanks,
> Stan
>
> From: scottmf
> Sent: 9 марта 2018 г. 6:51
> To: user@ignite.apache.org
> Subject: eviction performance
>
> Hi,
> I am prototyping using ignite to ingest lots of short lived events in a
> system. In order to ingest the data i'm using the kafka streamer mechanism.
> I'm pushing 20k events / second into the system into a partitioned off heap
> cache. I've run into a few performance issues but I've been able to wiggle
> my way out of it using information from the forums and ignite docs. But the
> latest issue I can't seem to find any information about.
>
> My cache is setup to evict the data after one hour. The insertion
> performance degrades substantially when the eviction mechanism kicks in
> where the latency of the insertion continues to degrade over time.
>
> WRT to the OS, memory and disk none of them are saturated. The load avg is
> low, the gc is not under pressure and the disk is not fully utilized.
>
> I've experimented around with different eviction times to make sure that it
> is indeed the eviction that is causing this. After some idle time I startup
> our event simulator and wait for the evictions to start and sure enough the
> latency of insertions increases immediately when evictions start.
>
> Each node is 32 GB / 8 cpus.
>
> Here are my jvm opts: "-Xmx12g -Xms12g -XX:MaxDirectMemorySize=10g
> -XX:PermSize=192m -XX:MaxPermSize=192m -XX:+UseG1GC -XX:ConcGCThreads=12
> -XX:ParallelGCThreads=22 -XX:MaxGCPauseMillis=1000 -XX:G1HeapWastePercent=2
> -XX:G1ReservePercent=15 -XX:+UnlockExperimentalVMOptions
> -XX:G1OldCSetRegionThresholdPercent=15 -XX:G1MixedGCLiveThresholdPercent=90
> -XX:G1MaxNewSizePercent=25 -Dcom.sun.management.jmxremote
> -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError"
>
> I was wondering if there is no tuning I can do to avoid this overhead, I am
> partitioning my data by time - is there a performant way to clear a
> partition in a cache one at a time? (i was thinking of something similar to
> a truncate command in a rdbms)
>
> Here is my schema:
> CachePojo {
> @QuerySqlField(index=false)
> private String buffer0;
> @QuerySqlField(index=true)
> private String buffer1;
> @QuerySqlField
> private String buffer2;
> @QuerySqlField(index=true, inlineSize=8, descending=true)
> private Timestamp timestamp;
> @QuerySqlField(index=false)
> private Map<String, String> fields = new HashMap<>();
> }
>
> Ignite config:
> IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
> igniteConfiguration.setIgniteHome(igniteHome);
> // this is probably not needed
> igniteConfiguration.setTransactionConfiguration(new
> TransactionConfiguration());
> igniteConfiguration.setPeerClassLoadingEnabled(true);
> igniteConfiguration.setIncludeEventTypes(new int[0]);
> igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi());
> igniteConfiguration.setDataStreamerThreadPoolSize(8);
> igniteConfiguration.setPublicThreadPoolSize(16);
> igniteConfiguration.setSystemThreadPoolSize(16);
> DataStorageConfiguration dsc = dataStorageConfiguration();
> igniteConfiguration.setDataStorageConfiguration(dsc);
>
> dataStorageConfig:
> DataStorageConfiguration dataStorageConfiguration = new
> DataStorageConfiguration();
> DataRegionConfiguration dataRegionConfiguration =
> dataRegionConfiguration();
>
> dataStorageConfiguration.setDefaultDataRegionConfiguration(dataRegionConfiguration);
>
> dataStorageConfiguration.setDataRegionConfigurations(largeDataRegionConfiguration());
> dataStorageConfiguration.setPageSize(8192);
> dataStorageConfiguration.setMetricsEnabled(true);
> dataStorageConfiguration.setWriteThrottlingEnabled(true);
>
> dataStorageConfiguration.setStoragePath("/var/lib/ignite/persistence");
> dataStorageConfiguration.setWalMode(WALMode.NONE);
> dataStorageConfiguration.setWalPath("/var/lib/ignite/wal");
>
> dataStorageConfiguration.setWalArchivePath("/var/lib/ignite/wal/archive");
>
> cacheDataRegionConfig:
> DataRegionConfiguration dataRegionConfiguration = new
> DataRegionConfiguration();
> dataRegionConfiguration.setPersistenceEnabled(true);
> dataRegionConfiguration.setName("dataRegion");
>
> dataRegionConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
> dataRegionConfiguration.setInitialSize(500l * 1024 * 1024);
> dataRegionConfiguration.setMaxSize(8l * 1024 * 1024 * 1024);
> dataRegionConfiguration.setMetricsEnabled(true);
> dataRegionConfiguration.setEmptyPagesPoolSize(1024);
>
> cacheConfig:
> CacheConfiguration<Key, CachePojo> cacheConfig = new
> CacheConfiguration<>();
> cacheConfig.setSqlFunctionClasses(SearchFunctions.class);
> cacheConfig.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(new
> Duration(TimeUnit.MINUTES, 60)));
> cacheConfig.setName("mycache");
> cacheConfig.setTypes(Key.class, CachePojo.class);
> cacheConfig.setIndexedTypes(Key.class, CachePojo.class);
> cacheConfig.setCacheMode(CacheMode.PARTITIONED);
> cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> cacheConfig.setQueryParallelism(64);
> cacheConfig.setStatisticsEnabled(true);
> cacheConfig.setGroupName("cache");
> cacheConfig.setDataRegionName("dataRegion");
> // partitions in one minute chunks evenly across all nodes
> cacheConfig.setAffinity(new CacheAffinityFunction());
> cacheConfig.setRebalanceBatchSize(1024 * 1024 * 4);
> cacheConfig.setRebalanceThreadPoolSize(4);
> cacheConfig.setRebalanceTimeout(TimeUnit.MINUTES.toMillis(10));
> cacheConfig.setRebalanceDelay(TimeUnit.MINUTES.toMillis(2));
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
RE: eviction performance
Posted by Stanislav Lukyanov <st...@gmail.com>.
Hi,
A terminology nitpicking: it seems that you’re talking about expiry; eviction is a similar mechanism but is based on the data size, not time.
Have you tried setting CacheConfiguration.eagerTtl = false (https://apacheignite.readme.io/v2.3/docs/expiry-policies#section-eager-ttl)?
With that setting the expired entries will be removed on the next access instead of doing that in the background.
If you add entries in large batches but access them more uniformly over time, it could redistribute the cost of expiring the data more evenly.
Thanks,
Stan
From: scottmf
Sent: 9 марта 2018 г. 6:51
To: user@ignite.apache.org
Subject: eviction performance
Hi,
I am prototyping using ignite to ingest lots of short lived events in a
system. In order to ingest the data i'm using the kafka streamer mechanism.
I'm pushing 20k events / second into the system into a partitioned off heap
cache. I've run into a few performance issues but I've been able to wiggle
my way out of it using information from the forums and ignite docs. But the
latest issue I can't seem to find any information about.
My cache is setup to evict the data after one hour. The insertion
performance degrades substantially when the eviction mechanism kicks in
where the latency of the insertion continues to degrade over time.
WRT to the OS, memory and disk none of them are saturated. The load avg is
low, the gc is not under pressure and the disk is not fully utilized.
I've experimented around with different eviction times to make sure that it
is indeed the eviction that is causing this. After some idle time I startup
our event simulator and wait for the evictions to start and sure enough the
latency of insertions increases immediately when evictions start.
Each node is 32 GB / 8 cpus.
Here are my jvm opts: "-Xmx12g -Xms12g -XX:MaxDirectMemorySize=10g
-XX:PermSize=192m -XX:MaxPermSize=192m -XX:+UseG1GC -XX:ConcGCThreads=12
-XX:ParallelGCThreads=22 -XX:MaxGCPauseMillis=1000 -XX:G1HeapWastePercent=2
-XX:G1ReservePercent=15 -XX:+UnlockExperimentalVMOptions
-XX:G1OldCSetRegionThresholdPercent=15 -XX:G1MixedGCLiveThresholdPercent=90
-XX:G1MaxNewSizePercent=25 -Dcom.sun.management.jmxremote
-XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError"
I was wondering if there is no tuning I can do to avoid this overhead, I am
partitioning my data by time - is there a performant way to clear a
partition in a cache one at a time? (i was thinking of something similar to
a truncate command in a rdbms)
Here is my schema:
CachePojo {
@QuerySqlField(index=false)
private String buffer0;
@QuerySqlField(index=true)
private String buffer1;
@QuerySqlField
private String buffer2;
@QuerySqlField(index=true, inlineSize=8, descending=true)
private Timestamp timestamp;
@QuerySqlField(index=false)
private Map<String, String> fields = new HashMap<>();
}
Ignite config:
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setIgniteHome(igniteHome);
// this is probably not needed
igniteConfiguration.setTransactionConfiguration(new
TransactionConfiguration());
igniteConfiguration.setPeerClassLoadingEnabled(true);
igniteConfiguration.setIncludeEventTypes(new int[0]);
igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi());
igniteConfiguration.setDataStreamerThreadPoolSize(8);
igniteConfiguration.setPublicThreadPoolSize(16);
igniteConfiguration.setSystemThreadPoolSize(16);
DataStorageConfiguration dsc = dataStorageConfiguration();
igniteConfiguration.setDataStorageConfiguration(dsc);
dataStorageConfig:
DataStorageConfiguration dataStorageConfiguration = new
DataStorageConfiguration();
DataRegionConfiguration dataRegionConfiguration =
dataRegionConfiguration();
dataStorageConfiguration.setDefaultDataRegionConfiguration(dataRegionConfiguration);
dataStorageConfiguration.setDataRegionConfigurations(largeDataRegionConfiguration());
dataStorageConfiguration.setPageSize(8192);
dataStorageConfiguration.setMetricsEnabled(true);
dataStorageConfiguration.setWriteThrottlingEnabled(true);
dataStorageConfiguration.setStoragePath("/var/lib/ignite/persistence");
dataStorageConfiguration.setWalMode(WALMode.NONE);
dataStorageConfiguration.setWalPath("/var/lib/ignite/wal");
dataStorageConfiguration.setWalArchivePath("/var/lib/ignite/wal/archive");
cacheDataRegionConfig:
DataRegionConfiguration dataRegionConfiguration = new
DataRegionConfiguration();
dataRegionConfiguration.setPersistenceEnabled(true);
dataRegionConfiguration.setName("dataRegion");
dataRegionConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
dataRegionConfiguration.setInitialSize(500l * 1024 * 1024);
dataRegionConfiguration.setMaxSize(8l * 1024 * 1024 * 1024);
dataRegionConfiguration.setMetricsEnabled(true);
dataRegionConfiguration.setEmptyPagesPoolSize(1024);
cacheConfig:
CacheConfiguration<Key, CachePojo> cacheConfig = new
CacheConfiguration<>();
cacheConfig.setSqlFunctionClasses(SearchFunctions.class);
cacheConfig.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(new
Duration(TimeUnit.MINUTES, 60)));
cacheConfig.setName("mycache");
cacheConfig.setTypes(Key.class, CachePojo.class);
cacheConfig.setIndexedTypes(Key.class, CachePojo.class);
cacheConfig.setCacheMode(CacheMode.PARTITIONED);
cacheConfig.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheConfig.setQueryParallelism(64);
cacheConfig.setStatisticsEnabled(true);
cacheConfig.setGroupName("cache");
cacheConfig.setDataRegionName("dataRegion");
// partitions in one minute chunks evenly across all nodes
cacheConfig.setAffinity(new CacheAffinityFunction());
cacheConfig.setRebalanceBatchSize(1024 * 1024 * 4);
cacheConfig.setRebalanceThreadPoolSize(4);
cacheConfig.setRebalanceTimeout(TimeUnit.MINUTES.toMillis(10));
cacheConfig.setRebalanceDelay(TimeUnit.MINUTES.toMillis(2));
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/