You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Nicholas DiPiazza <ni...@gmail.com> on 2020/10/21 13:26:59 UTC
Ignite 2.x - can I take advantage of offheap storage and also still
have time based eviction (from the on & off heap storage)?
Trying to get some help with my stack overflow question here
https://stackoverflow.com/questions/64456578/apache-ignite-2-x-can-i-take-advantage-of-offheap-storage-and-also-still-have
Cross posting it here.
I am using Apache Ignite 2.8.1
I have a specific cache that I need to store off heap (because otherwise I
will be plagued with out of memory conditions). And I also need the cache
to be distributed.
But I also want a time based eviction policy to evict entries from on/off
memory cache.
Is this even possible with Apache Ignite 2.x?
See the snippet here. I have tried to configure this every which way, but
nothing evicts these entries from the cache except doing it manually.
When I run the test method below, all the entries remain in the cache.
import java.time.LocalDateTime;import java.util.Arrays;import
java.util.concurrent.TimeUnit;
import javax.cache.expiry.CreatedExpiryPolicy;import
javax.cache.expiry.Duration;
import org.apache.ignite.Ignite;import
org.apache.ignite.Ignition;import
org.apache.ignite.cache.CacheAtomicityMode;import
org.apache.ignite.cache.CacheMode;import
org.apache.ignite.cache.eviction.lru.LruEvictionPolicyFactory;import
org.apache.ignite.configuration.CacheConfiguration;import
org.apache.ignite.configuration.DataRegionConfiguration;import
org.apache.ignite.configuration.DataStorageConfiguration;import
org.apache.ignite.configuration.IgniteConfiguration;import
org.apache.ignite.configuration.NearCacheConfiguration;import
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;import
org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
public class IgniteCache {
private static final String ORG_CACHE =
IgniteCache.class.getSimpleName() + "Organizations";
private static Ignite ignite;
private static org.apache.ignite.IgniteCache cache;
public void start() {
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
DataStorageConfiguration dsCfg = new DataStorageConfiguration();
DataRegionConfiguration dfltDataRegConf = new DataRegionConfiguration();
dfltDataRegConf.setPersistenceEnabled(true);
dsCfg.setDefaultDataRegionConfiguration(dfltDataRegConf);
dsCfg.setStoragePath("E:\\igniteStorage");
igniteConfiguration.setDataStorageConfiguration(dsCfg);
TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
tcpDiscoverySpi.setIpFinder(ipFinder);
igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);
ignite = Ignition.start(igniteConfiguration);
ignite.active(true);
CacheConfiguration<Long, X12File> cacheCfg = new
CacheConfiguration<>(ORG_CACHE);
cacheCfg.setCacheMode(CacheMode.REPLICATED);
cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheCfg.setBackups(1);
cacheCfg.setEvictionPolicyFactory(new
LruEvictionPolicyFactory<>(50000, 40, 5050)); // this doesn't do
anything.
cacheCfg.setNearConfiguration(
new NearCacheConfiguration<Long, X12File>()
.setNearEvictionPolicyFactory(new
LruEvictionPolicyFactory<>(1_000_000))); // this doesn't do anything.
cache = ignite.getOrCreateCache(cacheCfg);
for (long i = 0; i < 4_000_000; i++) {
if (i > 0 && i % 10_000 == 0) {
System.out.println("Done: " + i);
}
cache.withExpiryPolicy(new CreatedExpiryPolicy(new
Duration(TimeUnit.SECONDS, 1))) // this expiry policy doesn't do
anything
.put(i, new X12File("x12file" + i,
LocalDateTime.now().toString()));
}
}
public void test() {
System.out.println("Checking if cache entries are being
properly evicted ...");
int matches = 0;
for (long i = 0; i < 4_000_000; i++) {
if (cache.get(i) != null) {
++matches;
}
}
System.out.println("Matches: " + matches);
}
}
Is this just a shortcoming of Apache Ignite 2.x in my particular use case?
Re: Ignite 2.x - can I take advantage of offheap storage and also
still have time based eviction (from the on & off heap storage)?
Posted by Nicholas DiPiazza <ni...@gmail.com>.
This is the explanation I was looking for. I indeed was just not clear on
how these worked, but this helped me clear it up. thanks.
On Wed, Oct 21, 2020 at 11:44 PM Ilya Kazakov <ka...@gmail.com>
wrote:
> Hello, Nicholas!
>
> As I understand from your code and from your explanation, it looks like
> you are a little confused about the Eviction Policies and the Expiry
> Policies.
>
> 1. Eviction policy determines which data should be removed from RAM if RAM
> runs out.
> 2. Expiry policy using for determining TTL of cache entries.
>
> In your case you need only Expiry policy. Look at my example:
>
> private static final String ORG_CACHE = IgniteCache.class.getSimpleName()
> + "Organizations";
>
> public static void main(String[] args) throws Exception {
> DataRegionConfiguration dfltDataRegConf = new
> DataRegionConfiguration();
> dfltDataRegConf.setPersistenceEnabled(true);
>
> DataStorageConfiguration dsCfg = new DataStorageConfiguration();
> dsCfg.setDefaultDataRegionConfiguration(dfltDataRegConf);
> dsCfg.setStoragePath("/home/kazakov/tmp");
>
> IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
> igniteConfiguration.setDataStorageConfiguration(dsCfg);
>
> TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
> TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
> tcpDiscoverySpi.setIpFinder(ipFinder);
>
> igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);
>
> try(Ignite ignite = Ignition.start(igniteConfiguration)) {
> ignite.active(true);
>
> CacheConfiguration<Long, X12File> cacheCfg = new
> CacheConfiguration<>(ORG_CACHE);
>
> cacheCfg.setCacheMode(CacheMode.REPLICATED);
> cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> cacheCfg.setBackups(1);
> cacheCfg.setOnheapCacheEnabled(true);
>
> IgniteCache<Long, X12File> cache =
> ignite.getOrCreateCache(cacheCfg).withExpiryPolicy(new
> CreatedExpiryPolicy(new Duration(TimeUnit.SECONDS, 1)));
>
> for (long i = 0; i < 4_000_000; i++) {
> if (i > 0 && i % 10_000 == 0)
> System.out.println("Done: " + i);
>
> cache.put(i, new X12File("x12file" + i,
> LocalDateTime.now().toString()));
> }
>
> Thread.sleep(5000);
>
> int matches = 0;
> for (long i = 0; i < 4_000_000; i++) {
> if (cache.get(i) != null)
> ++matches;
> }
> System.out.println("Matches: " + matches);
> }
> }
>
> Ilya Kazakov
>
>
> ср, 21 окт. 2020 г. в 21:27, Nicholas DiPiazza <
> nicholas.dipiazza@gmail.com>:
> >
> > Trying to get some help with my stack overflow question here
> https://stackoverflow.com/questions/64456578/apache-ignite-2-x-can-i-take-advantage-of-offheap-storage-and-also-still-have
> >
> > Cross posting it here.
> >
> > I am using Apache Ignite 2.8.1
> >
> > I have a specific cache that I need to store off heap (because otherwise
> I will be plagued with out of memory conditions). And I also need the cache
> to be distributed.
> >
> > But I also want a time based eviction policy to evict entries from
> on/off memory cache.
> >
> > Is this even possible with Apache Ignite 2.x?
> >
> > See the snippet here. I have tried to configure this every which way,
> but nothing evicts these entries from the cache except doing it manually.
> >
> > When I run the test method below, all the entries remain in the cache.
> >
> > import java.time.LocalDateTime;
> > import java.util.Arrays;
> > import java.util.concurrent.TimeUnit;
> >
> > import javax.cache.expiry.CreatedExpiryPolicy;
> > import javax.cache.expiry.Duration;
> >
> > import org.apache.ignite.Ignite;
> > import org.apache.ignite.Ignition;
> > import org.apache.ignite.cache.CacheAtomicityMode;
> > import org.apache.ignite.cache.CacheMode;
> > import org.apache.ignite.cache.eviction.lru.LruEvictionPolicyFactory;
> > import org.apache.ignite.configuration.CacheConfiguration;
> > import org.apache.ignite.configuration.DataRegionConfiguration;
> > import org.apache.ignite.configuration.DataStorageConfiguration;
> > import org.apache.ignite.configuration.IgniteConfiguration;
> > import org.apache.ignite.configuration.NearCacheConfiguration;
> > import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
> > import
> org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
> >
> > public class IgniteCache {
> > private static final String ORG_CACHE =
> IgniteCache.class.getSimpleName() + "Organizations";
> >
> > private static Ignite ignite;
> >
> > private static org.apache.ignite.IgniteCache cache;
> >
> > public void start() {
> >
> > IgniteConfiguration igniteConfiguration = new
> IgniteConfiguration();
> > DataStorageConfiguration dsCfg = new DataStorageConfiguration();
> > DataRegionConfiguration dfltDataRegConf = new
> DataRegionConfiguration();
> > dfltDataRegConf.setPersistenceEnabled(true);
> > dsCfg.setDefaultDataRegionConfiguration(dfltDataRegConf);
> > dsCfg.setStoragePath("E:\\igniteStorage");
> > igniteConfiguration.setDataStorageConfiguration(dsCfg);
> >
> > TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
> > TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> > ipFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
> > tcpDiscoverySpi.setIpFinder(ipFinder);
> >
> > igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);
> >
> > ignite = Ignition.start(igniteConfiguration);
> >
> > ignite.active(true);
> >
> > CacheConfiguration<Long, X12File> cacheCfg = new
> CacheConfiguration<>(ORG_CACHE);
> >
> > cacheCfg.setCacheMode(CacheMode.REPLICATED);
> > cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> > cacheCfg.setBackups(1);
> >
> > cacheCfg.setEvictionPolicyFactory(new
> LruEvictionPolicyFactory<>(50000, 40, 5050)); // this doesn't do anything.
> > cacheCfg.setNearConfiguration(
> > new NearCacheConfiguration<Long, X12File>()
> > .setNearEvictionPolicyFactory(new
> LruEvictionPolicyFactory<>(1_000_000))); // this doesn't do anything.
> >
> > cache = ignite.getOrCreateCache(cacheCfg);
> >
> > for (long i = 0; i < 4_000_000; i++) {
> > if (i > 0 && i % 10_000 == 0) {
> > System.out.println("Done: " + i);
> > }
> > cache.withExpiryPolicy(new CreatedExpiryPolicy(new
> Duration(TimeUnit.SECONDS, 1))) // this expiry policy doesn't do anything
> > .put(i, new X12File("x12file" + i,
> LocalDateTime.now().toString()));
> > }
> > }
> >
> > public void test() {
> > System.out.println("Checking if cache entries are being properly
> evicted ...");
> >
> > int matches = 0;
> > for (long i = 0; i < 4_000_000; i++) {
> > if (cache.get(i) != null) {
> > ++matches;
> > }
> > }
> > System.out.println("Matches: " + matches);
> > }
> > }
> >
> > Is this just a shortcoming of Apache Ignite 2.x in my particular use
> case?
>
Re: Ignite 2.x - can I take advantage of offheap storage and also
still have time based eviction (from the on & off heap storage)?
Posted by Ilya Kazakov <ka...@gmail.com>.
Hello, Nicholas!
As I understand from your code and from your explanation, it looks like you
are a little confused about the Eviction Policies and the Expiry Policies.
1. Eviction policy determines which data should be removed from RAM if RAM
runs out.
2. Expiry policy using for determining TTL of cache entries.
In your case you need only Expiry policy. Look at my example:
private static final String ORG_CACHE = IgniteCache.class.getSimpleName() +
"Organizations";
public static void main(String[] args) throws Exception {
DataRegionConfiguration dfltDataRegConf = new DataRegionConfiguration();
dfltDataRegConf.setPersistenceEnabled(true);
DataStorageConfiguration dsCfg = new DataStorageConfiguration();
dsCfg.setDefaultDataRegionConfiguration(dfltDataRegConf);
dsCfg.setStoragePath("/home/kazakov/tmp");
IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setDataStorageConfiguration(dsCfg);
TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
tcpDiscoverySpi.setIpFinder(ipFinder);
igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);
try(Ignite ignite = Ignition.start(igniteConfiguration)) {
ignite.active(true);
CacheConfiguration<Long, X12File> cacheCfg = new
CacheConfiguration<>(ORG_CACHE);
cacheCfg.setCacheMode(CacheMode.REPLICATED);
cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheCfg.setBackups(1);
cacheCfg.setOnheapCacheEnabled(true);
IgniteCache<Long, X12File> cache =
ignite.getOrCreateCache(cacheCfg).withExpiryPolicy(new
CreatedExpiryPolicy(new Duration(TimeUnit.SECONDS, 1)));
for (long i = 0; i < 4_000_000; i++) {
if (i > 0 && i % 10_000 == 0)
System.out.println("Done: " + i);
cache.put(i, new X12File("x12file" + i,
LocalDateTime.now().toString()));
}
Thread.sleep(5000);
int matches = 0;
for (long i = 0; i < 4_000_000; i++) {
if (cache.get(i) != null)
++matches;
}
System.out.println("Matches: " + matches);
}
}
Ilya Kazakov
ср, 21 окт. 2020 г. в 21:27, Nicholas DiPiazza <nicholas.dipiazza@gmail.com
>:
>
> Trying to get some help with my stack overflow question here
https://stackoverflow.com/questions/64456578/apache-ignite-2-x-can-i-take-advantage-of-offheap-storage-and-also-still-have
>
> Cross posting it here.
>
> I am using Apache Ignite 2.8.1
>
> I have a specific cache that I need to store off heap (because otherwise
I will be plagued with out of memory conditions). And I also need the cache
to be distributed.
>
> But I also want a time based eviction policy to evict entries from on/off
memory cache.
>
> Is this even possible with Apache Ignite 2.x?
>
> See the snippet here. I have tried to configure this every which way, but
nothing evicts these entries from the cache except doing it manually.
>
> When I run the test method below, all the entries remain in the cache.
>
> import java.time.LocalDateTime;
> import java.util.Arrays;
> import java.util.concurrent.TimeUnit;
>
> import javax.cache.expiry.CreatedExpiryPolicy;
> import javax.cache.expiry.Duration;
>
> import org.apache.ignite.Ignite;
> import org.apache.ignite.Ignition;
> import org.apache.ignite.cache.CacheAtomicityMode;
> import org.apache.ignite.cache.CacheMode;
> import org.apache.ignite.cache.eviction.lru.LruEvictionPolicyFactory;
> import org.apache.ignite.configuration.CacheConfiguration;
> import org.apache.ignite.configuration.DataRegionConfiguration;
> import org.apache.ignite.configuration.DataStorageConfiguration;
> import org.apache.ignite.configuration.IgniteConfiguration;
> import org.apache.ignite.configuration.NearCacheConfiguration;
> import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi;
> import
org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder;
>
> public class IgniteCache {
> private static final String ORG_CACHE =
IgniteCache.class.getSimpleName() + "Organizations";
>
> private static Ignite ignite;
>
> private static org.apache.ignite.IgniteCache cache;
>
> public void start() {
>
> IgniteConfiguration igniteConfiguration = new
IgniteConfiguration();
> DataStorageConfiguration dsCfg = new DataStorageConfiguration();
> DataRegionConfiguration dfltDataRegConf = new
DataRegionConfiguration();
> dfltDataRegConf.setPersistenceEnabled(true);
> dsCfg.setDefaultDataRegionConfiguration(dfltDataRegConf);
> dsCfg.setStoragePath("E:\\igniteStorage");
> igniteConfiguration.setDataStorageConfiguration(dsCfg);
>
> TcpDiscoverySpi tcpDiscoverySpi = new TcpDiscoverySpi();
> TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509"));
> tcpDiscoverySpi.setIpFinder(ipFinder);
>
> igniteConfiguration.setDiscoverySpi(tcpDiscoverySpi);
>
> ignite = Ignition.start(igniteConfiguration);
>
> ignite.active(true);
>
> CacheConfiguration<Long, X12File> cacheCfg = new
CacheConfiguration<>(ORG_CACHE);
>
> cacheCfg.setCacheMode(CacheMode.REPLICATED);
> cacheCfg.setAtomicityMode(CacheAtomicityMode.ATOMIC);
> cacheCfg.setBackups(1);
>
> cacheCfg.setEvictionPolicyFactory(new
LruEvictionPolicyFactory<>(50000, 40, 5050)); // this doesn't do anything.
> cacheCfg.setNearConfiguration(
> new NearCacheConfiguration<Long, X12File>()
> .setNearEvictionPolicyFactory(new
LruEvictionPolicyFactory<>(1_000_000))); // this doesn't do anything.
>
> cache = ignite.getOrCreateCache(cacheCfg);
>
> for (long i = 0; i < 4_000_000; i++) {
> if (i > 0 && i % 10_000 == 0) {
> System.out.println("Done: " + i);
> }
> cache.withExpiryPolicy(new CreatedExpiryPolicy(new
Duration(TimeUnit.SECONDS, 1))) // this expiry policy doesn't do anything
> .put(i, new X12File("x12file" + i,
LocalDateTime.now().toString()));
> }
> }
>
> public void test() {
> System.out.println("Checking if cache entries are being properly
evicted ...");
>
> int matches = 0;
> for (long i = 0; i < 4_000_000; i++) {
> if (cache.get(i) != null) {
> ++matches;
> }
> }
> System.out.println("Matches: " + matches);
> }
> }
>
> Is this just a shortcoming of Apache Ignite 2.x in my particular use case?