You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Subash Chaturanga <su...@gmail.com> on 2018/03/07 19:25:30 UTC
Ignite Expiry Inconsistency with Native Persistence
Hi team,
With a cache having CreatedExpiryPolicy for 1min duration and having ignite
persistence enabled as per docs. And did this.
- cache put and then cache get, gives me the value
- wait for 1min
- cache get, returns null. Perfectly fine up to now.
- then recycled JVM.
- now only cache.get, no puts, and return == null is my expectation. But it
wasn’t the case. It returned me the value.
This means cache expiry doesn’t remove the entry from native persistent
layer. But if you do a cache.remove() it will remove it from native
persistent later too. Meaning JVM recycle will return null.
So compared to cache.remove working as expected, cache expiry behavior is
very inconsistent.
Can someone please clarify ?
--
/subash
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by Dmitry Pavlov <dp...@gmail.com>.
Hi,
It seems this is very similar with
https://issues.apache.org/jira/browse/IGNITE-6964
After node restart there can be some delay in removal. When some other
entry will be removed, entry which survived during restart should be
removed also.
I guess entry may come from WAL after restart and not removed because of
this delay.
Sincerely
Dmitry Pavlov
чт, 8 мар. 2018 г., 17:15 Subash Chaturanga <su...@gmail.com>:
> Ideally 3 nodes.
>
> So expiry inconsistency is a known issue ?
>
> But I reproduce the expiry inconsistency issue even in a single node with
> the steps mentioned earlier.
>
> We are evaluating Ignite vs Redis for our use case these days, and wanted
> to make sure everything works fine as in docs. Can you please confirm the
> cache eviction bahavior also, like can we put a limit to the persistence
> later too ?
>
>
> On Thu, Mar 8, 2018 at 6:51 AM Dmitry Pavlov <dp...@gmail.com>
> wrote:
>
>> Hi,
>>
>> There was some well known issue with native persistence and rebalancing
>> data between nodes, later I can try to find its Id .
>>
>> How much nodes do you use?
>>
>> Sincerely,
>> Dmitry Pavlov
>>
>> ср, 7 мар. 2018 г., 22:25 Subash Chaturanga <su...@gmail.com>:
>>
>>> Hi team,
>>> With a cache having CreatedExpiryPolicy for 1min duration and having
>>> ignite persistence enabled as per docs. And did this.
>>>
>>> - cache put and then cache get, gives me the value
>>> - wait for 1min
>>> - cache get, returns null. Perfectly fine up to now.
>>> - then recycled JVM.
>>> - now only cache.get, no puts, and return == null is my expectation. But
>>> it wasn’t the case. It returned me the value.
>>>
>>> This means cache expiry doesn’t remove the entry from native persistent
>>> layer. But if you do a cache.remove() it will remove it from native
>>> persistent later too. Meaning JVM recycle will return null.
>>>
>>> So compared to cache.remove working as expected, cache expiry behavior
>>> is very inconsistent.
>>>
>>> Can someone please clarify ?
>>>
>>> --
>>> /subash
>>>
>> --
> /subash
>
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by Subash Chaturanga <su...@gmail.com>.
Thank you very much Val!! Will keep an eye on when it get resolved.
On Thu, Mar 22, 2018 at 8:24 PM vkulichenko <va...@gmail.com>
wrote:
> Subash,
>
> Yes, I reproduced it now. Looks like a bug to me, I created a ticket:
> https://issues.apache.org/jira/browse/IGNITE-8027
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
/subash
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by vkulichenko <va...@gmail.com>.
Subash,
Yes, I reproduced it now. Looks like a bug to me, I created a ticket:
https://issues.apache.org/jira/browse/IGNITE-8027
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by Subash Chaturanga <su...@gmail.com>.
I tried with proper ignite.stopAll() and it worked as expected!!
So with a node crash, those expiration related metadata not got persisted.
Is there a time for it to write and flush to disk ?
On Wed, Mar 21, 2018 at 12:55 AM Subash Chaturanga <su...@gmail.com>
wrote:
> Hi Val,
> Only one node running. And at step 2 when you say, wait for output and
> stop, did you mean Ignite.stop() or kill -9 jvm.
>
> What I do is kill -9.
>
> On Tue, Mar 20, 2018 at 9:21 PM vkulichenko <va...@gmail.com>
> wrote:
>
>> Subash,
>>
>> When you run the test, is this the only node, or there are some other
>> nodes
>> running? I actually can't reproduce the issue even with your code. Here is
>> what I do:
>> - Clean up the work directory.
>> - Run the test, wait for output, stop.
>> - Comment out 'put' invocations.
>> - Run again.
>>
>> The output on the second one is the following:
>>
>> Tue Mar 20 18:10:30 PDT 2018---->>>null
>> Tue Mar 20 18:10:30 PDT 2018---->>>null
>> Tue Mar 20 18:10:50 PDT 2018---->>>null
>> Tue Mar 20 18:10:50 PDT 2018---->>>null
>>
>> So expiration works as expected to me. Am I missing something?
>>
>> -Val
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
> --
> /subash
>
--
/subash
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by Subash Chaturanga <su...@gmail.com>.
Hi Val,
Only one node running. And at step 2 when you say, wait for output and
stop, did you mean Ignite.stop() or kill -9 jvm.
What I do is kill -9.
On Tue, Mar 20, 2018 at 9:21 PM vkulichenko <va...@gmail.com>
wrote:
> Subash,
>
> When you run the test, is this the only node, or there are some other nodes
> running? I actually can't reproduce the issue even with your code. Here is
> what I do:
> - Clean up the work directory.
> - Run the test, wait for output, stop.
> - Comment out 'put' invocations.
> - Run again.
>
> The output on the second one is the following:
>
> Tue Mar 20 18:10:30 PDT 2018---->>>null
> Tue Mar 20 18:10:30 PDT 2018---->>>null
> Tue Mar 20 18:10:50 PDT 2018---->>>null
> Tue Mar 20 18:10:50 PDT 2018---->>>null
>
> So expiration works as expected to me. Am I missing something?
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
/subash
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by vkulichenko <va...@gmail.com>.
Subash,
When you run the test, is this the only node, or there are some other nodes
running? I actually can't reproduce the issue even with your code. Here is
what I do:
- Clean up the work directory.
- Run the test, wait for output, stop.
- Comment out 'put' invocations.
- Run again.
The output on the second one is the following:
Tue Mar 20 18:10:30 PDT 2018---->>>null
Tue Mar 20 18:10:30 PDT 2018---->>>null
Tue Mar 20 18:10:50 PDT 2018---->>>null
Tue Mar 20 18:10:50 PDT 2018---->>>null
So expiration works as expected to me. Am I missing something?
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by Subash Chaturanga <su...@gmail.com>.
If you can send me the working code I can try to compare what’s missing.
On Thu, Mar 15, 2018 at 10:14 AM Subash Chaturanga <su...@gmail.com>
wrote:
> Here’s my code.
>
> *public static void *main(String[] args) *throws*InterruptedException {
>
> String region = *"4GRegion"*;
> String cacheName = *"bar2"*;
> String path = *"C:**\\**dev**\\**dpsrc**\\**simple-java**\\*
> *work"*;
>
> DataStorageConfiguration storageCfg = *new*
> DataStorageConfiguration();
>
> DataRegionConfiguration regionCfg = *new*
> DataRegionConfiguration();
> regionCfg.setName(region);
> regionCfg.setInitialSize(10L * 1024 *1024);
> regionCfg.setMaxSize(100L * 1024 * 1024);
> regionCfg.setPageEvictionMode(DataPageEvictionMode.*RANDOM_LRU*);
> regionCfg.setPersistenceEnabled(*true*);
>
> storageCfg.setDataRegionConfigurations(regionCfg);
>
> IgniteConfiguration cfg = *new*IgniteConfiguration();
> cfg.setWorkDirectory(path);
> cfg.setDataStorageConfiguration(storageCfg);
> TcpDiscoverySpi spi = *new*TcpDiscoverySpi();
> TcpDiscoveryVmIpFinder ipFinder = *new*TcpDiscoveryVmIpFinder();
> ipFinder.setAddresses(Arrays.*asList*(*"127.0.0.1:47500..47509"*
> ));
> spi.setIpFinder(ipFinder);
> cfg.setDiscoverySpi(spi);
>
> Ignite ignite = Ignition.*start*(cfg);
>
> ignite.active(*true*);
>
> CacheConfiguration<String,String> cc = *new *
> CacheConfiguration<>();
> cc.setDataRegionName(region);
> cc.setPartitionLossPolicy(PartitionLossPolicy.*READ_WRITE_ALL*);
> cc.setName(cacheName);
> cc.setCacheMode(CacheMode.*PARTITIONED*);
> cc.setGroupName(cacheName);
> cc.setExpiryPolicyFactory(CreatedExpiryPolicy.*factoryOf*(*new*
> Duration(TimeUnit.*SECONDS*, 15)));
> cc.setStatisticsEnabled(*true*);
> ignite.resetLostPartitions(Arrays.*asList*(*new *
> String[]{cacheName}));
>
>
>
>
> *// ignite.getOrCreateCache(cc).put("A1","V1");//
> ignite.getOrCreateCache(cc).put("A2","V2"); *System.*out*.println(
> *new *Date().toString() + *"---->>>" *+ ignite.getOrCreateCache(cc).get(
> *"A1"*));
> System.*out*.println(*new *Date().toString() + *"---->>>" *+
> ignite.getOrCreateCache(cc).get(*"A2"*));
>
> Thread.*sleep*(20*1000);
>
> System.*out*.println(*new *Date().toString() + *"---->>>" *+
> ignite.getOrCreateCache(cc).get(*"A1"*));
> System.*out*.println(*new *Date().toString() + *"---->>>" *+
> ignite.getOrCreateCache(cc).get(*"A2"*));
> }
>
>
>
> On Wed, Mar 14, 2018 at 5:07 PM vkulichenko <va...@gmail.com>
> wrote:
>
>> Subash,
>>
>> This is weird, I'm doing exactly the same and not able to reproduce the
>> issue. Can you share your whole test so that I can run it as-is?
>>
>> -Val
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
> --
> /subash
>
--
/subash
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by Subash Chaturanga <su...@gmail.com>.
Here’s my code.
*public static void *main(String[] args) *throws*InterruptedException {
String region = *"4GRegion"*;
String cacheName = *"bar2"*;
String path = *"C:**\\**dev**\\**dpsrc**\\**simple-java**\\**work"*;
DataStorageConfiguration storageCfg = *new*
DataStorageConfiguration();
DataRegionConfiguration regionCfg = *new*DataRegionConfiguration();
regionCfg.setName(region);
regionCfg.setInitialSize(10L * 1024 *1024);
regionCfg.setMaxSize(100L * 1024 * 1024);
regionCfg.setPageEvictionMode(DataPageEvictionMode.*RANDOM_LRU*);
regionCfg.setPersistenceEnabled(*true*);
storageCfg.setDataRegionConfigurations(regionCfg);
IgniteConfiguration cfg = *new*IgniteConfiguration();
cfg.setWorkDirectory(path);
cfg.setDataStorageConfiguration(storageCfg);
TcpDiscoverySpi spi = *new*TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = *new*TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.*asList*(*"127.0.0.1:47500..47509"*));
spi.setIpFinder(ipFinder);
cfg.setDiscoverySpi(spi);
Ignite ignite = Ignition.*start*(cfg);
ignite.active(*true*);
CacheConfiguration<String,String> cc = *new *CacheConfiguration<>();
cc.setDataRegionName(region);
cc.setPartitionLossPolicy(PartitionLossPolicy.*READ_WRITE_ALL*);
cc.setName(cacheName);
cc.setCacheMode(CacheMode.*PARTITIONED*);
cc.setGroupName(cacheName);
cc.setExpiryPolicyFactory(CreatedExpiryPolicy.*factoryOf*(*new*
Duration(TimeUnit.*SECONDS*, 15)));
cc.setStatisticsEnabled(*true*);
ignite.resetLostPartitions(Arrays.*asList*(*new *
String[]{cacheName}));
*// ignite.getOrCreateCache(cc).put("A1","V1");//
ignite.getOrCreateCache(cc).put("A2","V2"); *System.*out*.println(
*new *Date().toString() + *"---->>>" *+ ignite.getOrCreateCache(cc).get(
*"A1"*));
System.*out*.println(*new *Date().toString() + *"---->>>" *+
ignite.getOrCreateCache(cc).get(*"A2"*));
Thread.*sleep*(20*1000);
System.*out*.println(*new *Date().toString() + *"---->>>" *+
ignite.getOrCreateCache(cc).get(*"A1"*));
System.*out*.println(*new *Date().toString() + *"---->>>" *+
ignite.getOrCreateCache(cc).get(*"A2"*));
}
On Wed, Mar 14, 2018 at 5:07 PM vkulichenko <va...@gmail.com>
wrote:
> Subash,
>
> This is weird, I'm doing exactly the same and not able to reproduce the
> issue. Can you share your whole test so that I can run it as-is?
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
/subash
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by vkulichenko <va...@gmail.com>.
Subash,
This is weird, I'm doing exactly the same and not able to reproduce the
issue. Can you share your whole test so that I can run it as-is?
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by Subash Chaturanga <su...@gmail.com>.
Hi Val, I am using ignite 2.3
Have a simple java Main method that has following
1. dataStorageConfig.set(dataRegionConfig)
2. ignitconfig.setDataStorageConfig(dataStorageConfig)
3. IgniteConfig.setDiscoverySpi(spi)
4.Ignition.start(igniteConfig)
5. CacheConfiguration cc = buildFromDataRegion(regionName); // sets
CreatedExpiryPolicy to 15 sec
6. Cache c = ignite.getOrCreate(cc)
7. c.put(“k”, “v”);
8. Log(cache.get(“k”);
9. Sleep(16000);
10. Log(cache.get(“k”);
at first run, when it comes to line 10, it’s expired.
Now comment line 7. I should expect it should print null in line 8 when JVM
restarts. But it gives me value.
On Tue, Mar 13, 2018 at 8:34 PM vkulichenko <va...@gmail.com>
wrote:
> I was actually not correct here. Although there are some known issues,
> expiration is supposed to work with both memory and persistence.
>
> Subash, can you provide more details on how you reproduce the issue? I'm
> getting null even after I restart the node. Are you doing anything else
> there?
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
/subash
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by vkulichenko <va...@gmail.com>.
I was actually not correct here. Although there are some known issues,
expiration is supposed to work with both memory and persistence.
Subash, can you provide more details on how you reproduce the issue? I'm
getting null even after I restart the node. Are you doing anything else
there?
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by Dmitry Pavlov <dp...@gmail.com>.
Thank you for explanation. Currently Ignite does not have any settings
which allow to limit used disk space in persistent storage.
вт, 13 мар. 2018 г. в 21:06, Subash Chaturanga <su...@gmail.com>:
> I mean, for example; we provide max value 500MB for durable memory under
> eviction policy, that will make sure Ignite will only use 500MB from the
> RAM no matter what. When native persistence comes into the picture, Ignite
> store entries in disk. So if we run on production, we want to make sure we
> don't run out of disk space for any reason. So something configurable like
> X GBs max for persistence.
>
> On Tue, Mar 13, 2018 at 3:30 AM, Dmitry Pavlov <dp...@gmail.com>
> wrote:
>
>> Hi, I'm not sure I understand what high watermark for persistence mean.
>> Could you please explain?
>>
>> пн, 12 мар. 2018 г. в 19:43, Subash Chaturanga <su...@gmail.com>:
>>
>>> Thank you very much for the responses.
>>>
>>> Will keep an eye on when it will be released. Any estimated release for
>>> the fix ?
>>>
>>> One more question I asked was, can we provide a high watermark for
>>> persistence store ?
>>>
>>> On Mon, Mar 12, 2018 at 12:24 PM vkulichenko <
>>> valentin.kulichenko@gmail.com> wrote:
>>>
>>>> Currently expired entry is removed only from memory. The same will be
>>>> supported for persistence layer after this ticket is implemented:
>>>> https://issues.apache.org/jira/browse/IGNITE-5874.
>>>>
>>>> -Val
>>>>
>>>>
>>>>
>>>> --
>>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>>
>>> --
>>> /subash
>>>
>>
>
>
> --
> /subash
>
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by Subash Chaturanga <su...@gmail.com>.
I mean, for example; we provide max value 500MB for durable memory under
eviction policy, that will make sure Ignite will only use 500MB from the
RAM no matter what. When native persistence comes into the picture, Ignite
store entries in disk. So if we run on production, we want to make sure we
don't run out of disk space for any reason. So something configurable like
X GBs max for persistence.
On Tue, Mar 13, 2018 at 3:30 AM, Dmitry Pavlov <dp...@gmail.com>
wrote:
> Hi, I'm not sure I understand what high watermark for persistence mean.
> Could you please explain?
>
> пн, 12 мар. 2018 г. в 19:43, Subash Chaturanga <su...@gmail.com>:
>
>> Thank you very much for the responses.
>>
>> Will keep an eye on when it will be released. Any estimated release for
>> the fix ?
>>
>> One more question I asked was, can we provide a high watermark for
>> persistence store ?
>>
>> On Mon, Mar 12, 2018 at 12:24 PM vkulichenko <
>> valentin.kulichenko@gmail.com> wrote:
>>
>>> Currently expired entry is removed only from memory. The same will be
>>> supported for persistence layer after this ticket is implemented:
>>> https://issues.apache.org/jira/browse/IGNITE-5874.
>>>
>>> -Val
>>>
>>>
>>>
>>> --
>>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>>
>> --
>> /subash
>>
>
--
/subash
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by Dmitry Pavlov <dp...@gmail.com>.
Hi, I'm not sure I understand what high watermark for persistence mean.
Could you please explain?
пн, 12 мар. 2018 г. в 19:43, Subash Chaturanga <su...@gmail.com>:
> Thank you very much for the responses.
>
> Will keep an eye on when it will be released. Any estimated release for
> the fix ?
>
> One more question I asked was, can we provide a high watermark for
> persistence store ?
>
> On Mon, Mar 12, 2018 at 12:24 PM vkulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
>> Currently expired entry is removed only from memory. The same will be
>> supported for persistence layer after this ticket is implemented:
>> https://issues.apache.org/jira/browse/IGNITE-5874.
>>
>> -Val
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
> --
> /subash
>
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by Subash Chaturanga <su...@gmail.com>.
Thank you very much for the responses.
Will keep an eye on when it will be released. Any estimated release for the
fix ?
One more question I asked was, can we provide a high watermark for
persistence store ?
On Mon, Mar 12, 2018 at 12:24 PM vkulichenko <va...@gmail.com>
wrote:
> Currently expired entry is removed only from memory. The same will be
> supported for persistence layer after this ticket is implemented:
> https://issues.apache.org/jira/browse/IGNITE-5874.
>
> -Val
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
--
/subash
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by vkulichenko <va...@gmail.com>.
Currently expired entry is removed only from memory. The same will be
supported for persistence layer after this ticket is implemented:
https://issues.apache.org/jira/browse/IGNITE-5874.
-Val
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by Subash Chaturanga <su...@gmail.com>.
Ideally 3 nodes.
So expiry inconsistency is a known issue ?
But I reproduce the expiry inconsistency issue even in a single node with
the steps mentioned earlier.
We are evaluating Ignite vs Redis for our use case these days, and wanted
to make sure everything works fine as in docs. Can you please confirm the
cache eviction bahavior also, like can we put a limit to the persistence
later too ?
On Thu, Mar 8, 2018 at 6:51 AM Dmitry Pavlov <dp...@gmail.com> wrote:
> Hi,
>
> There was some well known issue with native persistence and rebalancing
> data between nodes, later I can try to find its Id .
>
> How much nodes do you use?
>
> Sincerely,
> Dmitry Pavlov
>
> ср, 7 мар. 2018 г., 22:25 Subash Chaturanga <su...@gmail.com>:
>
>> Hi team,
>> With a cache having CreatedExpiryPolicy for 1min duration and having
>> ignite persistence enabled as per docs. And did this.
>>
>> - cache put and then cache get, gives me the value
>> - wait for 1min
>> - cache get, returns null. Perfectly fine up to now.
>> - then recycled JVM.
>> - now only cache.get, no puts, and return == null is my expectation. But
>> it wasn’t the case. It returned me the value.
>>
>> This means cache expiry doesn’t remove the entry from native persistent
>> layer. But if you do a cache.remove() it will remove it from native
>> persistent later too. Meaning JVM recycle will return null.
>>
>> So compared to cache.remove working as expected, cache expiry behavior is
>> very inconsistent.
>>
>> Can someone please clarify ?
>>
>> --
>> /subash
>>
> --
/subash
Re: Ignite Expiry Inconsistency with Native Persistence
Posted by Dmitry Pavlov <dp...@gmail.com>.
Hi,
There was some well known issue with native persistence and rebalancing
data between nodes, later I can try to find its Id .
How much nodes do you use?
Sincerely,
Dmitry Pavlov
ср, 7 мар. 2018 г., 22:25 Subash Chaturanga <su...@gmail.com>:
> Hi team,
> With a cache having CreatedExpiryPolicy for 1min duration and having
> ignite persistence enabled as per docs. And did this.
>
> - cache put and then cache get, gives me the value
> - wait for 1min
> - cache get, returns null. Perfectly fine up to now.
> - then recycled JVM.
> - now only cache.get, no puts, and return == null is my expectation. But
> it wasn’t the case. It returned me the value.
>
> This means cache expiry doesn’t remove the entry from native persistent
> layer. But if you do a cache.remove() it will remove it from native
> persistent later too. Meaning JVM recycle will return null.
>
> So compared to cache.remove working as expected, cache expiry behavior is
> very inconsistent.
>
> Can someone please clarify ?
>
> --
> /subash
>