my doubts about the current CocoonStoreJanitor/StoreJanitorImpl

[Ard Schrijvers <a....@hippo.nl>]

Hello,

I am not sure if everybody is familiar with the CocoonStoreJanitor/StoreJanitorImpl, but in short it tries to free memory from cocoon's different caches, and does this according some configuration. For example, 

<store-janitor logger="core.store.janitor">
    <!-- How much free memory shall be available in the jvm -->
    <parameter name="freememory" value="2048000"/>
    <!-- Indicates the limit of the jvm memory consumption. The default max
         heapsize for Sun's JVM is (almost) 64Mb -->
    <parameter name="heapsize" value="30000000"/>
    <!-- How often shall the cleanup thread check memory -->
    <parameter name="cleanupthreadinterval" value="10"/>
    <!-- Experimental adaptive algorithm for cleanup interval
    <parameter name="adaptivethreadinterval" value="true"/>
    --> 
    <!-- Indicates the thread priority of the cleanup thread -->
    <parameter name="threadpriority" value="5"/>
    <!-- How much percent of the elements of each registered Store
         shall be removed when low on memory. Default 10% -->
    <parameter name="percent_to_free" value="10"/>
    <!-- Invoke the garbage collector when low memory is reached -->
    <parameter name="invokegc" value="false"/>
</store-janitor>

Now, suppose, the JVM is low on memory. Now, I am used to have 3 stores in my apps, namely defaultTransientStore, eventAwareTransientStore and the EHDefaultStore. I am not sure how about projects of other people/companies, but I know I can only get some memory free from the EHDefaultStore. The other stores either store very small responses, or, like the defaultTransientStore, it only caches a couple of xslt's, which are needed for so many pages, that removing them makes no sense. The StoreJanitorImpl just removes keys+responses from one single cache at a time, and moves to the next cache the next time. 

In my opinion, the StoreJanitorImpl makes my cocoon app even worse than before freeing the cache memory is the following:
1) It removes from caches I think no gain is in
2) It removes cachekeys from the EHDefaultStore in the worst possible way: because the eviction policy from ehcache is not available from within the EHDefaultStore, cachekeys + reponses are just removed by starting with index 0. These are frequently the more important keys, which are regenerated on the next request (so this request is uncached, making it even harder for the JVM to stay away from OOM).

I have had an email conversation with Greg Luck about this, the main developer from ehcache. Since you cannot remove cachekeys + responses according the eviction policy defined in the ehcache, he came up with the idea to add keys with empty responses. 

So, in the StoreJanitor, I check now wether the store is instanceof EHDefaultStore, and if so, and if the number of keys in memoryStore is larger then the (maxobjects - limit) (limit is computed in calcToFree(store) as the number of keys to be removed), I add #limit number of keys with response "". This means, that when maxobjects is reached, you can have 2 cases:

1) overflowToDisk=true: the reponse is serialized to diskStore, and the cachekey is moved to diskStore cachekey array (by the way still in memory, but the response not anymore )
2) overflowToDisk=false: the cachekey + response is removed entirely

Now, at the end the added "dummy keys" are removed.

So, 

WDYT about this new way to free memory from the store. And WDYT about only trying to free memory from the EHDefaultStore?

Regards Ard

-- 

Hippo
Oosteinde 11
1017WT Amsterdam
The Netherlands
Tel  +31 (0)20 5224466
-------------------------------------------------------------
a.schrijvers@hippo.nl / http://www.hippo.nl
-------------------------------------------------------------- 

Re: my doubts about the current CocoonStoreJanitor/StoreJanitorImpl

[Vadim Gritsenko <va...@reverycodes.com>]

Antonio Gallardo wrote:
> Vadim Gritsenko escribió:
>> Ard Schrijvers wrote:
>>>
>>> Now, suppose, the JVM is low on memory. Now, I am used to have 3 
>>> stores in my apps, namely defaultTransientStore, 
>>> eventAwareTransientStore and the EHDefaultStore. I am not sure how 
>>> about projects of other people/companies,
>>
>> (we don't use EHDefaultStore in production. Since you've seen the 
>> source code, you already know why)
> Hi Vadim,
> 
> Would you share what are you using instead of the EHDefaultStore? JCS?

I'm using DefaultStore everywhere, at the moment.

I noticed that latest EHCache does support storing of Objects now, so it looks 
like (with some work) it can take on role of the default Cocoon cache.

JGroups integration is another interesting feature.


Vadim

Re: my doubts about the current CocoonStoreJanitor/StoreJanitorImpl

[Antonio Gallardo <ag...@agssa.net>]

Vadim Gritsenko escribió:
> Ard Schrijvers wrote:
>>
>> Now, suppose, the JVM is low on memory. Now, I am used to have 3 
>> stores in my apps, namely defaultTransientStore, 
>> eventAwareTransientStore and the EHDefaultStore. I am not sure how 
>> about projects of other people/companies,
>
> (we don't use EHDefaultStore in production. Since you've seen the 
> source code, you already know why)
Hi Vadim,

Would you share what are you using instead of the EHDefaultStore? JCS?

Best Regards,

Antonio Gallardo.

Re: my doubts about the current CocoonStoreJanitor/StoreJanitorImpl

[Vadim Gritsenko <va...@reverycodes.com>]

Ard Schrijvers wrote:
> 
> Now, suppose, the JVM is low on memory. Now, I am used to have 3 stores in my apps, namely defaultTransientStore, eventAwareTransientStore and the EHDefaultStore. I am not sure how about projects of other people/companies,

(we don't use EHDefaultStore in production. Since you've seen the source code, 
you already know why)


> but I know I can only get some memory free from the EHDefaultStore. The other stores either store very small responses, or, like the defaultTransientStore, it only caches a couple of xslt's, which are needed for so many pages, that removing them makes no sense. The StoreJanitorImpl just removes keys+responses from one single cache at a time, and moves to the next cache the next time. 

I can suggest you two easy enhancements to improve the situation.

1. Stores should have configuration which specifies whether store should 
register itself with the StoreJanitor or not. Default transient store can be 
configured to *not* register with janitor, so that all your XSLTs stay in memory.

2. StoreJanitor can have another eviction algorithm, namely, it should have an 
option to remove configured % of entries from *all* registered stores. So that 
it empties some part of all stores on each eviction run. It would make system 
behavior more even and stable, in situations were there are multiple stores with 
varying object sizes.


> I have had an email conversation with Greg Luck about this, the main developer from ehcache. Since you cannot remove cachekeys + responses according the eviction policy defined in the ehcache, he came up with the idea to add keys with empty responses. 

 From my POV, this is a flaw in the ehcache: if there is an eviction policy, 
there should be a way to use it.


> So, in the StoreJanitor, I check now wether the store is instanceof EHDefaultStore,

That's a really bad idea, in and by itself. Never rely on concrete 
implementations, always work against contracts (interfaces).


Vadim