Turning off default MRU store

[Corin Moss <Co...@tvnz.co.nz>]

Hi Guys,

I might be getting ahead of myself a bit here, but I'm going to try and
turn off the default MRU store, in favour of the JCS based persistent
store.  I'd like to try some tests on performance without the default
MRU - has anyone else tried anything similar? I've simply set the core
store's role to point to the JCS store implementation.

The rationale behind this is that JCS implements a MRU store all on its
own - this simplifies things a bit as far as I'm concerned

Any thoughts? :)

Corin

Corin Moss
Lead Developer
TVNZ Interactive

+64 9 916 7367
+64 21 403 054
corin.moss@tvnz.co.nz




================================================================
CAUTION: This e-mail and any attachment(s) contains information
that is intended to be read only by the named recipient(s). It
may contain information that is confidential, proprietary or the
subject of legal privilege. This information is not to be used by
any other person and/or organisation. If you are not the intended
recipient, please advise us immediately and delete this e-mail
from your system. Do not use any information contained in it.

================================================================
For more information on the Television New Zealand Group, visit
us online at http://www.tvnz.co.nz
================================================================

Re: Turning off default MRU store

[Antonio Gallardo <ag...@agssa.net>]

Unico Hommes dijo:
> Geoff Howard wrote:
>
>> Unico Hommes wrote:
>>
>>> Sylvain Wallez wrote:
>>>
>>>> Unico Hommes wrote:
>>>>
>>>>>
>>>>> Unico Hommes wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> Corin Moss wrote:
>>>>>>
>>>>>>> Hi Guys,
>>>>>>>
>>>>>>> I might be getting ahead of myself a bit here, but I'm going to
>>>>>>> try and turn off the default MRU store, in favour of the JCS
>>>>>>> based persistent store.  I'd like to try some tests on
>>>>>>> performance without the default MRU - has anyone else tried
>>>>>>> anything similar? I've simply set the core store's role to point
>>>>>>> to the JCS store implementation.
>>>>>>>
>>>>>>
>>>>>> I guess I already got ahead of you when I renamed
>>>>>> JCSPersistentStore JCSStore just now :-) (And merged it with the
>>>>>> AbstractJCSStore BTW). It seems to me that JCS is both and it
>>>>>> could replace all three stores: DefaultStore, TransientStore and
>>>>>> PersistentStore.
>>>>>>
>>>>>
>>>>> I have to add that this is not the whole story. We do actually need
>>>>> to distinguish between persistent and transient storage. Some
>>>>> components explicitly want to persist some data instead of just
>>>>> cache it for speed. But as far as caching is concerned I think it
>>>>> we can leave it completely up to JCS.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Some components are explicitely using the transient-store to keep
>>>> data that shouldn't (or cannot) be serialized. But AFAIK, no
>>>> component directly uses the persistent-store except the store itself.
>>>>
>>>
>>> To my knowlegde there is one in eventcache block that uses the
>>> PersistentStore to persist some info it needs to recover upon next
>>> startup.
>>>
>>> It does not look to me as though JCS would fit the PersistentStore
>>> role very well. I was thinking that perhaps. We could have JCSStore
>>> as a replacement for TransientStore and Store roles and use
>>> FilesystemStore for the PersistentStore role.
>>
>>
>>
>> Wait a minute.  The original issue that brought JCS to the front as
>> that the persistent store was broken and had license problems.  Are
>> you saying that JCS isn't a good candidate for persisting the cached
>> info to disk, or that the fact that it by default has an MRU memory
>> front-end makes it not map directly to persistent-store role cleanly?
>>
> IIUC, JCS will always use a memory store in front yes. And it suggests
> on the JCS website that the persistence process is not very reliable:
>
> from http://jakarta.apache.org/turbine/jcs/IndexedDiskAuxCache.html :
>
> "When the disk cache is properly shutdown, the memory index is written
> to disk and the value file is defragmented. When the cache starts up,
> the disk cache can be configured to read or delete the index file. This
> provides an unreliable persistence mechanism."
>
>> The use of persistent store in eventcache shouldn't be a problem with
>> JCS as long as at shutdown, it persists everything it has in its MRU
>> if it hasn't already.  If there is some problem it would be much
>> better to just go back to the old serializing persistence for the
>> event cache data because the only benefit to putting it in the store
>> is that you more or less guaranteed that the events and cached
>> responses would be kept together.
>
>
> Yes, giving up that feature would be too bad.

Why?

If this is a "must needed feature" we can collaborate with the JCS team to
extend the work.

WDYT?

Best Regards,

Antonio Gallardo.

Re: Turning off default MRU store

[Unico Hommes <un...@hippo.nl>]

Geoff Howard wrote:

> Unico Hommes wrote:
>
>> Sylvain Wallez wrote:
>>
>>> Unico Hommes wrote:
>>>
>>>>
>>>> Unico Hommes wrote:
>>>>
>>>>>
>>>>>
>>>>> Corin Moss wrote:
>>>>>
>>>>>> Hi Guys,
>>>>>>
>>>>>> I might be getting ahead of myself a bit here, but I'm going to 
>>>>>> try and turn off the default MRU store, in favour of the JCS 
>>>>>> based persistent store.  I'd like to try some tests on 
>>>>>> performance without the default MRU - has anyone else tried 
>>>>>> anything similar? I've simply set the core store's role to point 
>>>>>> to the JCS store implementation.
>>>>>>
>>>>>
>>>>> I guess I already got ahead of you when I renamed 
>>>>> JCSPersistentStore JCSStore just now :-) (And merged it with the 
>>>>> AbstractJCSStore BTW). It seems to me that JCS is both and it 
>>>>> could replace all three stores: DefaultStore, TransientStore and 
>>>>> PersistentStore.
>>>>>
>>>>
>>>> I have to add that this is not the whole story. We do actually need 
>>>> to distinguish between persistent and transient storage. Some 
>>>> components explicitly want to persist some data instead of just 
>>>> cache it for speed. But as far as caching is concerned I think it 
>>>> we can leave it completely up to JCS.
>>>
>>>
>>>
>>>
>>>
>>> Some components are explicitely using the transient-store to keep 
>>> data that shouldn't (or cannot) be serialized. But AFAIK, no 
>>> component directly uses the persistent-store except the store itself.
>>>
>>
>> To my knowlegde there is one in eventcache block that uses the 
>> PersistentStore to persist some info it needs to recover upon next 
>> startup.
>>
>> It does not look to me as though JCS would fit the PersistentStore 
>> role very well. I was thinking that perhaps. We could have JCSStore 
>> as a replacement for TransientStore and Store roles and use 
>> FilesystemStore for the PersistentStore role.
>
>
>
> Wait a minute.  The original issue that brought JCS to the front as 
> that the persistent store was broken and had license problems.  Are 
> you saying that JCS isn't a good candidate for persisting the cached 
> info to disk, or that the fact that it by default has an MRU memory 
> front-end makes it not map directly to persistent-store role cleanly?
>
IIUC, JCS will always use a memory store in front yes. And it suggests 
on the JCS website that the persistence process is not very reliable:

from http://jakarta.apache.org/turbine/jcs/IndexedDiskAuxCache.html :

"When the disk cache is properly shutdown, the memory index is written 
to disk and the value file is defragmented. When the cache starts up, 
the disk cache can be configured to read or delete the index file. This 
provides an unreliable persistence mechanism."

> The use of persistent store in eventcache shouldn't be a problem with 
> JCS as long as at shutdown, it persists everything it has in its MRU 
> if it hasn't already.  If there is some problem it would be much 
> better to just go back to the old serializing persistence for the 
> event cache data because the only benefit to putting it in the store 
> is that you more or less guaranteed that the events and cached 
> responses would be kept together.


Yes, giving up that feature would be too bad.

Unico

Re: Turning off default MRU store

[Geoff Howard <co...@leverageweb.com>]

Unico Hommes wrote:

> Sylvain Wallez wrote:
>
>> Unico Hommes wrote:
>>
>>>
>>> Unico Hommes wrote:
>>>
>>>>
>>>>
>>>> Corin Moss wrote:
>>>>
>>>>> Hi Guys,
>>>>>
>>>>> I might be getting ahead of myself a bit here, but I'm going to 
>>>>> try and turn off the default MRU store, in favour of the JCS based 
>>>>> persistent store.  I'd like to try some tests on performance 
>>>>> without the default MRU - has anyone else tried anything similar? 
>>>>> I've simply set the core store's role to point to the JCS store 
>>>>> implementation.
>>>>>
>>>>
>>>> I guess I already got ahead of you when I renamed 
>>>> JCSPersistentStore JCSStore just now :-) (And merged it with the 
>>>> AbstractJCSStore BTW). It seems to me that JCS is both and it could 
>>>> replace all three stores: DefaultStore, TransientStore and 
>>>> PersistentStore.
>>>>
>>>
>>> I have to add that this is not the whole story. We do actually need 
>>> to distinguish between persistent and transient storage. Some 
>>> components explicitly want to persist some data instead of just 
>>> cache it for speed. But as far as caching is concerned I think it we 
>>> can leave it completely up to JCS.
>>
>>
>>
>>
>> Some components are explicitely using the transient-store to keep 
>> data that shouldn't (or cannot) be serialized. But AFAIK, no 
>> component directly uses the persistent-store except the store itself.
>>
>
> To my knowlegde there is one in eventcache block that uses the 
> PersistentStore to persist some info it needs to recover upon next 
> startup.
>
> It does not look to me as though JCS would fit the PersistentStore 
> role very well. I was thinking that perhaps. We could have JCSStore as 
> a replacement for TransientStore and Store roles and use 
> FilesystemStore for the PersistentStore role.


Wait a minute.  The original issue that brought JCS to the front as that 
the persistent store was broken and had license problems.  Are you 
saying that JCS isn't a good candidate for persisting the cached info to 
disk, or that the fact that it by default has an MRU memory front-end 
makes it not map directly to persistent-store role cleanly?

The use of persistent store in eventcache shouldn't be a problem with 
JCS as long as at shutdown, it persists everything it has in its MRU if 
it hasn't already.  If there is some problem it would be much better to 
just go back to the old serializing persistence for the event cache data 
because the only benefit to putting it in the store is that you more or 
less guaranteed that the events and cached responses would be kept together.

Geoff

>> So if the JCSStore includes a MRU-like memory front-end, it can 
>> simply replace the <store> component in cocoon.xconf.
>>
> My thoughts exactly

Re: Turning off default MRU store

[Unico Hommes <un...@hippo.nl>]

Sylvain Wallez wrote:

> Unico Hommes wrote:
>
>>
>> Unico Hommes wrote:
>>
>>>
>>>
>>> Corin Moss wrote:
>>>
>>>> Hi Guys,
>>>>
>>>> I might be getting ahead of myself a bit here, but I'm going to try 
>>>> and turn off the default MRU store, in favour of the JCS based 
>>>> persistent store.  I'd like to try some tests on performance 
>>>> without the default MRU - has anyone else tried anything similar? 
>>>> I've simply set the core store's role to point to the JCS store 
>>>> implementation.
>>>>
>>>
>>> I guess I already got ahead of you when I renamed JCSPersistentStore 
>>> JCSStore just now :-) (And merged it with the AbstractJCSStore BTW). 
>>> It seems to me that JCS is both and it could replace all three 
>>> stores: DefaultStore, TransientStore and PersistentStore.
>>>
>>
>> I have to add that this is not the whole story. We do actually need 
>> to distinguish between persistent and transient storage. Some 
>> components explicitly want to persist some data instead of just cache 
>> it for speed. But as far as caching is concerned I think it we can 
>> leave it completely up to JCS.
>
>
>
> Some components are explicitely using the transient-store to keep data 
> that shouldn't (or cannot) be serialized. But AFAIK, no component 
> directly uses the persistent-store except the store itself.
>

To my knowlegde there is one in eventcache block that uses the 
PersistentStore to persist some info it needs to recover upon next startup.

It does not look to me as though JCS would fit the PersistentStore role 
very well. I was thinking that perhaps. We could have JCSStore as a 
replacement for TransientStore and Store roles and use FilesystemStore 
for the PersistentStore role.

> So if the JCSStore includes a MRU-like memory front-end, it can simply 
> replace the <store> component in cocoon.xconf.
>
My thoughts exactly

Unico

Re: Turning off default MRU store

[Sylvain Wallez <sy...@apache.org>]

Unico Hommes wrote:

>
> Unico Hommes wrote:
>
>>
>>
>> Corin Moss wrote:
>>
>>> Hi Guys,
>>>
>>> I might be getting ahead of myself a bit here, but I'm going to try 
>>> and turn off the default MRU store, in favour of the JCS based 
>>> persistent store.  I'd like to try some tests on performance without 
>>> the default MRU - has anyone else tried anything similar? I've 
>>> simply set the core store's role to point to the JCS store 
>>> implementation.
>>>
>>
>> I guess I already got ahead of you when I renamed JCSPersistentStore 
>> JCSStore just now :-) (And merged it with the AbstractJCSStore BTW). 
>> It seems to me that JCS is both and it could replace all three 
>> stores: DefaultStore, TransientStore and PersistentStore.
>>
>
> I have to add that this is not the whole story. We do actually need to 
> distinguish between persistent and transient storage. Some components 
> explicitly want to persist some data instead of just cache it for 
> speed. But as far as caching is concerned I think it we can leave it 
> completely up to JCS.


Some components are explicitely using the transient-store to keep data 
that shouldn't (or cannot) be serialized. But AFAIK, no component 
directly uses the persistent-store except the store itself.

So if the JCSStore includes a MRU-like memory front-end, it can simply 
replace the <store> component in cocoon.xconf.

Sylvain

-- 
Sylvain Wallez                                  Anyware Technologies
http://www.apache.org/~sylvain           http://www.anyware-tech.com
{ XML, Java, Cocoon, OpenSource }*{ Training, Consulting, Projects }

Re: Turning off default MRU store

[Unico Hommes <un...@hippo.nl>]

Unico Hommes wrote:
> 
> 
> Corin Moss wrote:
> 
>> Hi Guys,
>>
>> I might be getting ahead of myself a bit here, but I'm going to try and
>> turn off the default MRU store, in favour of the JCS based persistent
>> store.  I'd like to try some tests on performance without the default
>> MRU - has anyone else tried anything similar? I've simply set the core
>> store's role to point to the JCS store implementation.
>>
> 
> I guess I already got ahead of you when I renamed JCSPersistentStore 
> JCSStore just now :-) (And merged it with the AbstractJCSStore BTW). It 
> seems to me that JCS is both and it could replace all three stores: 
> DefaultStore, TransientStore and PersistentStore.
> 

I have to add that this is not the whole story. We do actually need to 
distinguish between persistent and transient storage. Some components 
explicitly want to persist some data instead of just cache it for speed. 
But as far as caching is concerned I think it we can leave it completely 
up to JCS.

Unico

Re: Turning off default MRU store

[Unico Hommes <un...@hippo.nl>]

Corin Moss wrote:
> Hi Guys,
> 
> I might be getting ahead of myself a bit here, but I'm going to try and
> turn off the default MRU store, in favour of the JCS based persistent
> store.  I'd like to try some tests on performance without the default
> MRU - has anyone else tried anything similar? I've simply set the core
> store's role to point to the JCS store implementation.
> 

I guess I already got ahead of you when I renamed JCSPersistentStore 
JCSStore just now :-) (And merged it with the AbstractJCSStore BTW). It 
seems to me that JCS is both and it could replace all three stores: 
DefaultStore, TransientStore and PersistentStore.

> The rationale behind this is that JCS implements a MRU store all on its
> own - this simplifies things a bit as far as I'm concerned
> 

Does it back memory stores with persistent stores in the same way that 
DefaultStore is doing? That is, a memory store upfront and a background 
task moving out items into a more persistent region when memory runs low?

Unico