You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@forrest.apache.org by Tim Williams <wi...@gmail.com> on 2006/02/11 02:17:31 UTC
more on locationmap caching
It seems to me that implementing CacheableProcessingComponent with an
input module like LM isn't feasible since there's only one instance of
it and it won't help us cache at the more granular level. I think I
confirmed that by following it through it's lifecycle this evening.
Anyway, Ross is on to correcting the validity issue so I thought I'd
spend some time on figuring out how to get away from our
homegrown-hashmap-cache. I think instead of trying to use the cocoon
cache, the answer is to manage validity ourselves and go directly to
the store. This gets our little lm cache "managed" with the real
cocoon store as i think it should be.
The only problem that I can foresee is our current transient store is set with:
<parameter name="maxobjects" value="100"/>
100 seems extremely small to me anyway but I think if we started to
use it for the lm, we'd find that it's definitely too small and would
spend as much resources cleaning itself as to make it not worth it.
Anyone know why it shouldn't be larger?
Does storing lm cached hints in the transient store seem reasonable?
Thanks,
--tim
Re: more on locationmap caching
Posted by Ross Gardler <rg...@apache.org>.
Tim Williams wrote:
> It seems to me that implementing CacheableProcessingComponent with an
> input module like LM isn't feasible since there's only one instance of
> it and it won't help us cache at the more granular level. I think I
> confirmed that by following it through it's lifecycle this evening.
>
> Anyway, Ross is on to correcting the validity issue so I thought I'd
> spend some time on figuring out how to get away from our
> homegrown-hashmap-cache.
That's brilliant, I love Open Source :-))
> I think instead of trying to use the cocoon
> cache, the answer is to manage validity ourselves and go directly to
> the store. This gets our little lm cache "managed" with the real
> cocoon store as i think it should be.
Cool.
> The only problem that I can foresee is our current transient store is set with:
>
> <parameter name="maxobjects" value="100"/>
>
> 100 seems extremely small to me anyway but I think if we started to
> use it for the lm, we'd find that it's definitely too small and would
> spend as much resources cleaning itself as to make it not worth it.
> Anyone know why it shouldn't be larger?
I'm afraid I know nothing. Just replying to show my support. However, if
I do read anything in the Cocoon docs I'll be sure to comment on them.
Ross
Re: more on locationmap caching
Posted by David Crossley <cr...@apache.org>.
Tim Williams wrote:
> It seems to me that implementing CacheableProcessingComponent with an
> input module like LM isn't feasible since there's only one instance of
> it and it won't help us cache at the more granular level. I think I
> confirmed that by following it through it's lifecycle this evening.
>
> Anyway, Ross is on to correcting the validity issue so I thought I'd
> spend some time on figuring out how to get away from our
> homegrown-hashmap-cache. I think instead of trying to use the cocoon
> cache, the answer is to manage validity ourselves and go directly to
> the store. This gets our little lm cache "managed" with the real
> cocoon store as i think it should be.
>
> The only problem that I can foresee is our current transient store is set with:
>
> <parameter name="maxobjects" value="100"/>
>
> 100 seems extremely small to me anyway but I think if we started to
> use it for the lm, we'd find that it's definitely too small and would
> spend as much resources cleaning itself as to make it not worth it.
> Anyone know why it shouldn't be larger?
We have many Cocoon defaults that need to be investigated
and fine-tuned. I think that we already have a Jira issue
for that. There are some cache tuning docs at Cocoon or
in their Wiki.
We should first re-compare our settings with cocoon-2.2
I see that we have some comments mixed up in our
WEB-INF/xconf/forrest-core.xconf so perhaps we have
some parameters out-of-sync too.
Now that we have the serverStatus plugin we can start to
refine our settings.
> Does storing lm cached hints in the transient store seem reasonable?
Using the stores in general sounds great.
-David
Re: more on locationmap caching
Posted by Ross Gardler <rg...@apache.org>.
I'll come back to this in about a week - other things have taken priory
right now.
In general I agree with your comments and will commit something ASAP.
Ross
Tim Williams wrote:
> On 2/15/06, Ross Gardler <rg...@apache.org> wrote:
>
>>Tim Williams wrote:
>>
>>>On 2/10/06, Tim Williams <wi...@gmail.com> wrote:
>>>
>>>
>>>>It seems to me that implementing CacheableProcessingComponent with an
>>>>input module like LM isn't feasible since there's only one instance of
>>>>it and it won't help us cache at the more granular level. I think I
>>>>confirmed that by following it through it's lifecycle this evening.
>>>>
>>>>Anyway, Ross is on to correcting the validity issue so I thought I'd
>>>>spend some time on figuring out how to get away from our
>>>>homegrown-hashmap-cache. I think instead of trying to use the cocoon
>>>>cache, the answer is to manage validity ourselves and go directly to
>>>>the store. This gets our little lm cache "managed" with the real
>>>>cocoon store as i think it should be.
>>>>
>>>>The only problem that I can foresee is our current transient store is set with:
>>>>
>>>><parameter name="maxobjects" value="100"/>
>>>>
>>>>100 seems extremely small to me anyway but I think if we started to
>>>>use it for the lm, we'd find that it's definitely too small and would
>>>>spend as much resources cleaning itself as to make it not worth it.
>>>>Anyone know why it shouldn't be larger?
>>>>
>>>>Does storing lm cached hints in the transient store seem reasonable?
>>>>
>>>>Thanks,
>>>>--tim
>>>
>>>
>>>
>>>FYI. I'm not very keen on implementing it yet, but we may be forced to
>>>decide just how important this is to us. We'll see if I get any other
>>>responses, but I'm not holding my breath...
>>>
>>>http://marc.theaimsgroup.com/?t=113995323300001&r=1&w=2
>>
>>Thanks for following through Tim.
>>
>>I've been looking at the use of MultiSourceValidity. It's not trivial
>>unfortunately. The way mounted sitemaps are handled is causing problems
>>(simply put they are handled differently depending on whether they are
>>in a select or not). We need to refactor the handling of mounted
>>locationmaps.
>
>
> By "handled" I gather you mean created and ran through the lifecycle.
> This is a result of our implementing the locationmap grammar "inline".
> In other words, we don't have a grammar and a treeprocessor with
> differentiation between nodes/nodebuilders. Each node is responsible
> for building and instantiating its allowable child nodes. When I
> first started looking into this stuff I considered whether a more
> robust approach (a la sitemap) should be taken or not but was
> [probably rightly] put off [1]. I suspect that you're seeing what I
> saw when I wrote that email.
>
>
>>I've got a solution, but it seems "hacky". I want to find the time to
>>look at how Cocoon handles mounted sitemaps and see if we can learn from
>>that.
>
>
> I think any solution is going to "seem" hacky. Cocoon has the benefit
> of a TreeProcessor which does most of the heavy lifting for it.
>
>
>>With respect to the use of Cocoons store Vs. our current home grown
>>solution. I think it is unlikely that we will hit the memory limits
>>within the locationmap. The items we are storing are small (strings) and
>>the number is relatively low. Even in a very large site it is unlikely
>>to go beyond a few thousand items.
>>
>>If it becomes a problem we can reconsider.
>
>
> Ok, we should at least comment on this in the source so that if we
> ever run into memory issues we know where to look. For example,
> without ?Ron? looking into our current memory consumption, it would
> have taken some time to find that the XmlFileModule is the culprit.
>
>
>>Speaking personally, I need the ability to invalidate the locationmap
>>tree when any of the mounted maps are changed. The move to
>>MultiSourceValidity will solve this problem. I'm not too concerned right
>>now about being able to control the locationmap cache from within the
>>sitemap, as long as we can control it from forrest.properties (which we
>>already can).
>
>
> How can we control it from forrest.properties? Right now, it requires
> a restart and it's either on or off in forrest.xconf.
>
> --tim
>
>
> [1] - http://marc.theaimsgroup.com/?l=xml-cocoon-dev&m=112112847425768&w=2
>
>
Re: more on locationmap caching
Posted by Tim Williams <wi...@gmail.com>.
On 2/15/06, Ross Gardler <rg...@apache.org> wrote:
> Tim Williams wrote:
> > On 2/10/06, Tim Williams <wi...@gmail.com> wrote:
> >
> >>It seems to me that implementing CacheableProcessingComponent with an
> >>input module like LM isn't feasible since there's only one instance of
> >>it and it won't help us cache at the more granular level. I think I
> >>confirmed that by following it through it's lifecycle this evening.
> >>
> >>Anyway, Ross is on to correcting the validity issue so I thought I'd
> >>spend some time on figuring out how to get away from our
> >>homegrown-hashmap-cache. I think instead of trying to use the cocoon
> >>cache, the answer is to manage validity ourselves and go directly to
> >>the store. This gets our little lm cache "managed" with the real
> >>cocoon store as i think it should be.
> >>
> >>The only problem that I can foresee is our current transient store is set with:
> >>
> >><parameter name="maxobjects" value="100"/>
> >>
> >>100 seems extremely small to me anyway but I think if we started to
> >>use it for the lm, we'd find that it's definitely too small and would
> >>spend as much resources cleaning itself as to make it not worth it.
> >>Anyone know why it shouldn't be larger?
> >>
> >>Does storing lm cached hints in the transient store seem reasonable?
> >>
> >>Thanks,
> >>--tim
> >
> >
> >
> > FYI. I'm not very keen on implementing it yet, but we may be forced to
> > decide just how important this is to us. We'll see if I get any other
> > responses, but I'm not holding my breath...
> >
> > http://marc.theaimsgroup.com/?t=113995323300001&r=1&w=2
>
> Thanks for following through Tim.
>
> I've been looking at the use of MultiSourceValidity. It's not trivial
> unfortunately. The way mounted sitemaps are handled is causing problems
> (simply put they are handled differently depending on whether they are
> in a select or not). We need to refactor the handling of mounted
> locationmaps.
By "handled" I gather you mean created and ran through the lifecycle.
This is a result of our implementing the locationmap grammar "inline".
In other words, we don't have a grammar and a treeprocessor with
differentiation between nodes/nodebuilders. Each node is responsible
for building and instantiating its allowable child nodes. When I
first started looking into this stuff I considered whether a more
robust approach (a la sitemap) should be taken or not but was
[probably rightly] put off [1]. I suspect that you're seeing what I
saw when I wrote that email.
> I've got a solution, but it seems "hacky". I want to find the time to
> look at how Cocoon handles mounted sitemaps and see if we can learn from
> that.
I think any solution is going to "seem" hacky. Cocoon has the benefit
of a TreeProcessor which does most of the heavy lifting for it.
> With respect to the use of Cocoons store Vs. our current home grown
> solution. I think it is unlikely that we will hit the memory limits
> within the locationmap. The items we are storing are small (strings) and
> the number is relatively low. Even in a very large site it is unlikely
> to go beyond a few thousand items.
>
> If it becomes a problem we can reconsider.
Ok, we should at least comment on this in the source so that if we
ever run into memory issues we know where to look. For example,
without ?Ron? looking into our current memory consumption, it would
have taken some time to find that the XmlFileModule is the culprit.
> Speaking personally, I need the ability to invalidate the locationmap
> tree when any of the mounted maps are changed. The move to
> MultiSourceValidity will solve this problem. I'm not too concerned right
> now about being able to control the locationmap cache from within the
> sitemap, as long as we can control it from forrest.properties (which we
> already can).
How can we control it from forrest.properties? Right now, it requires
a restart and it's either on or off in forrest.xconf.
--tim
[1] - http://marc.theaimsgroup.com/?l=xml-cocoon-dev&m=112112847425768&w=2
Re: more on locationmap caching
Posted by Ross Gardler <rg...@apache.org>.
Ross Gardler wrote:
> Tim Williams wrote:
>
>> On 2/10/06, Tim Williams <wi...@gmail.com> wrote:
>>
>>> It seems to me that implementing CacheableProcessingComponent with an
>>> input module like LM isn't feasible since there's only one instance of
>>> it and it won't help us cache at the more granular level. I think I
>>> confirmed that by following it through it's lifecycle this evening.
>>>
>>> Anyway, Ross is on to correcting the validity issue so I thought I'd
>>> spend some time on figuring out how to get away from our
>>> homegrown-hashmap-cache. I think instead of trying to use the cocoon
>>> cache, the answer is to manage validity ourselves and go directly to
>>> the store. This gets our little lm cache "managed" with the real
>>> cocoon store as i think it should be.
>>>
>>> The only problem that I can foresee is our current transient store is
>>> set with:
>>>
>>> <parameter name="maxobjects" value="100"/>
>>>
>>> 100 seems extremely small to me anyway but I think if we started to
>>> use it for the lm, we'd find that it's definitely too small and would
>>> spend as much resources cleaning itself as to make it not worth it.
>>> Anyone know why it shouldn't be larger?
>>>
>>> Does storing lm cached hints in the transient store seem reasonable?
>>>
>>> Thanks,
>>> --tim
>>
>>
>>
>>
>> FYI. I'm not very keen on implementing it yet, but we may be forced to
>> decide just how important this is to us. We'll see if I get any other
>> responses, but I'm not holding my breath...
>>
>> http://marc.theaimsgroup.com/?t=113995323300001&r=1&w=2
>
>
> Thanks for following through Tim.
>
> I've been looking at the use of MultiSourceValidity. It's not trivial
> unfortunately. The way mounted sitemaps are handled is causing problems
> (simply put they are handled differently depending on whether they are
> in a select or not). We need to refactor the handling of mounted
> locationmaps.
That should read "the way mounted *locationmaps* are handled". This is a
Forrest issue, not a Cocoon one.
Ross
> I've got a solution, but it seems "hacky". I want to find the time to
> look at how Cocoon handles mounted sitemaps and see if we can learn from
> that.
>
> With respect to the use of Cocoons store Vs. our current home grown
> solution. I think it is unlikely that we will hit the memory limits
> within the locationmap. The items we are storing are small (strings) and
> the number is relatively low. Even in a very large site it is unlikely
> to go beyond a few thousand items.
>
> If it becomes a problem we can reconsider.
>
> Speaking personally, I need the ability to invalidate the locationmap
> tree when any of the mounted maps are changed. The move to
> MultiSourceValidity will solve this problem. I'm not too concerned right
> now about being able to control the locationmap cache from within the
> sitemap, as long as we can control it from forrest.properties (which we
> already can).
>
> Ross
>
>
Re: more on locationmap caching
Posted by Ross Gardler <rg...@apache.org>.
Tim Williams wrote:
> On 2/10/06, Tim Williams <wi...@gmail.com> wrote:
>
>>It seems to me that implementing CacheableProcessingComponent with an
>>input module like LM isn't feasible since there's only one instance of
>>it and it won't help us cache at the more granular level. I think I
>>confirmed that by following it through it's lifecycle this evening.
>>
>>Anyway, Ross is on to correcting the validity issue so I thought I'd
>>spend some time on figuring out how to get away from our
>>homegrown-hashmap-cache. I think instead of trying to use the cocoon
>>cache, the answer is to manage validity ourselves and go directly to
>>the store. This gets our little lm cache "managed" with the real
>>cocoon store as i think it should be.
>>
>>The only problem that I can foresee is our current transient store is set with:
>>
>><parameter name="maxobjects" value="100"/>
>>
>>100 seems extremely small to me anyway but I think if we started to
>>use it for the lm, we'd find that it's definitely too small and would
>>spend as much resources cleaning itself as to make it not worth it.
>>Anyone know why it shouldn't be larger?
>>
>>Does storing lm cached hints in the transient store seem reasonable?
>>
>>Thanks,
>>--tim
>
>
>
> FYI. I'm not very keen on implementing it yet, but we may be forced to
> decide just how important this is to us. We'll see if I get any other
> responses, but I'm not holding my breath...
>
> http://marc.theaimsgroup.com/?t=113995323300001&r=1&w=2
Thanks for following through Tim.
I've been looking at the use of MultiSourceValidity. It's not trivial
unfortunately. The way mounted sitemaps are handled is causing problems
(simply put they are handled differently depending on whether they are
in a select or not). We need to refactor the handling of mounted
locationmaps.
I've got a solution, but it seems "hacky". I want to find the time to
look at how Cocoon handles mounted sitemaps and see if we can learn from
that.
With respect to the use of Cocoons store Vs. our current home grown
solution. I think it is unlikely that we will hit the memory limits
within the locationmap. The items we are storing are small (strings) and
the number is relatively low. Even in a very large site it is unlikely
to go beyond a few thousand items.
If it becomes a problem we can reconsider.
Speaking personally, I need the ability to invalidate the locationmap
tree when any of the mounted maps are changed. The move to
MultiSourceValidity will solve this problem. I'm not too concerned right
now about being able to control the locationmap cache from within the
sitemap, as long as we can control it from forrest.properties (which we
already can).
Ross
Re: more on locationmap caching
Posted by Tim Williams <wi...@gmail.com>.
On 2/10/06, Tim Williams <wi...@gmail.com> wrote:
> It seems to me that implementing CacheableProcessingComponent with an
> input module like LM isn't feasible since there's only one instance of
> it and it won't help us cache at the more granular level. I think I
> confirmed that by following it through it's lifecycle this evening.
>
> Anyway, Ross is on to correcting the validity issue so I thought I'd
> spend some time on figuring out how to get away from our
> homegrown-hashmap-cache. I think instead of trying to use the cocoon
> cache, the answer is to manage validity ourselves and go directly to
> the store. This gets our little lm cache "managed" with the real
> cocoon store as i think it should be.
>
> The only problem that I can foresee is our current transient store is set with:
>
> <parameter name="maxobjects" value="100"/>
>
> 100 seems extremely small to me anyway but I think if we started to
> use it for the lm, we'd find that it's definitely too small and would
> spend as much resources cleaning itself as to make it not worth it.
> Anyone know why it shouldn't be larger?
>
> Does storing lm cached hints in the transient store seem reasonable?
>
> Thanks,
> --tim
FYI. I'm not very keen on implementing it yet, but we may be forced to
decide just how important this is to us. We'll see if I get any other
responses, but I'm not holding my breath...
http://marc.theaimsgroup.com/?t=113995323300001&r=1&w=2
--tim