You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cocoon.apache.org by rufio <ru...@op.pl> on 2003/08/11 01:25:30 UTC
forced caching of volatile data
Hi all
Is there a way to enfoce an asumption that volatile data (eg. stored in
DB) changes once a specific interval?
Now if we have 100 requests/s SQLTransformer querys DB 100 times/s,
right?
I'd like to set it up to query only once a, say minute and cache the
result.
Regards, Rufio
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: forced caching of volatile data
Posted by Miles Elam <mi...@pcextremist.com>.
Geoff Howard wrote:
> This is definitely an interesting idea, but I can't believe that this
> sort of backwards-incompatibility would fly. One option would be to
> put a null validity implementation in the Abstract* so subclasses
> don't have to do anything, but I can't see that happening in a 2.1
> branch now that it's released.
That's what I was suggesting: that the default implementation returns a
null validity. As far as missing the 2.1 branch, c'est la vie. As long
as 2.2 (or whatever) doesn't take as long to be released as 2.1, I have
no problem with that. ;-) I've been using CVS HEAD from time to time
anyway, and I can always patch it myself if need be. I'm not trying to
short-circuit the way development works. If people want it, they will
come. If not, I'll either use it anyway with a patch or, more likely,
use the implementation that someone thought up that does the job
better. (Like perhaps your event cache?)
>> Can anyone think of a use case where prevention of caching (not just
>> apathy about its cacheability) would be necessary? Is there a case
>> where a developer would say, "I don't care what the sitemap
>> maintainer says; My component must never be cached or exceptions
>> will fly."
>
>
> Yes, secure and sensitive data (which I'm working with now) often has
> the recommendation (sometimes requirement) to disallow caching.
I'd be shame that the sitemap maintainer and the component developer
would have such lack of communication. Nevertheless, what about
overriding getKey to return null to *enforce* a lack of caching. In
other words, effort must be made to make sure that the component (and
subsequent pipeline elements) cannot ever be cached. In other words,
making the behavior as it is today.
> The other centers around performance, which may not be an issue if
> done right. Stefano's recent RT on adaptive caching is getting at the
> cost/benefit analysis of caching overhead. Currently we do have some
> overhead - we do some instanceof's to determine cacheability. If done
> right, your idea wouldn't necessarily be worse but it might be.
I was under the impression that the determination of cacheability wasn't
anywhere near the overhead of actually caching (registering with the
cache manager, putting the output in a CachedResponse object, etc.).
But you're right, the only to tell for sure would be to implement it. I
just wanted to make sure that people were at least slightly receptive to
the idea.
>> By itself, the component still doesn't cache even when in a caching
>> pipeline. Only when the expiry is sent is the pipeline bludgeoned
>> into the cache. Would this still be considered a violation of the
>> contract?
>
>
> Another problem I see with this is it locks into a time-based
> (expires) cache validity. That is certainly the simplest but not
> always best. As I see it, there are three main classes of
> cache-invalidation all of which Cocoon should support, and all of
> which may make sense for any given component:
How so? There is nothing stopping a component from specifying its
proper mode of caching like it done today. The time-based validity is
only substituted when there is no other mode available (and expiry is
requested). In fact, there is nothing (that I can see) that would bar
other validity substitutions if the need arose. As it happens, without
proper knowledge of the resource, only time-based expiration seems
appropriate.
> That's why I think the factory/delegate solution cooked up by Unico
> might be a good solution if the end-user experience can be made
> uncomplicated.
The more, the merrier.
- Miles Elam
Re: forced caching of volatile data
Posted by Geoff Howard <co...@leverageweb.com>.
Miles Elam wrote:
> Gianugo Rabellino wrote:
>
> > Miles Elam wrote:
> >
> >> If the Abstracts (AbstractGenerator, AbstractTransformer, etc.) were
> >> updated to reflect this, most folks using Cocoon would only have to
> do a
> >> recompile. Folks who implemented the interfaces directly (Generator,
> >> Transformer, etc.) would have more to do, but a cut and paste from the
> >> appropriate Abstract would do in 90% of the cases I should think.
> >> (Assuming that the developer hasn't made their component cacheable
> >> already.)
> >
> > I can't really think of a reason for this not to work. The only problem
> > is that doing so would somehow break a contract: today if you extend
> > Abstract* you know that your class wont be cacheable, and this might be
> > done on purpose. I don't know if this might actually impact users, but I
> > sure can see a point here.
This is definitely an interesting idea, but I can't believe that this
sort of backwards-incompatibility would fly. One option would be to put
a null validity implementation in the Abstract* so subclasses don't have
to do anything, but I can't see that happening in a 2.1 branch now that
it's released.
> Hmmm... Actually, the extended class would still be uncacheable on its
> own. The change makes a system-wide policy change rather than a
> per-interface contract renegotiation.
>
> Can anyone think of a use case where prevention of caching (not just
> apathy about its cacheability) would be necessary? Is there a case
> where a developer would say, "I don't care what the sitemap maintainer
> says; My component must never be cached or exceptions will fly."
Yes, secure and sensitive data (which I'm working with now) often has
the recommendation (sometimes requirement) to disallow caching.
The other centers around performance, which may not be an issue if done
right. Stefano's recent RT on adaptive caching is getting at the
cost/benefit analysis of caching overhead. Currently we do have some
overhead - we do some instanceof's to determine cacheability. If done
right, your idea wouldn't necessarily be worse but it might be.
> By itself, the component still doesn't cache even when in a caching
> pipeline. Only when the expiry is sent is the pipeline bludgeoned into
> the cache. Would this still be considered a violation of the contract?
Another problem I see with this is it locks into a time-based (expires)
cache validity. That is certainly the simplest but not always best. As
I see it, there are three main classes of cache-invalidation all of
which Cocoon should support, and all of which may make sense for any
given component:
- Time based (expires)
- Source based (last-modified)
- Event based (discrete events)
That's why I think the factory/delegate solution cooked up by Unico
might be a good solution if the end-user experience can be made
uncomplicated.
Geoff
Re: forced caching of volatile data
Posted by Miles Elam <mi...@pcextremist.com>.
Gianugo Rabellino wrote:
> Miles Elam wrote:
>
>> If the Abstracts (AbstractGenerator, AbstractTransformer, etc.) were
>> updated to reflect this, most folks using Cocoon would only have to
do a
>> recompile. Folks who implemented the interfaces directly (Generator,
>> Transformer, etc.) would have more to do, but a cut and paste from the
>> appropriate Abstract would do in 90% of the cases I should think.
>> (Assuming that the developer hasn't made their component cacheable
>> already.)
>
> I can't really think of a reason for this not to work. The only problem
> is that doing so would somehow break a contract: today if you extend
> Abstract* you know that your class wont be cacheable, and this might be
> done on purpose. I don't know if this might actually impact users, but I
> sure can see a point here.
Hmmm... Actually, the extended class would still be uncacheable on its
own. The change makes a system-wide policy change rather than a
per-interface contract renegotiation.
Can anyone think of a use case where prevention of caching (not just
apathy about its cacheability) would be necessary? Is there a case
where a developer would say, "I don't care what the sitemap maintainer
says; My component must never be cached or exceptions will fly."
By itself, the component still doesn't cache even when in a caching
pipeline. Only when the expiry is sent is the pipeline bludgeoned into
the cache. Would this still be considered a violation of the contract?
- Miles Elam
Re: forced caching of volatile data
Posted by Gianugo Rabellino <gi...@apache.org>.
Miles Elam wrote:
> If the Abstracts (AbstractGenerator, AbstractTransformer, etc.) were
> updated to reflect this, most folks using Cocoon would only have to do a
> recompile. Folks who implemented the interfaces directly (Generator,
> Transformer, etc.) would have more to do, but a cut and paste from the
> appropriate Abstract would do in 90% of the cases I should think.
> (Assuming that the developer hasn't made their component cacheable
> already.)
I can't really think of a reason for this not to work. The only problem
is that doing so would somehow break a contract: today if you extend
Abstract* you know that your class wont be cacheable, and this might be
done on purpose. I don't know if this might actually impact users, but I
sure can see a point here.
Ciao,
--
Gianugo Rabellino
Pro-netics s.r.l. - http://www.pro-netics.com
Orixo, the XML business alliance - http://www.orixo.com
(Now blogging at: http://blogs.cocoondev.org/gianugo/)
Re: forced caching of volatile data
Posted by Miles Elam <mi...@pcextremist.com>.
Okay, here's a wonky thought: why not make all components cacheable?
...well ...sort of...
What percentage of content is cacheable right now? Perhaps a better
question would be what percentage would everyone like to be cacheable
right now? In a perfect world, almost everything would be cacheable and
allow simple cache expiration. There are certain things that we want to
be up to the moment, but by and large, the reason for this is that the
highly dynamic data is not accurately cacheable (cache doesn't expire
fast enough or the trigger for expiration isn't available). Even pages
that have the current time (assuming only year, month, day, hour and
minute displayed -- no nanoseconds et al.) can have a minute-level cache.
Currently, one must implement CacheableProcessingComponent and write
getKey and getValidity methods to cache a resource. What if all
PipelineComponents were CacheableProcessingComponents? The default
behavior of getKey would be to serialize the parameters and the value of
"src". The default behavior of getValidity would be to return null.
During pipeline processing, instead of checking if the pipeline
components are instanceof CacheableProcessingComponent, it simply grabs
the key and the validity. If the validity is null and the pipeline
doesn't have an expiry set, it is uncacheable just as before with only
the time taken for key generation lost. If the validity is null and the
pipeline has an expiry, a no-op validity object is created for the
component, and the CachedResponse object is created as is normal.
I know. I know. My pet project is warping my brain and making me see
through glasses polarized to see the problems at hand.
Nevertheless, if expiries can determine cacheability, then *everything*
is indeed potentially cacheable. Any XSPs could be invoked only once an
hour or once a minute from the sitemap where most other configurations
happen anyway. The cacheable XSP document in samples would become obsolete.
If the Abstracts (AbstractGenerator, AbstractTransformer, etc.) were
updated to reflect this, most folks using Cocoon would only have to do a
recompile. Folks who implemented the interfaces directly (Generator,
Transformer, etc.) would have more to do, but a cut and paste from the
appropriate Abstract would do in 90% of the cases I should think.
(Assuming that the developer hasn't made their component cacheable already.)
So am I off my rocker for proposing this?
- Miles Elam
Re: forced caching of volatile data
Posted by Geoff Howard <co...@leverageweb.com>.
Gianugo Rabellino wrote:
> Geoff Howard wrote:
>
>>> 2. There is no transformer yet (even if, agreed, it would be pretty
>>> easy to build one);
>>
>> True, but that seemed quite a bit less work than the solutions you
>> have been looking at. (though I was only following the discussion
>> casually)
>
> Not really, since it would be transparent to the user. With your
> solution the user would have to configure delegates for every non
> cacheable components.
I meant less work to develop - you're right that forced pipeline caching
based on an expires parameter might be less work for the user. See my
email to Miles with my concern though about only fitting one of the
three broad cache invalidation schemes which I think Cocoon should support.
>> I think of expires as solely in the realm of external to cocoon caches
>> (browser, or proxy) which is why proposing a cocoon-cache centered
>> solution seemed right to me.
>
> This is not the case actually. Expires are used internally too, since
> internally the cache implementation has this information available, so
> that : the Cocoon cache will serve a resource from the cache using the
> expires algorithm even to internal cocoon requests.
Hmm. I'll have to go back and look at that more carefully.
Geoff
Re: forced caching of volatile data
Posted by Gianugo Rabellino <gi...@apache.org>.
Geoff Howard wrote:
>> 2. There is no transformer yet (even if, agreed, it would be pretty
>> easy to build one);
>
> True, but that seemed quite a bit less work than the solutions you have
> been looking at. (though I was only following the discussion casually)
Not really, since it would be transparent to the user. With your
solution the user would have to configure delegates for every non
cacheable components.
> I think of expires as solely in the realm of external to cocoon caches
> (browser, or proxy) which is why proposing a cocoon-cache centered
> solution seemed right to me.
This is not the case actually. Expires are used internally too, since
internally the cache implementation has this information available, so
that : the Cocoon cache will serve a resource from the cache using the
expires algorithm even to internal cocoon requests.
>>> The second possibly simpler solution is to create a new pipeline
>>> implementation that always caches. So, when you have a pipeline that
>>> should be forced to cache, you segregate it into its own map:pipeline
>>> section with a parameter that speficies how long to cache.
>>
>>
>> Yes and no. We have been through this before, but again there is both
>> a backward incompatibilty to consider (the current
>> CachingProcessingPipeline supports Expires) and a fair amount of code
>> duplication needed between the two implementations. But again, it's
>> something to consider.
>
> It supports an expires header going to the client, not a SourceValidity
> object with an expire (invalid) timestamp.
Not really. As I said above, the expires information is kept internally
too. It's not a SourceValidity object but an attribute of cache entries,
which is checked by the pipeline to decide whether to go through each
SourceValidity object or just serve it right away.
> 2) Investigate the feasibility of wildcard matches on events (and
> whether they're even necessary). Some implementation ideas could
> possibly require additions to the interfaces, but I'd want to plan those
> in backwards compatible ways even though the alpha block status may not
> require it.
>
> The examples that have come up for this by the way are file system
> events and date/time based events. Both of these can be handled in
> other ways so they may not be necessary at all (why I've ignored them
> for now). If a directory structure is used for validity (say
> /opt/foo/bar/) and you want to signal that all of /opt/foo/ has been
> updated you'd want to do something like /opt/foo/* but the current
> implementation might not deal with this well.
This looks very interesting indeed. I'll for sure give it a run. :-)
Ciao,
--
Gianugo Rabellino
Pro-netics s.r.l. - http://www.pro-netics.com
Orixo, the XML business alliance - http://www.orixo.com
(Now blogging at: http://blogs.cocoondev.org/gianugo/)
Re: forced caching of volatile data
Posted by Geoff Howard <co...@leverageweb.com>.
Gianugo,
Sorry for the long delay. Mini-vacation and only had this email at home
(pop3 here set to remove from server...)
Gianugo Rabellino wrote:
> Geoff Howard wrote:
>
>> I apologize if I've already mentioned this, but I think there are some
>> different angles on solving your problem which you may want to consider.
>>
>> One is implemented in part in the event-cache block but it could be
>> generalized to work for any component. Unico Hommes wrote a Generator
>> (in that block, but don't know its name) which uses a Delegation
>> pattern to wrap any other arbitrary Generator. It delegates all
>> interface methods to the wrapped generator except for the get Validity
>> (and currently the key as well, but I think that might need to
>> change. He then uses a factory which specifies how to construct the
>> overridden validity. The plan was to generalize this to work with any
>> component. Hope that makes sense.
>
> Interesting and worth a look, but I see a few problems:
>
> 1. Usability wise it might be uncomfortable for users to wrap every non
> cacheable sitemap component into a delegate (provided that users know
> what is cacheable and what is not);
True. I think the majority of cases could be covered by a few
pre-configured components though which may mitigate this.
> 2. There is no transformer yet (even if, agreed, it would be pretty easy
> to build one);
True, but that seemed quite a bit less work than the solutions you have
been looking at. (though I was only following the discussion casually)
> 3. AFAIU, in order to use the delegate components, you should be using
> the EventAwareCacheImpl, am I wrong? This might be a serious showstopper.
Only as it is now. I meant to convey that the same technique might have
more general use and could be refactored with that in mind.
> The whole idea of this expires-enabled stuff is that of a simple
> solution for a simple scenario where the user knows in advance for how
> long a pipeline might be valid. A sitemap parameter, and you are ready
> to go, just as httpd's mod_expires does ... I'm not sure that the
> delegate solution is understandable by a non java programmer: it
> requires quite a bit of understanding of how the event cache works and a
> good knowledge of Cocoon internals, don't you think so?
Well, with the EventAware code two factories are defined by default and
the same would make sense in the more general case. In the end, for the
most common use case (exactly what you've described) it'd be simply a
matter of changing the map:transform type=xxx and a few parameters.
Most users in your case would need the same modifications, and they
would be pretty easy to configure.
I think of expires as solely in the realm of external to cocoon caches
(browser, or proxy) which is why proposing a cocoon-cache centered
solution seemed right to me.
>> The second possibly simpler solution is to create a new pipeline
>> implementation that always caches. So, when you have a pipeline that
>> should be forced to cache, you segregate it into its own map:pipeline
>> section with a parameter that speficies how long to cache.
>
> Yes and no. We have been through this before, but again there is both a
> backward incompatibilty to consider (the current
> CachingProcessingPipeline supports Expires) and a fair amount of code
> duplication needed between the two implementations. But again, it's
> something to consider.
It supports an expires header going to the client, not a SourceValidity
object with an expire (invalid) timestamp. Code duplication could be
problem because of the different pipeline implementations already in
place if some (as I think might be the case) some orthogonal
relationships occur.
>> A third is to consider the event based cache stuff itself.
>
> Which looks really cool, and definitely it's something to work on, apart
> from the Expires stuff. How much do you still consider it alpha stuff?
Ooh. I meant to remove all my dire warnings from the source files -
working on that now. They were only meant to be there because I
comitted a first draft that I planned on refactoring but wanted the
original in cvs for collaboration with a few people who showed some
interest.
I'd still say the block should be alpha since to my knowledge no one is
using it. But I'd consider the API (which is really only a few simple
interfaces) stable. I have only two outstanding issues I'm investigating:
1) Using TransientStore to persist the state of the event-key mappings.
I have this 90% on my harddrive and will keep plunking away.
2) Investigate the feasibility of wildcard matches on events (and
whether they're even necessary). Some implementation ideas could
possibly require additions to the interfaces, but I'd want to plan those
in backwards compatible ways even though the alpha block status may not
require it.
The examples that have come up for this by the way are file system
events and date/time based events. Both of these can be handled in
other ways so they may not be necessary at all (why I've ignored them
for now). If a directory structure is used for validity (say
/opt/foo/bar/) and you want to signal that all of /opt/foo/ has been
updated you'd want to do something like /opt/foo/* but the current
implementation might not deal with this well.
Geoff
Re: forced caching of volatile data
Posted by Gianugo Rabellino <gi...@apache.org>.
Geoff Howard wrote:
> I apologize if I've already mentioned this, but I think there are some
> different angles on solving your problem which you may want to consider.
>
> One is implemented in part in the event-cache block but it could be
> generalized to work for any component. Unico Hommes wrote a Generator
> (in that block, but don't know its name) which uses a Delegation pattern
> to wrap any other arbitrary Generator. It delegates all interface
> methods to the wrapped generator except for the get Validity (and
> currently the key as well, but I think that might need to change. He
> then uses a factory which specifies how to construct the overridden
> validity. The plan was to generalize this to work with any component.
> Hope that makes sense.
Interesting and worth a look, but I see a few problems:
1. Usability wise it might be uncomfortable for users to wrap every non
cacheable sitemap component into a delegate (provided that users know
what is cacheable and what is not);
2. There is no transformer yet (even if, agreed, it would be pretty easy
to build one);
3. AFAIU, in order to use the delegate components, you should be using
the EventAwareCacheImpl, am I wrong? This might be a serious showstopper.
The whole idea of this expires-enabled stuff is that of a simple
solution for a simple scenario where the user knows in advance for how
long a pipeline might be valid. A sitemap parameter, and you are ready
to go, just as httpd's mod_expires does ... I'm not sure that the
delegate solution is understandable by a non java programmer: it
requires quite a bit of understanding of how the event cache works and a
good knowledge of Cocoon internals, don't you think so?
> The second possibly simpler solution is to create a new pipeline
> implementation that always caches. So, when you have a pipeline that
> should be forced to cache, you segregate it into its own map:pipeline
> section with a parameter that speficies how long to cache.
Yes and no. We have been through this before, but again there is both a
backward incompatibilty to consider (the current
CachingProcessingPipeline supports Expires) and a fair amount of code
duplication needed between the two implementations. But again, it's
something to consider.
> A third is to consider the event based cache stuff itself.
Which looks really cool, and definitely it's something to work on, apart
from the Expires stuff. How much do you still consider it alpha stuff?
Ciao,
--
Gianugo Rabellino
Pro-netics s.r.l. - http://www.pro-netics.com
Orixo, the XML business alliance - http://www.orixo.com
(Now blogging at: http://blogs.cocoondev.org/gianugo/)
Re: forced caching of volatile data
Posted by Geoff Howard <co...@leverageweb.com>.
I apologize if I've already mentioned this, but I think there are some
different angles on solving your problem which you may want to consider.
One is implemented in part in the event-cache block but it could be
generalized to work for any component. Unico Hommes wrote a Generator
(in that block, but don't know its name) which uses a Delegation pattern
to wrap any other arbitrary Generator. It delegates all interface
methods to the wrapped generator except for the get Validity (and
currently the key as well, but I think that might need to change. He
then uses a factory which specifies how to construct the overridden
validity. The plan was to generalize this to work with any component.
Hope that makes sense.
The second possibly simpler solution is to create a new pipeline
implementation that always caches. So, when you have a pipeline that
should be forced to cache, you segregate it into its own map:pipeline
section with a parameter that speficies how long to cache.
A third is to consider the event based cache stuff itself.
Geoff
Gianugo Rabellino wrote:
>
> Sorry for the late reply. Vacation time + lousy GPRS + heavy spam/worm
> activities nearly killed my ability to read email.
>
> Miles Elam wrote:
>
>> I'm running into a snag and was wondering if anyone had some wisdom to
>> grant to me. The current behavior of caching pipelines is to
>> aggregate the keys of all cacheable pipeline components and use them
>> as the cache hash lookup. In the case of a pipeline that has
>> uncacheable components but has an expiry, this scheme doesn't work:
>> the key is incomplete or nonexistent. My first thought was to use the
>> request URI but therein lies the snag; Any actions or selectors in
>> use that would fundamentally alter the request would be erroneously
>> cached. In 90% of the cases, I don't see this as a problem. But in
>> cases like when different formats are sent to different clients for
>> the same URI
>
>
> Yes, this was exactly my showstopper when I tried to quickly force
> caching uncacheable stuff. I really need to catch up on caching
> internals as it's been a few months since I looked at it, but I was
> kinda of planning to add a fake cache key for each component in the
> pipeline which is uncacheable. The problem is coming up with a syntax
> that is reversible, so that the resulting key can be successfully looked
> up.
>
> The trick should lie somewhere in
> o.a.c.components.pipeline.impl.AbstractCachingProcessingPipeline#generateCachingKey():
> there should be, in particular, some kind of surrogate that returns a
> key for each non cacheable component. I'm still uncertain whether this
> key should be unique (based, i.e., on the positional parameter of the
> component in the sitemap) or if a simple label could work. I'm for the
> latter ATM: of course I need to check it out, but the following snippet
> (extended to transformers & serializer) might work.
>
> // is the generator cacheable?
> if (super.generator instanceof CacheableProcessingComponent) {
> key = ((CacheableProcessingComponent)super.generator).getKey();
> this.generatorIsCacheableProcessingComponent = true;
> } else if (super.generator instanceof Cacheable) {
> key = new Long(((Cacheable)super.generator).generateKey());
> } else if (expires != 0) {
> // Component is not cacheable, but an explicit expires
> // was set. Provide a fake cache key.
> key = new String("Forced-Expires");
> }
>
> Doing so, we would end up with a PipelineCacheKey that's made up of
> different caching keys, some of which are real and some of which are
> just placeholders. I'm going to try this path and see what comes out of it.
>
> Ciao,
>
Re: forced caching of volatile data
Posted by Gianugo Rabellino <gi...@apache.org>.
Sorry for the late reply. Vacation time + lousy GPRS + heavy spam/worm
activities nearly killed my ability to read email.
Miles Elam wrote:
> I'm running into a snag and was wondering if anyone had some wisdom to
> grant to me. The current behavior of caching pipelines is to aggregate
> the keys of all cacheable pipeline components and use them as the cache
> hash lookup. In the case of a pipeline that has uncacheable components
> but has an expiry, this scheme doesn't work: the key is incomplete or
> nonexistent. My first thought was to use the request URI but therein
> lies the snag; Any actions or selectors in use that would fundamentally
> alter the request would be erroneously cached. In 90% of the cases, I
> don't see this as a problem. But in cases like when different formats
> are sent to different clients for the same URI
Yes, this was exactly my showstopper when I tried to quickly force
caching uncacheable stuff. I really need to catch up on caching
internals as it's been a few months since I looked at it, but I was
kinda of planning to add a fake cache key for each component in the
pipeline which is uncacheable. The problem is coming up with a syntax
that is reversible, so that the resulting key can be successfully looked up.
The trick should lie somewhere in
o.a.c.components.pipeline.impl.AbstractCachingProcessingPipeline#generateCachingKey():
there should be, in particular, some kind of surrogate that returns a
key for each non cacheable component. I'm still uncertain whether this
key should be unique (based, i.e., on the positional parameter of the
component in the sitemap) or if a simple label could work. I'm for the
latter ATM: of course I need to check it out, but the following snippet
(extended to transformers & serializer) might work.
// is the generator cacheable?
if (super.generator instanceof CacheableProcessingComponent) {
key = ((CacheableProcessingComponent)super.generator).getKey();
this.generatorIsCacheableProcessingComponent = true;
} else if (super.generator instanceof Cacheable) {
key = new Long(((Cacheable)super.generator).generateKey());
} else if (expires != 0) {
// Component is not cacheable, but an explicit expires
// was set. Provide a fake cache key.
key = new String("Forced-Expires");
}
Doing so, we would end up with a PipelineCacheKey that's made up of
different caching keys, some of which are real and some of which are
just placeholders. I'm going to try this path and see what comes out of it.
Ciao,
--
Gianugo Rabellino
Pro-netics s.r.l. - http://www.pro-netics.com
Orixo, the XML business alliance - http://www.orixo.com
(Now blogging at: http://blogs.cocoondev.org/gianugo/)
Re: forced caching of volatile data
Posted by Miles Elam <mi...@pcextremist.com>.
Gianugo Rabellino wrote:
> Miles Elam wrote:
>
>> It would be possible to add in some code that if the resource has no
>> validity objects and the pipeline has an expiry, it creates a dummy
>> validity object that, if asked, always returns "invalid" but has the
>> expires timestamp set to maintain the entry.
>
> This was the only missing piece yet, and I wanted to tackle it when
> I'll be back from vacation. I like your approach, so if you feel like
> coding it, please go ahead. :-)
I'm running into a snag and was wondering if anyone had some wisdom to
grant to me. The current behavior of caching pipelines is to aggregate
the keys of all cacheable pipeline components and use them as the cache
hash lookup. In the case of a pipeline that has uncacheable components
but has an expiry, this scheme doesn't work: the key is incomplete or
nonexistent. My first thought was to use the request URI but therein
lies the snag; Any actions or selectors in use that would fundamentally
alter the request would be erroneously cached. In 90% of the cases, I
don't see this as a problem. But in cases like when different formats
are sent to different clients for the same URI
(eg. XML with XSLT processing instruction for newest browsers and HTML
to older clients)
<map:pipeline type="caching">
<map:parameter name="expires" value="access plus 10 minutes"/>
<map:match pattern="">
<map:generate src="index.xml"/>
<map:transform type="hypothetical_uncacheable"/>
<map:select type="browser">
<map:when test="ie">
<map:transform src="proc_inst.xslt">
<map:parameter name="stylesheet" value="index2html.xslt"/>
</map:transform>
<map:serialize type="xml"/>
</map:when>
<map:otherwise>
<map:transform src="index2html.xslt"/>
<map:serialize type="html"/>
</map:otherwise>
</map:select>
</map:match>
</map:pipeline>
if the cache key is simply the URI, older clients may end up with raw
and, in their case, useless XML.
This can be avoided of course by some minor organization of the pipelines:
<map:pipeline type="caching">
<map:match pattern="">
<map:select type="browser">
<map:when test="ie">
<map:generate src="cocoon:/index.xml"/>
<map:serialize type="xml"/>
</map:when>
<map:otherwise>
<map:generate src="cocoon:/index.html"/>
<map:serialize type="html"/>
</map:otherwise>
</map:select>
</map:match>
</map:pipeline>
<map:pipeline type="caching">
<map:parameter name="expires" value="access plus 10 minutes"/>
<map:match pattern="index.xml">
<map:generate src="index.xml"/>
<map:transform type="hypothetical_uncacheable"/>
<map:transform src="proc_inst.xslt">
<map:parameter name="stylesheet" value="index2html.xslt"/>
</map:transform>
<map:serialize type="html"/>
</map:match>
<map:match pattern="index.html">
<map:generate src="index.xml"/>
<map:transform type="hypothetical_uncacheable"/>
<map:transform src="index2html.xslt"/>
<map:serialize type="html"/>
</map:match>
</map:pipeline>
This is actually pretty close to how my site organizes things. Now it
seems that URIs as cache keys would work, but I can easily see where
quite a few support emails for help would come from on the user list.
I think it should be done because quite a few things are not cacheable
and also not up-to-the-second necessary. A view of an online discussion
need not be immediate (and is commonly not immediate), but a database
lookup by every reader of that discussion would be formidable. Having a
centralized expiry would allow folks to avoid putting in what amounts to
futile caching into each component (eg. a database transformer where it
is not clear whether the cache interval should be time-based or
info-based). An administrator can simply say, "This updates every hour
so that my PII-300 server doesn't fall over if I get Slashdotted."
------
So my question is this: Should I
a) use full URIs (in which case somehow the URI needs to be made
available to the pipeline code...which does not seem to be the case
currently)
b) use some other mechanism which currently eludes me
Either way, any ideas how to implement? The key is my only problem. A
cached response is the easy part.
- Miles Elam
Re: forced caching of volatile data
Posted by Gianugo Rabellino <gi...@apache.org>.
Miles Elam wrote:
>> Are you certain this works with _uncacheable_ pipelines? The pipeline
>> implementation I think only caches a pipeline up to the last cacheable
>> component whether you have set this expires attribute or not. That is
>> there I think just for http header in the response for proxy
>> friendlyness. If I'm right, since the SQLTransformer does not
>> implement Cacheable (or whatever) the pipeline will never cache it.
>
> I just made a quick glance at the source for AbstractProcessingPipeline,
> and you're right; It sets an Expires header, but does not cache the
> result. It would be possible to add in some code that if the resource
> has no validity objects and the pipeline has an expiry, it creates a
> dummy validity object that, if asked, always returns "invalid" but has
> the expires timestamp set to maintain the entry.
>
> Sound reasonable?
This was the only missing piece yet, and I wanted to tackle it when I'll
be back from vacation. I like your approach, so if you feel like coding
it, please go ahead. :-)
Ciao,
--
Gianugo Rabellino
Pro-netics s.r.l. - http://www.pro-netics.com
Orixo, the XML business alliance - http://www.orixo.com
(Now blogging at: http://blogs.cocoondev.org/gianugo/)
Re: forced caching of volatile data
Posted by Miles Elam <mi...@pcextremist.com>.
Geoff Howard wrote:
> Miles Elam wrote:
>
>> Upayavira is correct; The functionality is there. If it doesn't
>> work, either my syntax was faulty or the build is broken.
>>
> Are you certain this works with _uncacheable_ pipelines? The pipeline
> implementation I think only caches a pipeline up to the last cacheable
> component whether you have set this expires attribute or not. That is
> there I think just for http header in the response for proxy
> friendlyness. If I'm right, since the SQLTransformer does not
> implement Cacheable (or whatever) the pipeline will never cache it.
I just made a quick glance at the source for AbstractProcessingPipeline,
and you're right; It sets an Expires header, but does not cache the
result. It would be possible to add in some code that if the resource
has no validity objects and the pipeline has an expiry, it creates a
dummy validity object that, if asked, always returns "invalid" but has
the expires timestamp set to maintain the entry.
Sound reasonable?
- Miles Elam
Re: forced caching of volatile data
Posted by Miles Elam <mi...@pcextremist.com>.
Geoff Howard wrote:
> Miles Elam wrote:
>
>> Upayavira is correct; The functionality is there. If it doesn't
>> work, either my syntax was faulty or the build is broken.
>>
> Are you certain this works with _uncacheable_ pipelines? The pipeline
> implementation I think only caches a pipeline up to the last cacheable
> component whether you have set this expires attribute or not. That is
> there I think just for http header in the response for proxy
> friendlyness. If I'm right, since the SQLTransformer does not
> implement Cacheable (or whatever) the pipeline will never cache it.
I just made a quick glance at the source for AbstractProcessingPipeline,
and you're right; It sets an Expires header, but does not cache the
result. It would be possible to add in some code that if the resource
has no validity objects and the pipeline has an expiry, it creates a
dummy validity object that, if asked, always returns "invalid" but has
the expires timestamp set to maintain the entry.
Sound reasonable?
- Miles Elam
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: forced caching of volatile data
Posted by Geoff Howard <co...@leverageweb.com>.
Miles Elam wrote:
> I'm doing this from memory so bear with me. I'm pretty sure this is the
> syntax in the sitemap pipelines section.
>
>
> <map:pipeline type="caching">
> <map:parameter name="expires" value="access plus 1 minutes"/>
>
> <!-- Add your URI matchers below that you want to expire in the
> future -->
> </map:pipeline>
>
>
> Upayavira is correct; The functionality is there. If it doesn't work,
> either my syntax was faulty or the build is broken.
>
> - Miles Elam
>
>
>
> Upayavira wrote:
>
>> On 10 Aug 2003 at 22:26, Geoff Howard wrote:
>>
>>
>>> rufio wrote:
>>>
>>>
>>>> Hi all
>>>> Is there a way to enfoce an asumption that volatile data (eg. stored
>>>> in DB) changes once a specific interval?
>>>>
>>>
>>> To my knowledge there is no way to do this with the current
>>> SQLTransformer. You could extend the SQLTransformer to add caching
>>> support which would be quite trivial to do if you are really content
>>> to use such a simple algorithm as cache always for one minute.
>>>
>>
>> In Cocoon 2.1 isn't there something to do with 'expiry' in the cache?
>> I remember some discussion about making it possible to set a pipeline
>> to 'expire' after x seconds, regardless of its content. Wouldn't this
>> sort this problem?
>>
Are you certain this works with _uncacheable_ pipelines? The pipeline
implementation I think only caches a pipeline up to the last cacheable
component whether you have set this expires attribute or not. That is
there I think just for http header in the response for proxy
friendlyness. If I'm right, since the SQLTransformer does not implement
Cacheable (or whatever) the pipeline will never cache it.
Geoff
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: forced caching of volatile data
Posted by Miles Elam <mi...@pcextremist.com>.
I'm doing this from memory so bear with me. I'm pretty sure this is the
syntax in the sitemap pipelines section.
<map:pipeline type="caching">
<map:parameter name="expires" value="access plus 1 minutes"/>
<!-- Add your URI matchers below that you want to expire in the
future -->
</map:pipeline>
Upayavira is correct; The functionality is there. If it doesn't work,
either my syntax was faulty or the build is broken.
- Miles Elam
Upayavira wrote:
>On 10 Aug 2003 at 22:26, Geoff Howard wrote:
>
>
>>rufio wrote:
>>
>>
>>>Hi all
>>>Is there a way to enfoce an asumption that volatile data (eg. stored
>>>in DB) changes once a specific interval?
>>>
>>>
>>To my knowledge there is no way to do this with the current
>>SQLTransformer. You could extend the SQLTransformer to add caching
>>support which would be quite trivial to do if you are really content
>>to use such a simple algorithm as cache always for one minute.
>>
>>
>In Cocoon 2.1 isn't there something to do with 'expiry' in the cache? I remember
>some discussion about making it possible to set a pipeline to 'expire' after x seconds,
>regardless of its content. Wouldn't this sort this problem?
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: forced caching of volatile data
Posted by Upayavira <uv...@upaya.co.uk>.
On 10 Aug 2003 at 22:26, Geoff Howard wrote:
> rufio wrote:
> > Hi all
> > Is there a way to enfoce an asumption that volatile data (eg. stored
> > in DB) changes once a specific interval?
> >
> > Now if we have 100 requests/s SQLTransformer querys DB 100 times/s,
> > right? I'd like to set it up to query only once a, say minute and
> > cache the result.
> >
>
> To my knowledge there is no way to do this with the current
> SQLTransformer. You could extend the SQLTransformer to add caching
> support which would be quite trivial to do if you are really content
> to use such a simple algorithm as cache always for one minute.
In Cocoon 2.1 isn't there something to do with 'expiry' in the cache? I remember
some discussion about making it possible to set a pipeline to 'expire' after x seconds,
regardless of its content. Wouldn't this sort this problem?
Hope that helps, if a bit vague.
Regards, Upayavira
> Geoff
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org For
> additional commands, e-mail: users-help@cocoon.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org
Re: forced caching of volatile data
Posted by Geoff Howard <co...@leverageweb.com>.
rufio wrote:
> Hi all
> Is there a way to enfoce an asumption that volatile data (eg. stored in
> DB) changes once a specific interval?
>
> Now if we have 100 requests/s SQLTransformer querys DB 100 times/s,
> right?
> I'd like to set it up to query only once a, say minute and cache the
> result.
>
To my knowledge there is no way to do this with the current
SQLTransformer. You could extend the SQLTransformer to add caching
support which would be quite trivial to do if you are really content to
use such a simple algorithm as cache always for one minute.
Geoff
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@cocoon.apache.org
For additional commands, e-mail: users-help@cocoon.apache.org