You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Fredrik Widlund <fr...@qbrick.com> on 2007/01/19 10:23:12 UTC
mod_cache+mod_rewrite behaviour
I'm trying to get mod_cache to ignore the query_string part of the
request, since our customers use "clicktags" in references to static
banners. I need to cache these request to improve performance.
My idea was to "RewriteRule .* %{REQUEST_URI}?", however I have learned
that mod_cache is run as a "quick_handler". However the actual
cache_create_entity() call run _after_ the url being rewritten,
resulting in the behaviour below (some debug calls have been added to
mod_cache). One can see that create_select use the unrewritten request
"http://1.2.3.4:80/index.html?ref=x", and create_select the rewritten
"http://1.2.3.4:80/index.html?". This clearly makes mod_cache and
mod_rewrite incompatible.
[Thu Jan 18 18:40:06 2007] [debug] cache_storage.c(440): key generated:
http://1.2.3.4:80/index.html?ref=x
[Thu Jan 18 18:40:06 2007] [debug] cache_storage.c(198): cache_select
key: http://1.2.3.4:80/index.html?ref=x
[Thu Jan 18 18:40:06 2007] [debug] mod_cache.c(129): Adding CACHE_SAVE
filter for /index.html
[Thu Jan 18 18:40:06 2007] [debug] mod_cache.c(136): Adding
CACHE_REMOVE_URL filter for /index.html
[Thu Jan 18 18:40:06 2007] [debug] cache_storage.c(440): key generated:
http://1.2.3.4:80/index.html?
[Thu Jan 18 18:40:06 2007] [debug] cache_storage.c(80):
cache_create_entity key: http://1.2.3.4:80/index.html?
[Thu Jan 18 18:40:06 2007] [debug] mod_cache.c(609): cache: Caching url:
/index.html?ref=x
[Thu Jan 18 18:40:06 2007] [debug] mod_cache.c(615): cache: Removing
CACHE_REMOVE_URL filter.
[Thu Jan 18 18:40:06 2007] [debug] mod_cache.c(658): cache: Added date
header
[Thu Jan 18 18:40:06 2007] [debug] mod_disk_cache.c(954): disk_cache:
Stored headers for URL http://1.2.3.4:80/index.html?
[Thu Jan 18 18:40:06 2007] [debug] mod_disk_cache.c(1043): disk_cache:
Body for URL http://1.2.3.4:80/index.html? cached.
I understand this has been discussed earlier in the " mod_cache, don't
always run as a quick handler." discussion, and I'd like to second this
request. And/or I'd be very happy to add a "CacheIgnoreQueryString On"
option to mod_cache.
Kind regards,
Fredrik Widlund
Re: mod_cache+mod_rewrite behaviour
Posted by Bart van der Schans <sc...@hippo.nl>.
Akins, Brian wrote:
> In our home-grown cache module, the "rules" are actually provider based. We
> have providers that provide matches based on exact match, string match,
> regex, prefix, environment variable, and other assorted things.
>
> Something like:
>
> CacheEnable disk regex=\.gif$ ignore_query
> CacheEnable disk prefix=/somethingelse foo=bar
> Cacheenable mem exact=/strange/stuff.asp
> CacheDisable regex=\.php$
>
> Default "match provider" could be prefix to allow users to still do:
>
> CacheEnable disk /cache_me
That would be very useful in our setups (as reverse caching proxy).
Although it would be nice if the "normal" options would set the default
and the extra options or rules (maybe comma seperated?) would override
them. Maybe something like:
CacheIgnoreNoLastMod Off
CacheEnable disk regex=\.gif$ IgnoreQuery=on,IgnoreNoLastMod=on
Regards,
Bart
Re: mod_cache+mod_rewrite behaviour
Posted by "Akins, Brian" <Br...@turner.com>.
On 1/23/07 10:44 AM, "Niklas Edmundsson" <ni...@acc.umu.se> wrote:
> Ah, you can place CacheEnable-directives in the vhost context too.
> Then it should be sufficient, unless you want to say "ignore
> querystring for all .gif:s" or something like that. Perhaps use a
> regex instead?
In our home-grown cache module, the "rules" are actually provider based. We
have providers that provide matches based on exact match, string match,
regex, prefix, environment variable, and other assorted things.
Something like:
CacheEnable disk regex=\.gif$ ignore_query
CacheEnable disk prefix=/somethingelse foo=bar
Cacheenable mem exact=/strange/stuff.asp
CacheDisable regex=\.php$
Default "match provider" could be prefix to allow users to still do:
CacheEnable disk /cache_me
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: mod_cache+mod_rewrite behaviour
Posted by Niklas Edmundsson <ni...@acc.umu.se>.
On Tue, 23 Jan 2007, Brian Akins wrote:
> Niklas Edmundsson wrote:
>> Since mod_cache runs as a quick handler, matching based on URL would
>> probably be the easiest since you don't have the mime type info then.
>
> Maybe something like
>
> CacheEnable disk /special/path ignore_query
>
> Could add other options if future as well.
Ah, you can place CacheEnable-directives in the vhost context too.
Then it should be sufficient, unless you want to say "ignore
querystring for all .gif:s" or something like that. Perhaps use a
regex instead?
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se
---------------------------------------------------------------------------
StarFleet Express, accepted at more places than MasterCard.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: mod_cache+mod_rewrite behaviour
Posted by Brian Akins <br...@turner.com>.
Niklas Edmundsson wrote:
> Since mod_cache runs as a quick handler, matching based on URL would
> probably be the easiest since you don't have the mime type info then.
Maybe something like
CacheEnable disk /special/path ignore_query
Could add other options if future as well.
--
Brian Akins
Chief Operations Engineer
Turner Digital Media Technologies
Re: mod_cache+mod_rewrite behaviour
Posted by Niklas Edmundsson <ni...@acc.umu.se>.
On Fri, 19 Jan 2007, Bart van der Schans wrote:
> Fredrik Widlund wrote:
>>
>> I understand this has been discussed earlier in the " mod_cache, don't
>> always run as a quick handler." discussion, and I'd like to second this
>> request. And/or I'd be very happy to add a "CacheIgnoreQueryString On"
>> option to mod_cache.
> I also would like to have this option. It will break RFC 2616/13.9 if
> you turn it on though. If there's more support for this I could submit a
> patch for it...
Looking a bit forward, I would suggest some form of "selection
criteria" for ignoring the query string. The scenario of wanting to
use the query string for everything but a certain mimetype or path/URL
doesn't seem too unlikely.
Since mod_cache runs as a quick handler, matching based on URL would
probably be the easiest since you don't have the mime type info then.
/Nikke
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Niklas Edmundsson, Admin @ {acc,hpc2n}.umu.se | nikke@acc.umu.se
---------------------------------------------------------------------------
C programmers don't have the BASIC instincts.
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Re: mod_cache+mod_rewrite behaviour
Posted by Bart van der Schans <sc...@hippo.nl>.
Fredrik Widlund wrote:
>
> I understand this has been discussed earlier in the " mod_cache, don't
> always run as a quick handler." discussion, and I'd like to second this
> request. And/or I'd be very happy to add a "CacheIgnoreQueryString On"
> option to mod_cache.
I also would like to have this option. It will break RFC 2616/13.9 if
you turn it on though. If there's more support for this I could submit a
patch for it...
Regards,
Bart
--
Hippo
Oosteinde 11
1017WT Amsterdam
The Netherlands
Tel +31 (0)20 5224466
-------------------------------------------------------------
schans@hippo.nl / http://www.hippo.nl
--------------------------------------------------------------
Re: mod_cache+mod_rewrite behaviour
Posted by Fredrik Widlund <fr...@qbrick.com>.
I wrote a patch friday, had a few minutes over to test it and it seemed
to work fine. Of course it's a minimal patch, but I'll double check it
tomorrow anyway, and try to convince you guys to accept it. :)
Kind regards,
Fredrik Widlund
Ruediger Pluem skrev:
> On 01/19/2007 05:49 PM, Scott MacVicar wrote:
>
>
>> I already have a patch that we're using in production, attached are
>> patches for the 2.2 and trunk.
>>
>> I was under the impression that there wasn't any interest in a patch
>> like this hence I've not opened an issue to get this integrated into the
>> main package.
>>
>> The option is CacheStoreQueryString.
>>
>
> The option you introduce does not really solve Fredrik's problem.
> Your patch allows responses to requests with a query string to be cached
> if the response does not contain an Expires header.
> As far as I understood him the responses get cached (they seem to contain
> an Expires header), but he wants to avoid that the query string gets
> incorporated in the cache key. E.g.
>
> /abc.html?x=a
> /abc.html?x=b
>
> result in two different cache entities. He wants them to be the same, or more
> generally he wants all adjustments that have been made to the URI / query string
> during the processing to be considered for the generation of the cache key.
> Currently the original URI / query string is used for this.
>
> BTW: Although it is not exactly the same your problem can be solved without
> the patch by either:
>
> - Making the content handler (whether in httpd or somewhere else) set an Expires
> header in the response.
> - If this is not possible let mod_expires set one.
>
> Regards
>
> Rüdiger
>
Re: mod_cache+mod_rewrite behaviour
Posted by Ruediger Pluem <rp...@apache.org>.
On 01/21/2007 05:55 PM, Scott MacVicar wrote:
> Ruediger Pluem wrote:
>
>>
>> On 01/21/2007 04:09 PM, Scott MacVicar wrote:
>>
>>> We did use mod_expires but the Expires header was being passed on to the
>>> client, mod_headers didn't appear to be able to unset this during tests.
>>
>>
>> This is true, but what is the problem with passing the Expires header
>> to the
>> client? If you want to prevent the clients from requesting /
>> revalidating the resource
>> frequently you can simply set an expiration date far in the future
>> (about a year).
>>
> True this would work but requires configuration on each of our web
> servers, I'd also need to find an equivalent for our non-Apache
> machines. I guess I prefer having a centralised point in which to
> control the caching behaviour.
Sorry for being confused here. You would use mod_expire on the same box / httpd
you use for caching. So this setting would be as centralized as your caching
configuration. I admit that it is a pitty that you cannot make
ExpiresDefault
ExpiresByType
dependant on an environment variable as you can do with mod_headers.
Regards
Rüdiger
Re: mod_cache+mod_rewrite behaviour
Posted by Scott MacVicar <sc...@ntlworld.com>.
Ruediger Pluem wrote:
>
> On 01/21/2007 04:09 PM, Scott MacVicar wrote:
>
>> We did use mod_expires but the Expires header was being passed on to the
>> client, mod_headers didn't appear to be able to unset this during tests.
>
> This is true, but what is the problem with passing the Expires header to the
> client? If you want to prevent the clients from requesting / revalidating the resource
> frequently you can simply set an expiration date far in the future (about a year).
>
True this would work but requires configuration on each of our web
servers, I'd also need to find an equivalent for our non-Apache
machines. I guess I prefer having a centralised point in which to
control the caching behaviour.
>> The reason for our desire to cache is that a version number is used in
>> the query string and incremented when appropriate on the resource.
>
> So from my limited understanding of your environment it seems to make sense to set
> a long expiration time on this resource as a new version of a response will have
> a different query string.
>
>> Most browsers seem to ignore RFC2616 13.9 in regards to the query string
>> being present in the URI.
>
> What do you mean by this? That browsers cache the response even if no Expires
> header is present in the response?
>
Internet Explorer and Firefox both seem to cache content that have a
query string when there is no expires header, according to the RFC which
Apache is following there should be no caching.
>> Apologies for not getting the gist of the thread.
>
> No need to apologize at all for contributing to this discussion.
>
Reagards,
Scott
Re: mod_cache+mod_rewrite behaviour
Posted by Ruediger Pluem <rp...@apache.org>.
On 01/21/2007 04:09 PM, Scott MacVicar wrote:
>
> We did use mod_expires but the Expires header was being passed on to the
> client, mod_headers didn't appear to be able to unset this during tests.
This is true, but what is the problem with passing the Expires header to the
client? If you want to prevent the clients from requesting / revalidating the resource
frequently you can simply set an expiration date far in the future (about a year).
> The reason for our desire to cache is that a version number is used in
> the query string and incremented when appropriate on the resource.
So from my limited understanding of your environment it seems to make sense to set
a long expiration time on this resource as a new version of a response will have
a different query string.
>
> Most browsers seem to ignore RFC2616 13.9 in regards to the query string
> being present in the URI.
What do you mean by this? That browsers cache the response even if no Expires
header is present in the response?
>
> Apologies for not getting the gist of the thread.
No need to apologize at all for contributing to this discussion.
Regards
Rüdiger
Re: mod_cache+mod_rewrite behaviour
Posted by Scott MacVicar <sc...@ntlworld.com>.
Ruediger Pluem wrote:
>
> On 01/19/2007 05:49 PM, Scott MacVicar wrote:
>
>> I already have a patch that we're using in production, attached are
>> patches for the 2.2 and trunk.
>>
>> I was under the impression that there wasn't any interest in a patch
>> like this hence I've not opened an issue to get this integrated into the
>> main package.
>>
>> The option is CacheStoreQueryString.
>
> The option you introduce does not really solve Fredrik's problem.
> Your patch allows responses to requests with a query string to be cached
> if the response does not contain an Expires header.
> As far as I understood him the responses get cached (they seem to contain
> an Expires header), but he wants to avoid that the query string gets
> incorporated in the cache key. E.g.
>
> /abc.html?x=a
> /abc.html?x=b
>
> result in two different cache entities. He wants them to be the same, or more
> generally he wants all adjustments that have been made to the URI / query string
> during the processing to be considered for the generation of the cache key.
> Currently the original URI / query string is used for this.
>
> BTW: Although it is not exactly the same your problem can be solved without
> the patch by either:
>
> - Making the content handler (whether in httpd or somewhere else) set an Expires
> header in the response.
> - If this is not possible let mod_expires set one.
>
> Regards
>
> Rüdiger
>
We did use mod_expires but the Expires header was being passed on to the
client, mod_headers didn't appear to be able to unset this during tests.
The reason for our desire to cache is that a version number is used in
the query string and incremented when appropriate on the resource.
Most browsers seem to ignore RFC2616 13.9 in regards to the query string
being present in the URI.
Apologies for not getting the gist of the thread.
Scott
Re: mod_cache+mod_rewrite behaviour
Posted by Ruediger Pluem <rp...@apache.org>.
On 01/19/2007 05:49 PM, Scott MacVicar wrote:
>
> I already have a patch that we're using in production, attached are
> patches for the 2.2 and trunk.
>
> I was under the impression that there wasn't any interest in a patch
> like this hence I've not opened an issue to get this integrated into the
> main package.
>
> The option is CacheStoreQueryString.
The option you introduce does not really solve Fredrik's problem.
Your patch allows responses to requests with a query string to be cached
if the response does not contain an Expires header.
As far as I understood him the responses get cached (they seem to contain
an Expires header), but he wants to avoid that the query string gets
incorporated in the cache key. E.g.
/abc.html?x=a
/abc.html?x=b
result in two different cache entities. He wants them to be the same, or more
generally he wants all adjustments that have been made to the URI / query string
during the processing to be considered for the generation of the cache key.
Currently the original URI / query string is used for this.
BTW: Although it is not exactly the same your problem can be solved without
the patch by either:
- Making the content handler (whether in httpd or somewhere else) set an Expires
header in the response.
- If this is not possible let mod_expires set one.
Regards
Rüdiger
Re: mod_cache+mod_rewrite behaviour
Posted by Scott MacVicar <sc...@ntlworld.com>.
Fredrik Widlund wrote:
> Hi,
>
> Thanks for the information. Tried the patch and it mends it the
> behaviour, however it doesn't really help me of course since I indeed am
> trying to rewrite the url before it's cached.
>
> What are the chances of getting a patch that adds a
> "CacheIgnoreQueryString" option accepted? Who/where do I ask this?
>
> Kind regards,
> Fredrik Widlund
>
> Plüm wrote:
>>> -----Ursprüngliche Nachricht-----
>>> Von: Fredrik Widlund
>>> Gesendet: Freitag, 19. Januar 2007 10:23
>>> An: dev@httpd.apache.org
>>> Betreff: mod_cache+mod_rewrite behaviour
>>>
>>>
>>> I'm trying to get mod_cache to ignore the query_string part of the
>>> request, since our customers use "clicktags" in references to static
>>> banners. I need to cache these request to improve performance.
>>>
>>> My idea was to "RewriteRule .* %{REQUEST_URI}?", however I
>>> have learned
>>> that mod_cache is run as a "quick_handler". However the actual
>>> cache_create_entity() call run _after_ the url being rewritten,
>>> resulting in the behaviour below (some debug calls have been added to
>>> mod_cache). One can see that create_select use the unrewritten request
>>> "http://1.2.3.4:80/index.html?ref=x", and create_select the rewritten
>>> "http://1.2.3.4:80/index.html?". This clearly makes mod_cache and
>>> mod_rewrite incompatible.
>>>
>> This is a known issue and fixed on trunk and proposed for backport to 2.2.x.
>> Please have a look at PR40805 (http://issues.apache.org/bugzilla/show_bug.cgi?id=40805)
>> You will find references to a patch there.
>> The fix is to use the *unmodified* URL / query string consistently.
>>
>> Regards
>>
>> Rüdiger
>>
>>
>
>
I already have a patch that we're using in production, attached are
patches for the 2.2 and trunk.
I was under the impression that there wasn't any interest in a patch
like this hence I've not opened an issue to get this integrated into the
main package.
The option is CacheStoreQueryString.
Regards,
Scott
Re: mod_cache+mod_rewrite behaviour
Posted by Fredrik Widlund <fr...@qbrick.com>.
Hi,
This is regarding my
http://issues.apache.org/bugzilla/show_bug.cgi?id=41484 suggested
enhancement. I am attaching the same patch in this mail.
I have a need to use caching to improve performance when delivering for
example banners. Since query-string parameters are used to associate
unique meta-information to each request mod_cache will treat each
request as a new request. Since mod_cache is run as a quick handler it
is not possible to use mod_rewrite to remove the query-string part of
the request.
I believe this should be considered a relevant scenario, and that an
option to disable this behaviour is motivated.
I don't see any drawbacks as a result of this option.
This is very much a real-life scenario, I am a doing this on behalf of a
leading European CDN and we need to use this in a highly distributed
environment to be able to scale efficiently.
PPP regards,
Fredrik Widlund
Plüm skrev:
>
>> -----Ursprüngliche Nachricht-----
>> Von: Fredrik Widlund
>> Gesendet: Freitag, 19. Januar 2007 12:30
>> An: dev@httpd.apache.org
>> Betreff: Re: mod_cache+mod_rewrite behaviour
>>
>>
>> Hi,
>>
>> Thanks for the information. Tried the patch and it mends it the
>> behaviour, however it doesn't really help me of course since
>> I indeed am
>> trying to rewrite the url before it's cached.
>>
>> What are the chances of getting a patch that adds a
>> "CacheIgnoreQueryString" option accepted? Who/where do I ask this?
>>
>
> This is the right place for discussion. I would propose the following:
>
> 1. Create a bug report describing your problem and mark it as enhancement.
> 2. If you have a patch for CacheIgnoreQueryString attach it to the report.
> 3. Come back here with your problem (in this thread), refer to the report
> and attach the patch for convenience.
> 4. Give some arguments why this is not only useful for you but for everyone else.
> And if there are any drawbacks as a result of your patch why it is worth
> the tradeoff.
> 5. Be PPP (Patient, Polite, Persistent) :-). Keep on buging us from time to
> time if the reaction to your proposal is only inactivity and not decline.
>
> Regards
>
> Rüdiger
>
>
Re: mod_cache+mod_rewrite behaviour
Posted by Plüm,
Rüdiger,
VF EITO <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: Fredrik Widlund
> Gesendet: Freitag, 19. Januar 2007 12:30
> An: dev@httpd.apache.org
> Betreff: Re: mod_cache+mod_rewrite behaviour
>
>
> Hi,
>
> Thanks for the information. Tried the patch and it mends it the
> behaviour, however it doesn't really help me of course since
> I indeed am
> trying to rewrite the url before it's cached.
>
> What are the chances of getting a patch that adds a
> "CacheIgnoreQueryString" option accepted? Who/where do I ask this?
This is the right place for discussion. I would propose the following:
1. Create a bug report describing your problem and mark it as enhancement.
2. If you have a patch for CacheIgnoreQueryString attach it to the report.
3. Come back here with your problem (in this thread), refer to the report
and attach the patch for convenience.
4. Give some arguments why this is not only useful for you but for everyone else.
And if there are any drawbacks as a result of your patch why it is worth
the tradeoff.
5. Be PPP (Patient, Polite, Persistent) :-). Keep on buging us from time to
time if the reaction to your proposal is only inactivity and not decline.
Regards
Rüdiger
Re: mod_cache+mod_rewrite behaviour
Posted by Fredrik Widlund <fr...@qbrick.com>.
Hi,
Thanks for the information. Tried the patch and it mends it the
behaviour, however it doesn't really help me of course since I indeed am
trying to rewrite the url before it's cached.
What are the chances of getting a patch that adds a
"CacheIgnoreQueryString" option accepted? Who/where do I ask this?
Kind regards,
Fredrik Widlund
Plüm wrote:
>> -----Ursprüngliche Nachricht-----
>> Von: Fredrik Widlund
>> Gesendet: Freitag, 19. Januar 2007 10:23
>> An: dev@httpd.apache.org
>> Betreff: mod_cache+mod_rewrite behaviour
>>
>>
>> I'm trying to get mod_cache to ignore the query_string part of the
>> request, since our customers use "clicktags" in references to static
>> banners. I need to cache these request to improve performance.
>>
>> My idea was to "RewriteRule .* %{REQUEST_URI}?", however I
>> have learned
>> that mod_cache is run as a "quick_handler". However the actual
>> cache_create_entity() call run _after_ the url being rewritten,
>> resulting in the behaviour below (some debug calls have been added to
>> mod_cache). One can see that create_select use the unrewritten request
>> "http://1.2.3.4:80/index.html?ref=x", and create_select the rewritten
>> "http://1.2.3.4:80/index.html?". This clearly makes mod_cache and
>> mod_rewrite incompatible.
>>
>
> This is a known issue and fixed on trunk and proposed for backport to 2.2.x.
> Please have a look at PR40805 (http://issues.apache.org/bugzilla/show_bug.cgi?id=40805)
> You will find references to a patch there.
> The fix is to use the *unmodified* URL / query string consistently.
>
> Regards
>
> Rüdiger
>
>
Re: mod_cache+mod_rewrite behaviour
Posted by Plüm,
Rüdiger,
VF EITO <ru...@vodafone.com>.
> -----Ursprüngliche Nachricht-----
> Von: Fredrik Widlund
> Gesendet: Freitag, 19. Januar 2007 10:23
> An: dev@httpd.apache.org
> Betreff: mod_cache+mod_rewrite behaviour
>
>
> I'm trying to get mod_cache to ignore the query_string part of the
> request, since our customers use "clicktags" in references to static
> banners. I need to cache these request to improve performance.
>
> My idea was to "RewriteRule .* %{REQUEST_URI}?", however I
> have learned
> that mod_cache is run as a "quick_handler". However the actual
> cache_create_entity() call run _after_ the url being rewritten,
> resulting in the behaviour below (some debug calls have been added to
> mod_cache). One can see that create_select use the unrewritten request
> "http://1.2.3.4:80/index.html?ref=x", and create_select the rewritten
> "http://1.2.3.4:80/index.html?". This clearly makes mod_cache and
> mod_rewrite incompatible.
This is a known issue and fixed on trunk and proposed for backport to 2.2.x.
Please have a look at PR40805 (http://issues.apache.org/bugzilla/show_bug.cgi?id=40805)
You will find references to a patch there.
The fix is to use the *unmodified* URL / query string consistently.
Regards
Rüdiger