You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Simon Mott <de...@gmail.com> on 2016/06/01 15:23:14 UTC
[users@httpd] mod_cache, mod_deflate, mod_expires and a headache
Hi Guys
I asked about this on IRC but after some digging I think it might be better
suited to the mailing list.
I am looking to make use of mod_cache to improve response times on a
website. I have mod_cache and mod_disk cache working fine - Content gets
cached and recalled from the cache as needed no problem. I have added
mod_expires config so that the correct cache control/expires headers are in
place so thats working great
Once I add mod_deflate into the mix, things get interesting...
mod_cache is doing what Its configured to do by default on receipt of
gzipped data - Lets say my browser sends a request with "Accept-Encoding:
gzip, deflate, sdch" then this gets dealt with all nicely and stored in my
cache with a "Vary: accept-encoding" header.
So far so good.
Another client connects and sends a request with "Accept-Encoding: gzip,
deflate, sdch, br" and as expected, mod_cache stores a separate copy for
this because the header is different to the one in my cache. This is where
I am falling over...
I dont want to store multiple cached copies of my content so I am trying to
cache content uncompressed, and have mod_deflate gzip it before going to
the client. Documentation for mod_cache ( see
https://httpd.apache.org/docs/trunk/mod/mod_cache.html#finecontrol )
suggests this is indeed possible by using Output filters...
> # Cache content before optional compression
> CacheQuickHandler off
> AddOutputFilterByType CACHE;DEFLATE text/plain
Great... Except it doesnt do what it suggests.
Doing this results in "CACHE filter was added twice, or was added where the
cache has been bypassed and will be ignored" in debug output, and you can
see its being compressed before being stored to cache
> [cache:debug] [pid 9497] cache_storage.c(664): [client 10.0.0.20:40156]
AH00698: cache: Key for entity /index.php?(null) is
https://www.my_website.co.uk:443/index.php?
> [cache:debug] [pid 9497] mod_cache.c(486): [client 10.0.0.20:40156]
AH00757: Adding CACHE_SAVE filter for /index.php
> [cache:debug] [pid 9497] mod_cache.c(520): [client 10.0.0.20:40156]
AH00759: Adding CACHE_REMOVE_URL filter for /index.php
> [cache:debug] [pid 9497] mod_cache.c(1718): [client 10.0.0.20:40156]
AH00777: cache: CACHE filter was added twice, or was added where the cache
has been bypassed and will be ignored: /
> [deflate:debug] [pid 9497] mod_deflate.c(849): [client 10.0.0.20:40156]
AH01384: Zlib: Compressed 51365 to 12218 : URL /index.php
> [headers:debug] [pid 9497] mod_headers.c(845): AH01502: headers:
ap_headers_output_filter()
> [cache:debug] [pid 9497] mod_cache.c(1328): [client 10.0.0.20:40156]
AH00769: cache: Caching url: /
> [cache:debug] [pid 9497] mod_cache.c(1334): [client 10.0.0.20:40156]
AH00770: cache: Removing CACHE_REMOVE_URL filter.
> [cache_disk:debug] [pid 9497] mod_cache_disk.c(1349): [client
10.0.0.20:40156] AH00737: commit_entity: Headers and body for URL
https://www.my_website.co.uk:443/index.php? cached.
I do have a "Workaround" for this in that if I use "SetOutputFilter" and
then "AddOutputFilterByType", I sort of get what I want...
> SetOutputFilter CACHE
> AddOutputFilterByType DEFLATE text/html
This does result in correct behaviour in that content is cached before
being compressed and as such is only being cached once, however, its now
being done before mod_expires has chance to add correct cache
control/expiry headers so some things are being considered in-eligable for
storing in cache
> [cache:debug] [pid 9607] cache_storage.c(664): [client 10.0.0.20:40174]
AH00698: cache: Key for entity /index.php?(null) is
https://www.my_website.co.uk:443/index.php?
> [cache:debug] [pid 9607] mod_cache.c(486): [client 10.0.0.20:40174]
AH00757: Adding CACHE_SAVE filter for /index.php
> [cache:debug] [pid 9607] mod_cache.c(509): [client 10.0.0.20:40174]
AH00758: Replacing CACHE with CACHE_SAVE filter for /index.php
> [cache:debug] [pid 9607] mod_cache.c(520): [client 10.0.0.20:40174]
AH00759: Adding CACHE_REMOVE_URL filter for /index.php
> [cache:debug] [pid 9607] mod_cache.c(1328): [client 10.0.0.20:40174]
AH00769: cache: Caching url: /
> [cache:debug] [pid 9607] mod_cache.c(1334): [client 10.0.0.20:40174]
AH00770: cache: Removing CACHE_REMOVE_URL filter.
> [cache_disk:debug] [pid 9607] mod_cache_disk.c(1349): [client
10.0.0.20:40174] AH00737: commit_entity: Headers and body for URL
https://www.my_website.co.uk:443/index.php? cached.
> [deflate:debug] [pid 9607] mod_deflate.c(849): [client 10.0.0.20:40174]
AH01384: Zlib: Compressed 51365 to 12218 : URL /index.php
> [headers:debug] [pid 9607] mod_headers.c(845): AH01502: headers:
ap_headers_output_filter()
>
> [cache:debug] [pid 9609] cache_storage.c(664): [client 10.0.0.20:40175]
AH00698: cache: Key for entity /index.php?s=bob is
https://www.my_website.co.uk:443/index.php?s=bob
> [cache:debug] [pid 9609] mod_cache.c(486): [client 10.0.0.20:40175]
AH00757: Adding CACHE_SAVE filter for /index.php
> [cache:debug] [pid 9609] mod_cache.c(509): [client 10.0.0.20:40175]
AH00758: Replacing CACHE with CACHE_SAVE filter for /index.php
> [cache:debug] [pid 9609] mod_cache.c(520): [client 10.0.0.20:40175]
AH00759: Adding CACHE_REMOVE_URL filter for /index.php
> [cache:debug] [pid 9609] mod_cache.c(1210): [client 10.0.0.20:40175]
AH00768: cache: /?s=bob not cached. Reason: Query string present but no
explicit expiration time
> [deflate:debug] [pid 9609] mod_deflate.c(849): [client 10.0.0.20:40175]
AH01384: Zlib: Compressed 27993 to 6343 : URL /index.php
> [headers:debug] [pid 9609] mod_headers.c(845): AH01502: headers:
ap_headers_output_filter()
In sort, I guess what I am asking here seems to be: Is the documentation I
mentioned correct for AddOutputFilterByType CACHE;DEFLATE or is this
detailed above working as expected? If It is as expected, is there a way I
can force mod_expires to be applied before cache so that content is cached
correctly?
Slightly sanitized version of my current config:
> # Server version: Apache/2.4.7 (Ubuntu)
> # Server built: Mar 10 2015 13:05:59
>
> ServerName www.my_website.co.uk
> ServerAlias my_website.co.uk
> UseCanonicalName On
> Header always set Strict-Transport-Security "max-age=63072000;
includeSubdomains; preload"
>
> CacheQuickHandler off
> SetOutputFilter CACHE
> AddOutputFilterByType DEFLATE text/html text/plain text/css
application/javascript application/rss+xml
>
> CacheRoot /var/cache/apache2/mod_cache_disk/my_website.co.uk
> CacheEnable disk /
> CacheDirLevels 2
> CacheDirLength 1
> CacheMaxFileSize 2000000
> CacheIgnoreNoLastMod On
> CacheIgnoreCacheControl On
> CacheDefaultExpire 7200
> CacheLastModifiedFactor 0.5
> CacheIgnoreHeaders Set-Cookie
> CacheHeader on
> CacheLock on
>
> ExpiresActive on
> ExpiresDefault "access plus 1 week"
> ExpiresByType text/html "access plus 2 hours"
> FileETag All
Any help that can be offered would be greatly appreciated :)
Simon
Re: [users@httpd] mod_cache, mod_deflate, mod_expires and a headache
Posted by Eric Covener <co...@gmail.com>.
On Fri, Jun 3, 2016 at 4:04 PM, Simon Mott <de...@gmail.com> wrote:
> Still struggling with this one :)
>
> I've tried adding "MOD_EXPIRES" as a filter before "CACHE" but it doesn't
> seem to cause mod_expires to be evaluated before its cached (I don't really
> understand what AP_FTYPE_CONTENT_SET-2 means in the source for mod_expires)
> - Can what I'm trying to do be done?
Perhaps using mod_filter syntax would be more expressive and provide
some extra trace. But mod_expires should already be inserting itself
before mod_cache implicitly if it's enabled.
--
Eric Covener
covener@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_cache, mod_deflate, mod_expires and a headache
Posted by Eric Covener <co...@gmail.com>.
On Fri, Jun 3, 2016 at 4:04 PM, Simon Mott <de...@gmail.com> wrote:
> (I don't really understand what AP_FTYPE_CONTENT_SET-2 means in the source
> for mod_expires)
I am not sure the docs capture this, but you cannot really choose an
arbitrary order with SetOutputFilter, although you can with mod_filter
(native mod_filter directives and AddOutputFilterByType -- but by-type
won't work for you because you have the issue with needing the type
set before the handler)
The reason is because each filter has a predefined type (or rank)
where it wants to run and the order in SetOutputFilter doesn't
override it, but mod_filter does indirectly override it.
Here are the values:
typedef enum {
/** These filters are used to alter the content that is passed through
* them. Examples are SSI or PHP. */
AP_FTYPE_RESOURCE = 10,
/** These filters are used to alter the content as a whole, but after all
* AP_FTYPE_RESOURCE filters are executed. These filters should not
* change the content-type. An example is deflate. */
AP_FTYPE_CONTENT_SET = 20,
/** These filters are used to handle the protocol between server and
* client. Examples are HTTP and POP. */
AP_FTYPE_PROTOCOL = 30,
/** These filters implement transport encodings (e.g., chunking). */
AP_FTYPE_TRANSCODE = 40,
/** These filters will alter the content, but in ways that are
* more strongly associated with the connection. Examples are
* splitting an HTTP connection into multiple requests and
* buffering HTTP responses across multiple requests.
*
* It is important to note that these types of filters are not
* allowed in a sub-request. A sub-request's output can certainly
* be filtered by ::AP_FTYPE_RESOURCE filters, but all of the "final
* processing" is determined by the main request. */
AP_FTYPE_CONNECTION = 50,
/** These filters don't alter the content. They are responsible for
* sending/receiving data to/from the client. */
AP_FTYPE_NETWORK = 60
} ap_filter_type;
Filters declaring lower numbers run earlier (output from a handler
like mod_proxy/mod_cgi ->mod_include -> mod_deflate -> http filters ->
mod_ssl -> core output filter)
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_cache, mod_deflate, mod_expires and a headache
Posted by Simon Mott <de...@gmail.com>.
Hi All
Still struggling with this one :)
I've tried adding "MOD_EXPIRES" as a filter before "CACHE" but it doesn't
seem to cause mod_expires to be evaluated before its cached (I don't really
understand what AP_FTYPE_CONTENT_SET-2 means in the source for mod_expires)
- Can what I'm trying to do be done?
Essentially, I am trying to achieve something like:
> # Apply mod_expires config to all output BEFORE hitting mod_cache - Then
cache the output
> # Doing this ensures correct cache control headers are in place by
mod_expires prior to mod_cache doing its thing
>
> SetOutputFilter MOD_EXPIRES;CACHE
>
> # pass through mod_deflate after everything else is done for text content
> AddOutputFilterByType DEFLATE text/html text/plain text/css
application/javascript application/rss+xml
Kind Regards,
Simon
On 1 June 2016 at 19:41, Simon Mott <de...@gmail.com> wrote:
> Hi Eric
>
> Thanks for getting back to me. So if there is no mime type when it wants
> to cache which is why the AddOutputFilterType CACHE filter isn't doing what
> I expect, but SetOutputFilter CACHE gets caching before compression
> correct, is there any way I can apply mod_expires and mod_headers before
> passing to be cached also? This ensures that correct cache control headers
> are used prior to being dealt with by mod_cache
>
> Currently with:
>
> > SetOutputFilter CACHE
> > AddOutputFilterByType DEFLATE text/html text/plain text/css
> application/javascript application/rss+xml
>
> output is being cached BEFORE mod_expires adds its headers so its not
> adhering to content expiry times.
>
> Essentially I want Apache to do everything it needs to in order to
> generate content (uncompressed, so as to not maintain multiple copies of
> content based on the "Vary" header), then cache it and finally gzip it
> before it goes out if the client supports gzip via mod_deflate
>
> Is this something that can be achieved or does it require modification of
> modules?
>
> Thanks
>
> On 1 June 2016 at 16:34, Eric Covener <co...@gmail.com> wrote:
>
>> I think the absence of AH00758 in the AddOutputFilterByType case is
>> due to no mimetype being available early enough in processing to allow
>> "CACHE" to be replaced by the actual caching filters when mod_cache
>> wants to do it. It makes sense that no types would be available in
>> the early "handler" mod_cache uses to both serve from the cache and
>> setup caching when there's a miss.
>>
>> Sounds like the doc needs improvement.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>
Re: [users@httpd] mod_cache, mod_deflate, mod_expires and a headache
Posted by Simon Mott <de...@gmail.com>.
Hi Eric
Thanks for getting back to me. So if there is no mime type when it wants to
cache which is why the AddOutputFilterType CACHE filter isn't doing what I
expect, but SetOutputFilter CACHE gets caching before compression correct,
is there any way I can apply mod_expires and mod_headers before passing to
be cached also? This ensures that correct cache control headers are used
prior to being dealt with by mod_cache
Currently with:
> SetOutputFilter CACHE
> AddOutputFilterByType DEFLATE text/html text/plain text/css
application/javascript application/rss+xml
output is being cached BEFORE mod_expires adds its headers so its not
adhering to content expiry times.
Essentially I want Apache to do everything it needs to in order to generate
content (uncompressed, so as to not maintain multiple copies of content
based on the "Vary" header), then cache it and finally gzip it before it
goes out if the client supports gzip via mod_deflate
Is this something that can be achieved or does it require modification of
modules?
Thanks
On 1 June 2016 at 16:34, Eric Covener <co...@gmail.com> wrote:
> I think the absence of AH00758 in the AddOutputFilterByType case is
> due to no mimetype being available early enough in processing to allow
> "CACHE" to be replaced by the actual caching filters when mod_cache
> wants to do it. It makes sense that no types would be available in
> the early "handler" mod_cache uses to both serve from the cache and
> setup caching when there's a miss.
>
> Sounds like the doc needs improvement.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] mod_cache, mod_deflate, mod_expires and a headache
Posted by Eric Covener <co...@gmail.com>.
I think the absence of AH00758 in the AddOutputFilterByType case is
due to no mimetype being available early enough in processing to allow
"CACHE" to be replaced by the actual caching filters when mod_cache
wants to do it. It makes sense that no types would be available in
the early "handler" mod_cache uses to both serve from the cache and
setup caching when there's a miss.
Sounds like the doc needs improvement.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org