You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by "Bhattacharya, Sudip" <su...@genpact.com> on 2012/09/12 14:15:52 UTC

[users@httpd] Questions on Apache Caching 

I am using mod_cache module to enable caching of certain subfolders in VirtualHost section which is mapped to proxy requests to a remote server.

Is there any way to log requests specific to cached files? I want to know:
1. Which files are actually getting cached
2. How many requests are actually getting served from local cache. Effectively how much bandwidth I am saving by serving files from local cache.
3. Is it possible to avoid the default folder structure of the caching, and get a mirror structure instead of the remote path/folder/file structure instead?
4. Is it possible to enable caching for specific file extensions, or by MIME type

<VirtualHost *>
    ServerName foo.bar.com
.
.
.
    ProxyRequests Off
    ProxyPreserveHost Off
    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>
 
    ProxyPass / http://fooremote.bar.com /
    ProxyPassReverse / http://fooremote.bar.com/
.
.
	<IfModule mod_cache.c>
		CacheDefaultExpire 3600

		CacheEnable disk /css/
		CacheEnable disk /javascript/
		CacheEnable disk /dwr/

		CacheRoot "/cache/"
		CacheDirLevels 2
		CacheDirLength 1
		CacheMaxFileSize 1000000
		CacheMinFileSize 1
		CacheIgnoreCacheControl Off
		CacheIgnoreNoLastMod On
		CacheIgnoreQueryString Off
		CacheIgnoreHeaders None
		CacheLastModifiedFactor 0.1
		CacheDefaultExpire 3600
		CacheMaxExpire 86400
		CacheStoreNoStore On
		CacheStorePrivate On
	</IfModule>
</VirtualHost>
This e-mail (and any attachments), is confidential and may be privileged. It may be read, copied and used only
by intended recipients. Unauthorized access to this e-mail (or attachments) and disclosure or copying of its 
contents or any action taken in reliance on it is unlawful. Unintended recipients must notify the sender immediately 
by e-mail/phone & delete it from their system without making any copies or disclosing it to a third person.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] Questions on Apache Caching

Posted by Tom Evans <te...@googlemail.com>.
On Wed, Sep 12, 2012 at 3:37 PM, Bhattacharya, Sudip
<su...@genpact.com> wrote:
> Hi Tom,
>
> Can you explain further on the "Manual Synchronization" process?
> Do I have to manually copy paste the static files from backend server to my local cache folder(create a mirror copy) for achieving this?
> Or are you referring to hitting all the URLs manually to force Apache to cache them before users end start using the proxy.
>
> Also how can I check for local files and conditionally proxy to the backend?

So, this can only happen in a few scenarios. At $JOB, we have reverse
proxies which proxy back to application servers. The application
servers used to be directly internet facing, and so they have every
static file that is needed to be served locally on the application
server disks.

However, it is very wasteful and slow to reverse proxy to static
files, and to tie up these expensive application server web processes
(which are massive compared to the lightweight processes used on the
reverse proxy). Therefore, we checkout an identical copy of these
static files onto the reverse proxy, so that they can be served
directly to clients.

We then can use various methods to determine whether to proxy a
request, or to serve the file off disk. If the static files are all in
one sub directory, something like this works great on the proxy:

ProxyPass /static/ !
ProxyPass / http://backend/

Otherwise, rewrite rules can do the trick

RewriteCond %{REQUEST_URI} ^(.*)$
RewriteCond /path/to/docroot/%1 !-f
RewriteRule ^/(.*) http://backend/$1

(Brief description: capture the request uri in back-reference %1,
check to see if that file does not exist on disk, in which case proxy
to the backend)

Obviously, you then have to keep files in sync between proxy and
backend, and nothing automatically happens as it would with caching.

wrt configuring caching per file type, it is difficult in apache to do
things based upon the proxied response (or if there is a simple way,
I'm missing it). You would want to examine the returned mime type, and
then set appropriate cache-control headers. If I'm being dense and
missing an obvious solution, someone please point it out.

Cheers

Tom

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

RE: [users@httpd] Questions on Apache Caching

Posted by "Bhattacharya, Sudip" <su...@genpact.com>.
Hi Tom,

Can you explain further on the "Manual Synchronization" process? 
Do I have to manually copy paste the static files from backend server to my local cache folder(create a mirror copy) for achieving this?
Or are you referring to hitting all the URLs manually to force Apache to cache them before users end start using the proxy.

Also how can I check for local files and conditionally proxy to the backend?

Is there no way to specify caching by file extension/regex pattern? I am only able to mark specific folders for caching as of now. If a folder has mixed content, static and dynamic files, I am not able to mark only static files for caching.
I am working with a 3rd party backend application, for which source code is not available, so I can't set the proper caching headers from the backend application.

BTW, the %R gave me the exact detail on caching in the log that I was looking for. Thanks a lot for that :)

Regards,
Sudip

-----Original Message-----
From: Tom Evans [mailto:tevans.uk@googlemail.com] 
Sent: Wednesday, September 12, 2012 7:06 PM
To: users@httpd.apache.org
Subject: Re: [users@httpd] Questions on Apache Caching

On Wed, Sep 12, 2012 at 1:15 PM, Bhattacharya, Sudip <su...@genpact.com> wrote:
> I am using mod_cache module to enable caching of certain subfolders in VirtualHost section which is mapped to proxy requests to a remote server.
>
> Is there any way to log requests specific to cached files? I want to know:
> 1. Which files are actually getting cached 2. How many requests are 
> actually getting served from local cache. Effectively how much bandwidth I am saving by serving files from local cache.
> 3. Is it possible to avoid the default folder structure of the caching, and get a mirror structure instead of the remote path/folder/file structure instead?
> 4. Is it possible to enable caching for specific file extensions, or 
> by MIME type
>

In 2.2 you can alter the access log to output the handler that handled the request (the format for this is %R) - when an item is returned from the cache the handler will be empty.

In 2.4 you can add set "CacheHeader on", and each response will have a "X-Cache" header added to it that says whether it was a hit or not, which can then be logged.

You cannot alter the cache file structure in that manner. However, if you can synchronise all these files from the backend to the cache, you could then only proxy to the backend if the file is not present on the frontend. This would require manual synchronisation of files on both sides.

Whether a resource is stored in the cache or not depends upon the headers that the response has. See the section labelled "What Can be Cached" here:

http://httpd.apache.org/docs/2.2/caching.html

If you don't want a particular resource to be cached, or vice-versa, set appropriate response headers. "Cache-Control: private" would allow the client receiving the resource to cache it, but not your apache cache, nor any intermediate cache between there and the client.

Cheers

Tom

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


This e-mail (and any attachments), is confidential and may be privileged. It may be read, copied and used only
by intended recipients. Unauthorized access to this e-mail (or attachments) and disclosure or copying of its 
contents or any action taken in reliance on it is unlawful. Unintended recipients must notify the sender immediately 
by e-mail/phone & delete it from their system without making any copies or disclosing it to a third person.

Re: [users@httpd] Questions on Apache Caching

Posted by Tom Evans <te...@googlemail.com>.
On Wed, Sep 12, 2012 at 1:15 PM, Bhattacharya, Sudip
<su...@genpact.com> wrote:
> I am using mod_cache module to enable caching of certain subfolders in VirtualHost section which is mapped to proxy requests to a remote server.
>
> Is there any way to log requests specific to cached files? I want to know:
> 1. Which files are actually getting cached
> 2. How many requests are actually getting served from local cache. Effectively how much bandwidth I am saving by serving files from local cache.
> 3. Is it possible to avoid the default folder structure of the caching, and get a mirror structure instead of the remote path/folder/file structure instead?
> 4. Is it possible to enable caching for specific file extensions, or by MIME type
>

In 2.2 you can alter the access log to output the handler that handled
the request (the format for this is %R) - when an item is returned
from the cache the handler will be empty.

In 2.4 you can add set "CacheHeader on", and each response will have a
"X-Cache" header added to it that says whether it was a hit or not,
which can then be logged.

You cannot alter the cache file structure in that manner. However, if
you can synchronise all these files from the backend to the cache, you
could then only proxy to the backend if the file is not present on the
frontend. This would require manual synchronisation of files on both
sides.

Whether a resource is stored in the cache or not depends upon the
headers that the response has. See the section labelled "What Can be
Cached" here:

http://httpd.apache.org/docs/2.2/caching.html

If you don't want a particular resource to be cached, or vice-versa,
set appropriate response headers. "Cache-Control: private" would allow
the client receiving the resource to cache it, but not your apache
cache, nor any intermediate cache between there and the client.

Cheers

Tom

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org