You are viewing a plain text version of this content. The canonical link for it is here.
Posted to cvs@httpd.apache.org by mi...@apache.org on 2010/02/27 19:54:41 UTC
svn commit: r917013 - in /httpd/httpd/branches/2.2.x: CHANGES STATUS
docs/manual/mod/mod_cache.xml modules/cache/cache_storage.c
modules/cache/cache_util.c modules/cache/mod_cache.c
modules/cache/mod_cache.h
Author: minfrin
Date: Sat Feb 27 18:54:40 2010
New Revision: 917013
URL: http://svn.apache.org/viewvc?rev=917013&view=rev
Log:
mod_cache: Introduce the thundering herd lock, a mechanism to keep
the flood of requests at bay that strike a backend webserver as
a cached entity goes stale.
+1: minfrin, jim, pgollucci
Modified:
httpd/httpd/branches/2.2.x/CHANGES
httpd/httpd/branches/2.2.x/STATUS
httpd/httpd/branches/2.2.x/docs/manual/mod/mod_cache.xml
httpd/httpd/branches/2.2.x/modules/cache/cache_storage.c
httpd/httpd/branches/2.2.x/modules/cache/cache_util.c
httpd/httpd/branches/2.2.x/modules/cache/mod_cache.c
httpd/httpd/branches/2.2.x/modules/cache/mod_cache.h
Modified: httpd/httpd/branches/2.2.x/CHANGES
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/CHANGES?rev=917013&r1=917012&r2=917013&view=diff
==============================================================================
--- httpd/httpd/branches/2.2.x/CHANGES [utf-8] (original)
+++ httpd/httpd/branches/2.2.x/CHANGES [utf-8] Sat Feb 27 18:54:40 2010
@@ -9,6 +9,10 @@
access control is still vulnerable, unless using OpenSSL >= 0.9.8l.
[Joe Orton, Ruediger Pluem, Hartmut Keil <Hartmut.Keil adnovum.ch>]
+ *) mod_cache: Introduce the thundering herd lock, a mechanism to keep
+ the flood of requests at bay that strike a backend webserver as
+ a cached entity goes stale. [Graham Leggett]
+
*) mod_proxy_http: Make sure that when an ErrorDocument is served
from a reverse proxied URL, that the subrequest respects the status
of the original request. This brings the behaviour of proxy_handler
Modified: httpd/httpd/branches/2.2.x/STATUS
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/STATUS?rev=917013&r1=917012&r2=917013&view=diff
==============================================================================
--- httpd/httpd/branches/2.2.x/STATUS (original)
+++ httpd/httpd/branches/2.2.x/STATUS Sat Feb 27 18:54:40 2010
@@ -87,19 +87,6 @@
PATCHES ACCEPTED TO BACKPORT FROM TRUNK:
[ start all new proposals below, under PATCHES PROPOSED. ]
- * mod_cache: Introduce the thundering herd lock, a mechanism to keep
- the flood of requests at bay that strike a backend webserver as
- a cached entity goes stale.
- Trunk Patch: http://svn.apache.org/viewvc?rev=808212&view=rev
- http://svn.apache.org/viewvc?rev=808220&view=rev
- http://svn.apache.org/viewvc?rev=808649&view=rev
- http://svn.apache.org/viewvc?rev=808652&view=rev
- http://svn.apache.org/viewvc?rev=808656&view=rev
- http://svn.apache.org/viewvc?rev=809440&view=rev
- http://svn.apache.org/viewvc?rev=809665&view=rev
- 2.2.x Patch: http://people.apache.org/~minfrin/httpd-cache-thundering.patch
- +1: minfrin, jim, pgollucci
-
* rewrite_log: remove locking
trunk patch: http://svn.apache.org/viewvc?view=revision&revision=783734
Modified: httpd/httpd/branches/2.2.x/docs/manual/mod/mod_cache.xml
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/docs/manual/mod/mod_cache.xml?rev=917013&r1=917012&r2=917013&view=diff
==============================================================================
--- httpd/httpd/branches/2.2.x/docs/manual/mod/mod_cache.xml (original)
+++ httpd/httpd/branches/2.2.x/docs/manual/mod/mod_cache.xml Sat Feb 27 18:54:40 2010
@@ -125,6 +125,70 @@
</example>
</section>
+<section id="thunderingherd"><title>Avoiding the Thundering Herd</title>
+ <p>When a cached entry becomes stale, <module>mod_cache</module> will submit
+ a conditional request to the backend, which is expected to confirm whether the
+ cached entry is still fresh, and send an updated entity if not.</p>
+ <p>A small but finite amount of time exists between the time the cached entity
+ becomes stale, and the time the stale entity is fully refreshed. On a busy
+ server, a significant number of requests might arrive during this time, and
+ cause a <strong>thundering herd</strong> of requests to strike the backend
+ suddenly and unpredicably.</p>
+ <p>To keep the thundering herd at bay, the <directive>CacheLock</directive>
+ directive can be used to define a directory in which locks are created for
+ URLs <strong>in flight</strong>. The lock is used as a <strong>hint</strong>
+ by other requests to either suppress an attempt to cache (someone else has
+ gone to fetch the entity), or to indicate that a stale entry is being refreshed
+ (stale content will be returned in the mean time).
+ </p>
+ <section>
+ <title>Initial caching of an entry</title>
+ <p>When an entity is cached for the first time, a lock will be created for the
+ entity until the response has been fully cached. During the lifetime of the
+ lock, the cache will suppress the second and subsequent attempt to cache the
+ same entity. While this doesn't hold back the thundering herd, it does stop
+ the cache attempting to cache the same entity multiple times simultaneously.
+ </p>
+ </section>
+ <section>
+ <title>Refreshment of a stale entry</title>
+ <p>When an entity reaches its freshness lifetime and becomes stale, a lock
+ will be created for the entity until the response has either been confirmed as
+ still fresh, or replaced by the backend. During the lifetime of the lock, the
+ second and subsequent incoming request will cause stale data to be returned,
+ and the thundering herd is kept at bay.</p>
+ </section>
+ <section>
+ <title>Locks and Cache-Control: no-cache</title>
+ <p>Locks are used as a <strong>hint only</strong> to enable the cache to be
+ more gentle on backend servers, however the lock can be overridden if necessary.
+ If the client sends a request with a Cache-Control header forcing a reload, any
+ lock that may be present will be ignored, and the client's request will be
+ honoured immediately and the cached entry refreshed.</p>
+ <p>As a further safety mechanism, locks have a configurable maximum age.
+ Once this age has been reached, the lock is removed, and a new request is
+ given the opportunity to create a new lock. This maximum age can be set using
+ the <directive>CacheLockMaxAge</directive> directive, and defaults to 5
+ seconds.
+ </p>
+ </section>
+ <section>
+ <title>Example configuration</title>
+ <example><title>Enabling the cache lock</title>
+ #<br />
+ # Enable the cache lock<br />
+ #<br />
+ <IfModule mod_cache.c><br />
+ <indent>
+ CacheLock on<br />
+ CacheLockPath /tmp/mod_cache-lock<br />
+ CacheLockMaxAge 5<br />
+ </indent>
+ </IfModule>
+ </example>
+ </section>
+</section>
+
<directivesynopsis>
<name>CacheEnable</name>
<description>Enable caching of specified URLs using a specified storage
@@ -513,4 +577,66 @@
<seealso><directive module="mod_cache">CacheIgnoreCacheControl</directive></seealso>
<seealso><directive module="mod_cache">CacheStorePrivate</directive></seealso>
</directivesynopsis>
+
+<directivesynopsis>
+<name>CacheLock</name>
+<description>Enable the thundering herd lock.</description>
+<syntax>CacheLock <var>on|off</var></syntax>
+<default>CacheLock off</default>
+<contextlist><context>server config</context><context>virtual host</context>
+</contextlist>
+
+<usage>
+ <p>The <directive>CacheLock</directive> directive enables the thundering herd lock
+ for the given URL space.</p>
+
+ <p>In a minimal configuration the following directive is all that is needed to
+ enable the thundering herd lock in the default system temp directory.</p>
+
+ <example>
+ # Enable chache lock<br />
+ CacheLock on<br /><br />
+ </example>
+
+</usage>
+</directivesynopsis>
+
+<directivesynopsis>
+<name>CacheLockPath</name>
+<description>Set the lock path directory.</description>
+<syntax>CacheLockPath <var>directory</var></syntax>
+<default>CacheLockPath /tmp/mod_cache-lock</default>
+<contextlist><context>server config</context><context>virtual host</context>
+</contextlist>
+
+<usage>
+ <p>The <directive>CacheLockPath</directive> directive allows you to specify the
+ directory in which the locks are created. By default, the system's temporary
+ folder is used. Locks consist of empty files that only exist for stale URLs
+ in flight, so is significantly less resource intensive than the traditional
+ disk cache.</p>
+
+</usage>
+</directivesynopsis>
+
+<directivesynopsis>
+<name>CacheLockMaxAge</name>
+<description>Set the maximum possible age of a cache lock.</description>
+<syntax>CacheLockMaxAge <var>integer</var></syntax>
+<default>CacheLockMaxAge 5</default>
+<contextlist><context>server config</context><context>virtual host</context>
+</contextlist>
+
+<usage>
+ <p>The <directive>CacheLockMaxAge</directive> directive specifies the maximum
+ age of any cache lock.</p>
+
+ <p>A lock older than this value in seconds will be ignored, and the next
+ incoming request will be given the opportunity to re-establish the lock.
+ This mechanism prevents a slow client taking an excessively long time to refresh
+ an entity.</p>
+
+</usage>
+</directivesynopsis>
+
</modulesynopsis>
Modified: httpd/httpd/branches/2.2.x/modules/cache/cache_storage.c
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/cache/cache_storage.c?rev=917013&r1=917012&r2=917013&view=diff
==============================================================================
--- httpd/httpd/branches/2.2.x/modules/cache/cache_storage.c (original)
+++ httpd/httpd/branches/2.2.x/modules/cache/cache_storage.c Sat Feb 27 18:54:40 2010
@@ -37,7 +37,7 @@
/* Remove the stale cache entry if present. If not, we're
* being called from outside of a request; remove the
- * non-stalle handle.
+ * non-stale handle.
*/
h = cache->stale_handle ? cache->stale_handle : cache->handle;
if (!h) {
Modified: httpd/httpd/branches/2.2.x/modules/cache/cache_util.c
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/cache/cache_util.c?rev=917013&r1=917012&r2=917013&view=diff
==============================================================================
--- httpd/httpd/branches/2.2.x/modules/cache/cache_util.c (original)
+++ httpd/httpd/branches/2.2.x/modules/cache/cache_util.c Sat Feb 27 18:54:40 2010
@@ -22,6 +22,8 @@
/* -------------------------------------------------------------- */
+extern APR_OPTIONAL_FN_TYPE(ap_cache_generate_key) *cache_generate_key;
+
extern module AP_MODULE_DECLARE_DATA cache_module;
/* Determine if "url" matches the hostname, scheme and port and path
@@ -164,9 +166,182 @@
return apr_time_sec(current_age);
}
+/**
+ * Try obtain a cache wide lock on the given cache key.
+ *
+ * If we return APR_SUCCESS, we obtained the lock, and we are clear to
+ * proceed to the backend. If we return APR_EEXISTS, then the lock is
+ * already locked, someone else has gone to refresh the backend data
+ * already, so we must return stale data with a warning in the mean
+ * time. If we return anything else, then something has gone pear
+ * shaped, and we allow the request through to the backend regardless.
+ *
+ * This lock is created from the request pool, meaning that should
+ * something go wrong and the lock isn't deleted on return of the
+ * request headers from the backend for whatever reason, at worst the
+ * lock will be cleaned up when the request dies or finishes.
+ *
+ * If something goes truly bananas and the lock isn't deleted when the
+ * request dies, the lock will be trashed when its max-age is reached,
+ * or when a request arrives containing a Cache-Control: no-cache. At
+ * no point is it possible for this lock to permanently deny access to
+ * the backend.
+ */
+CACHE_DECLARE(apr_status_t) ap_cache_try_lock(cache_server_conf *conf,
+ request_rec *r, char *key) {
+ apr_status_t status;
+ const char *lockname;
+ const char *path;
+ char dir[5];
+ apr_time_t now = apr_time_now();
+ apr_finfo_t finfo;
+ apr_file_t *lockfile;
+ void *dummy;
+
+ finfo.mtime = 0;
+
+ if (!conf || !conf->lock || !conf->lockpath) {
+ /* no locks configured, leave */
+ return APR_SUCCESS;
+ }
+
+ /* lock already obtained earlier? if so, success */
+ apr_pool_userdata_get(&dummy, CACHE_LOCKFILE_KEY, r->pool);
+ if (dummy) {
+ return APR_SUCCESS;
+ }
+
+ /* create the key if it doesn't exist */
+ if (!key) {
+ cache_generate_key(r, r->pool, &key);
+ }
+
+ /* create a hashed filename from the key, and save it for later */
+ lockname = ap_cache_generate_name(r->pool, 0, 0, key);
+
+ /* lock files represent discrete just-went-stale URLs "in flight", so
+ * we support a simple two level directory structure, more is overkill.
+ */
+ dir[0] = '/';
+ dir[1] = lockname[0];
+ dir[2] = '/';
+ dir[3] = lockname[1];
+ dir[4] = 0;
+
+ /* make the directories */
+ path = apr_pstrcat(r->pool, conf->lockpath, dir, NULL);
+ if (APR_SUCCESS != (status = apr_dir_make_recursive(path,
+ APR_UREAD|APR_UWRITE|APR_UEXECUTE, r->pool))) {
+ ap_log_error(APLOG_MARK, APLOG_ERR, status, r->server,
+ "Could not create a cache lock directory: %s",
+ path);
+ return status;
+ }
+ lockname = apr_pstrcat(r->pool, path, "/", lockname, NULL);
+ apr_pool_userdata_set(lockname, CACHE_LOCKNAME_KEY, NULL, r->pool);
+
+ /* is an existing lock file too old? */
+ status = apr_stat(&finfo, lockname,
+ APR_FINFO_MTIME | APR_FINFO_NLINK, r->pool);
+ if (!(APR_STATUS_IS_ENOENT(status)) && APR_SUCCESS != status) {
+ ap_log_error(APLOG_MARK, APLOG_ERR, APR_EEXIST, r->server,
+ "Could not stat a cache lock file: %s",
+ lockname);
+ return status;
+ }
+ if (APR_SUCCESS == status && ((now - finfo.mtime) > conf->lockmaxage)
+ || (now < finfo.mtime)) {
+ ap_log_error(APLOG_MARK, APLOG_INFO, status, r->server,
+ "Cache lock file for '%s' too old, removing: %s",
+ r->uri, lockname);
+ apr_file_remove(lockname, r->pool);
+ }
+
+ /* try obtain a lock on the file */
+ if (APR_SUCCESS == (status = apr_file_open(&lockfile, lockname,
+ APR_WRITE | APR_CREATE | APR_EXCL | APR_DELONCLOSE,
+ APR_UREAD | APR_UWRITE, r->pool))) {
+ apr_pool_userdata_set(lockfile, CACHE_LOCKFILE_KEY, NULL, r->pool);
+ }
+ return status;
+
+}
+
+/**
+ * Remove the cache lock, if present.
+ *
+ * First, try to close the file handle, whose delete-on-close should
+ * kill the file. Otherwise, just delete the file by name.
+ *
+ * If no lock name has yet been calculated, do the calculation of the
+ * lock name first before trying to delete the file.
+ *
+ * If an optional bucket brigade is passed, the lock will only be
+ * removed if the bucket brigade contains an EOS bucket.
+ */
+CACHE_DECLARE(apr_status_t) ap_cache_remove_lock(cache_server_conf *conf,
+ request_rec *r, char *key, apr_bucket_brigade *bb) {
+ void *dummy;
+ const char *lockname;
+
+ if (!conf || !conf->lock || !conf->lockpath) {
+ /* no locks configured, leave */
+ return APR_SUCCESS;
+ }
+ if (bb) {
+ apr_bucket *e;
+ int eos_found = 0;
+ for (e = APR_BRIGADE_FIRST(bb);
+ e != APR_BRIGADE_SENTINEL(bb);
+ e = APR_BUCKET_NEXT(e))
+ {
+ if (APR_BUCKET_IS_EOS(e)) {
+ eos_found = 1;
+ break;
+ }
+ }
+ if (!eos_found) {
+ /* no eos found in brigade, don't delete anything just yet,
+ * we are not done.
+ */
+ return APR_SUCCESS;
+ }
+ }
+ apr_pool_userdata_get(&dummy, CACHE_LOCKFILE_KEY, r->pool);
+ if (dummy) {
+ return apr_file_close((apr_file_t *)dummy);
+ }
+ apr_pool_userdata_get(&dummy, CACHE_LOCKNAME_KEY, r->pool);
+ lockname = (const char *)dummy;
+ if (!lockname) {
+ char dir[5];
+
+ /* create the key if it doesn't exist */
+ if (!key) {
+ cache_generate_key(r, r->pool, &key);
+ }
+
+ /* create a hashed filename from the key, and save it for later */
+ lockname = ap_cache_generate_name(r->pool, 0, 0, key);
+
+ /* lock files represent discrete just-went-stale URLs "in flight", so
+ * we support a simple two level directory structure, more is overkill.
+ */
+ dir[0] = '/';
+ dir[1] = lockname[0];
+ dir[2] = '/';
+ dir[3] = lockname[1];
+ dir[4] = 0;
+
+ lockname = apr_pstrcat(r->pool, conf->lockpath, dir, "/", lockname, NULL);
+ }
+ return apr_file_remove(lockname, r->pool);
+}
+
CACHE_DECLARE(int) ap_cache_check_freshness(cache_handle_t *h,
request_rec *r)
{
+ apr_status_t status;
apr_int64_t age, maxage_req, maxage_cresp, maxage, smaxage, maxstale;
apr_int64_t minfresh;
const char *cc_cresp, *cc_req;
@@ -176,6 +351,7 @@
char *val;
apr_time_t age_c = 0;
cache_info *info = &(h->cache_obj->info);
+ const char *warn_head;
cache_server_conf *conf =
(cache_server_conf *)ap_get_module_config(r->server->module_config,
&cache_module);
@@ -338,7 +514,6 @@
((smaxage == -1) && (maxage == -1) &&
(info->expire != APR_DATE_BAD) &&
(age < (apr_time_sec(info->expire - info->date) + maxstale - minfresh)))) {
- const char *warn_head;
warn_head = apr_table_get(h->resp_hdrs, "Warning");
@@ -361,7 +536,7 @@
}
/*
* If none of Expires, Cache-Control: max-age, or Cache-Control:
- * s-maxage appears in the response, and the respose header age
+ * s-maxage appears in the response, and the response header age
* calculated is more than 24 hours add the warning 113
*/
if ((maxage_cresp == -1) && (smaxage == -1) &&
@@ -380,7 +555,71 @@
return 1; /* Cache object is fresh (enough) */
}
- return 0; /* Cache object is stale */
+ /*
+ * At this point we are stale, but: if we are under load, we may let
+ * a significant number of stale requests through before the first
+ * stale request successfully revalidates itself, causing a sudden
+ * unexpected thundering herd which in turn brings angst and drama.
+ *
+ * So.
+ *
+ * We want the first stale request to go through as normal. But the
+ * second and subsequent request, we must pretend to be fresh until
+ * the first request comes back with either new content or confirmation
+ * that the stale content is still fresh.
+ *
+ * To achieve this, we create a very simple file based lock based on
+ * the key of the cached object. We attempt to open the lock file with
+ * exclusive write access. If we succeed, woohoo! we're first, and we
+ * follow the stale path to the backend server. If we fail, oh well,
+ * we follow the fresh path, and avoid being a thundering herd.
+ *
+ * The lock lives only as long as the stale request that went on ahead.
+ * If the request succeeds, the lock is deleted. If the request fails,
+ * the lock is deleted, and another request gets to make a new lock
+ * and try again.
+ *
+ * At any time, a request marked "no-cache" will force a refresh,
+ * ignoring the lock, ensuring an extended lockout is impossible.
+ *
+ * A lock that exceeds a maximum age will be deleted, and another
+ * request gets to make a new lock and try again.
+ */
+ status = ap_cache_try_lock(conf, r, (char *)h->cache_obj->key);
+ if (APR_SUCCESS == status) {
+ /* we obtained a lock, follow the stale path */
+ ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, r->server,
+ "Cache lock obtained for stale cached URL, "
+ "revalidating entry: %s",
+ r->unparsed_uri);
+ return 0;
+ }
+ else if (APR_EEXIST == status) {
+ /* lock already exists, return stale data anyway, with a warning */
+ ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, r->server,
+ "Cache already locked for stale cached URL, "
+ "pretend it is fresh: %s",
+ r->unparsed_uri);
+
+ /* make sure we don't stomp on a previous warning */
+ warn_head = apr_table_get(h->resp_hdrs, "Warning");
+ if ((warn_head == NULL) ||
+ ((warn_head != NULL) && (ap_strstr_c(warn_head, "110") == NULL))) {
+ apr_table_merge(h->resp_hdrs, "Warning",
+ "110 Response is stale");
+ }
+
+ return 1;
+ }
+ else {
+ /* some other error occurred, just treat the object as stale */
+ ap_log_error(APLOG_MARK, APLOG_DEBUG, status, r->server,
+ "Attempt to obtain a cache lock for stale "
+ "cached URL failed, revalidating entry anyway: %s",
+ r->unparsed_uri);
+ return 0;
+ }
+
}
/*
Modified: httpd/httpd/branches/2.2.x/modules/cache/mod_cache.c
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/cache/mod_cache.c?rev=917013&r1=917012&r2=917013&view=diff
==============================================================================
--- httpd/httpd/branches/2.2.x/modules/cache/mod_cache.c (original)
+++ httpd/httpd/branches/2.2.x/modules/cache/mod_cache.c Sat Feb 27 18:54:40 2010
@@ -114,40 +114,56 @@
if (rv == DECLINED) {
if (!lookup) {
- /*
- * Add cache_save filter to cache this request. Choose
- * the correct filter by checking if we are a subrequest
- * or not.
+ /* try to obtain a cache lock at this point. if we succeed,
+ * we are the first to try and cache this url. if we fail,
+ * it means someone else is already trying to cache this
+ * url, and we should just let the request through to the
+ * backend without any attempt to cache. this stops
+ * duplicated simultaneous attempts to cache an entity.
*/
- if (r->main) {
- ap_log_error(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS,
- r->server,
- "Adding CACHE_SAVE_SUBREQ filter for %s",
- r->uri);
- ap_add_output_filter_handle(cache_save_subreq_filter_handle,
- NULL, r, r->connection);
+ rv = ap_cache_try_lock(conf, r, NULL);
+ if (APR_SUCCESS == rv) {
+
+ /*
+ * Add cache_save filter to cache this request. Choose
+ * the correct filter by checking if we are a subrequest
+ * or not.
+ */
+ if (r->main) {
+ ap_log_error(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS,
+ r->server,
+ "Adding CACHE_SAVE_SUBREQ filter for %s",
+ r->uri);
+ ap_add_output_filter_handle(cache_save_subreq_filter_handle,
+ NULL, r, r->connection);
+ }
+ else {
+ ap_log_error(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS,
+ r->server, "Adding CACHE_SAVE filter for %s",
+ r->uri);
+ ap_add_output_filter_handle(cache_save_filter_handle,
+ NULL, r, r->connection);
+ }
+
+ ap_log_error(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS, r->server,
+ "Adding CACHE_REMOVE_URL filter for %s",
+ r->uri);
+
+ /* Add cache_remove_url filter to this request to remove a
+ * stale cache entry if needed. Also put the current cache
+ * request rec in the filter context, as the request that
+ * is available later during running the filter maybe
+ * different due to an internal redirect.
+ */
+ cache->remove_url_filter =
+ ap_add_output_filter_handle(cache_remove_url_filter_handle,
+ cache, r, r->connection);
}
else {
- ap_log_error(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS,
- r->server, "Adding CACHE_SAVE filter for %s",
- r->uri);
- ap_add_output_filter_handle(cache_save_filter_handle,
- NULL, r, r->connection);
+ ap_log_error(APLOG_MARK, APLOG_DEBUG, rv,
+ r->server, "Cache locked for url, not caching "
+ "response: %s", r->uri);
}
-
- ap_log_error(APLOG_MARK, APLOG_DEBUG, APR_SUCCESS, r->server,
- "Adding CACHE_REMOVE_URL filter for %s",
- r->uri);
-
- /* Add cache_remove_url filter to this request to remove a
- * stale cache entry if needed. Also put the current cache
- * request rec in the filter context, as the request that
- * is available later during running the filter maybe
- * different due to an internal redirect.
- */
- cache->remove_url_filter =
- ap_add_output_filter_handle(cache_remove_url_filter_handle,
- cache, r, r->connection);
}
else {
if (cache->stale_headers) {
@@ -166,7 +182,7 @@
/* error */
ap_log_error(APLOG_MARK, APLOG_ERR, rv, r->server,
"cache: error returned while checking for cached "
- "file by %s cache", cache->provider_name);
+ "file by '%s' cache", cache->provider_name);
}
return DECLINED;
}
@@ -313,6 +329,10 @@
* Check to see if we *can* save this particular response.
* If we can, call cache_create_entity() and save the headers and body
* Finally, pass the data to the next filter (the network or whatever)
+ *
+ * After the various failure cases, the cache lock is proactively removed, so
+ * that another request is given the opportunity to attempt to cache without
+ * waiting for a potentially slow client to acknowledge the failure.
*/
static int cache_save_filter(ap_filter_t *f, apr_bucket_brigade *in)
@@ -328,6 +348,7 @@
cache_info *info = NULL;
char *reason;
apr_pool_t *p;
+ apr_bucket *e;
conf = (cache_server_conf *) ap_get_module_config(r->server->module_config,
&cache_module);
@@ -370,7 +391,19 @@
ap_log_error(APLOG_MARK, APLOG_DEBUG, rv, r->server,
"cache: Cache provider's store_body failed!");
ap_remove_output_filter(f);
+
+ /* give someone else the chance to cache the file */
+ ap_cache_remove_lock(conf, r, cache->handle ?
+ (char *)cache->handle->cache_obj->key : NULL, NULL);
}
+ else {
+
+ /* proactively remove the lock as soon as we see the eos bucket */
+ ap_cache_remove_lock(conf, r, cache->handle ?
+ (char *)cache->handle->cache_obj->key : NULL, in);
+
+ }
+
return ap_pass_brigade(f->next, in);
}
@@ -441,17 +474,17 @@
* telling us to serve the cached copy.
*/
if (exps != NULL || cc_out != NULL) {
- /* We are also allowed to cache any response given that it has a
- * valid Expires or Cache Control header. If we find a either of
- * those here, we pass request through the rest of the tests. From
+ /* We are also allowed to cache any response given that it has a
+ * valid Expires or Cache Control header. If we find a either of
+ * those here, we pass request through the rest of the tests. From
* the RFC:
*
- * A response received with any other status code (e.g. status
- * codes 302 and 307) MUST NOT be returned in a reply to a
- * subsequent request unless there are cache-control directives or
- * another header(s) that explicitly allow it. For example, these
- * include the following: an Expires header (section 14.21); a
- * "max-age", "s-maxage", "must-revalidate", "proxy-revalidate",
+ * A response received with any other status code (e.g. status
+ * codes 302 and 307) MUST NOT be returned in a reply to a
+ * subsequent request unless there are cache-control directives or
+ * another header(s) that explicitly allow it. For example, these
+ * include the following: an Expires header (section 14.21); a
+ * "max-age", "s-maxage", "must-revalidate", "proxy-revalidate",
* "public" or "private" cache-control directive (section 14.9).
*/
}
@@ -544,7 +577,7 @@
"*", NULL)) {
reason = "Vary header contains '*'";
}
- else if (apr_table_get(r->subprocess_env, "no-cache") != NULL) {
+ else if (apr_table_get(r->subprocess_env, "no-cache") != NULL) {
reason = "environment variable 'no-cache' is set";
}
else if (r->no_cache) {
@@ -560,6 +593,10 @@
/* remove this filter from the chain */
ap_remove_output_filter(f);
+ /* remove the lock file unconditionally */
+ ap_cache_remove_lock(conf, r, cache->handle ?
+ (char *)cache->handle->cache_obj->key : NULL, NULL);
+
/* ship the data up the stack */
return ap_pass_brigade(f->next, in);
}
@@ -584,7 +621,6 @@
/* if we don't get the content-length, see if we have all the
* buckets and use their length to calculate the size
*/
- apr_bucket *e;
int all_buckets_here=0;
int unresolved_length = 0;
size=0;
@@ -661,6 +697,8 @@
if (rv != OK) {
/* Caching layer declined the opportunity to cache the response */
ap_remove_output_filter(f);
+ ap_cache_remove_lock(conf, r, cache->handle ?
+ (char *)cache->handle->cache_obj->key : NULL, NULL);
return ap_pass_brigade(f->next, in);
}
@@ -849,16 +887,23 @@
ap_log_error(APLOG_MARK, APLOG_DEBUG, rv, r->server,
"cache: attempt to remove url from cache unsuccessful.");
}
+
}
+ /* let someone else attempt to cache */
+ ap_cache_remove_lock(conf, r, cache->handle ?
+ (char *)cache->handle->cache_obj->key : NULL, NULL);
+
return ap_pass_brigade(f->next, bb);
}
- if(rv != APR_SUCCESS) {
+ if (rv != APR_SUCCESS) {
ap_log_error(APLOG_MARK, APLOG_DEBUG, rv, r->server,
"cache: store_headers failed");
- ap_remove_output_filter(f);
+ ap_remove_output_filter(f);
+ ap_cache_remove_lock(conf, r, cache->handle ?
+ (char *)cache->handle->cache_obj->key : NULL, NULL);
return ap_pass_brigade(f->next, in);
}
@@ -867,8 +912,15 @@
ap_log_error(APLOG_MARK, APLOG_DEBUG, rv, r->server,
"cache: store_body failed");
ap_remove_output_filter(f);
+ ap_cache_remove_lock(conf, r, cache->handle ?
+ (char *)cache->handle->cache_obj->key : NULL, NULL);
+ return ap_pass_brigade(f->next, in);
}
+ /* proactively remove the lock as soon as we see the eos bucket */
+ ap_cache_remove_lock(conf, r, cache->handle ?
+ (char *)cache->handle->cache_obj->key : NULL, in);
+
return ap_pass_brigade(f->next, in);
}
@@ -912,6 +964,7 @@
ap_remove_output_filter(f);
return ap_pass_brigade(f->next, in);
}
+
/* Now remove this cache entry from the cache */
cache_remove_url(cache, r->pool);
@@ -925,6 +978,7 @@
static void * create_cache_config(apr_pool_t *p, server_rec *s)
{
+ const char *tmppath;
cache_server_conf *ps = apr_pcalloc(p, sizeof(cache_server_conf));
/* array of URL prefixes for which caching is enabled */
@@ -957,6 +1011,13 @@
/* array of identifiers that should not be used for key calculation */
ps->ignore_session_id = apr_array_make(p, 10, sizeof(char *));
ps->ignore_session_id_set = CACHE_IGNORE_SESSION_ID_UNSET;
+ ps->lock = 0; /* thundering herd lock defaults to off */
+ ps->lock_set = 0;
+ apr_temp_dir_get(&tmppath, p);
+ if (tmppath) {
+ ps->lockpath = apr_pstrcat(p, tmppath, DEFAULT_CACHE_LOCKPATH, NULL);
+ }
+ ps->lockmaxage = apr_time_from_sec(DEFAULT_CACHE_MAXAGE);
return ps;
}
@@ -1010,6 +1071,18 @@
(overrides->ignore_session_id_set == CACHE_IGNORE_SESSION_ID_UNSET)
? base->ignore_session_id
: overrides->ignore_session_id;
+ ps->lock =
+ (overrides->lock_set == 0)
+ ? base->lock
+ : overrides->lock;
+ ps->lockpath =
+ (overrides->lockpath_set == 0)
+ ? base->lockpath
+ : overrides->lockpath;
+ ps->lockmaxage =
+ (overrides->lockmaxage_set == 0)
+ ? base->lockmaxage
+ : overrides->lockmaxage;
return ps;
}
static const char *set_cache_ignore_no_last_mod(cmd_parms *parms, void *dummy,
@@ -1229,6 +1302,55 @@
return NULL;
}
+static const char *set_cache_lock(cmd_parms *parms, void *dummy,
+ int flag)
+{
+ cache_server_conf *conf;
+
+ conf =
+ (cache_server_conf *)ap_get_module_config(parms->server->module_config,
+ &cache_module);
+ conf->lock = flag;
+ conf->lock_set = 1;
+ return NULL;
+}
+
+static const char *set_cache_lock_path(cmd_parms *parms, void *dummy,
+ const char *arg)
+{
+ cache_server_conf *conf;
+
+ conf =
+ (cache_server_conf *)ap_get_module_config(parms->server->module_config,
+ &cache_module);
+
+ conf->lockpath = ap_server_root_relative(parms->pool, arg);
+ if (!conf->lockpath) {
+ return apr_pstrcat(parms->pool, "Invalid CacheLockPath path ",
+ arg, NULL);
+ }
+ conf->lockpath_set = 1;
+ return NULL;
+}
+
+static const char *set_cache_lock_maxage(cmd_parms *parms, void *dummy,
+ const char *arg)
+{
+ cache_server_conf *conf;
+ apr_int64_t seconds;
+
+ conf =
+ (cache_server_conf *)ap_get_module_config(parms->server->module_config,
+ &cache_module);
+ seconds = apr_atoi64(arg);
+ if (seconds <= 0) {
+ return "CacheLockMaxAge value must be a non-zero positive integer";
+ }
+ conf->lockmaxage = apr_time_from_sec(seconds);
+ conf->lockmaxage_set = 1;
+ return NULL;
+}
+
static int cache_post_config(apr_pool_t *p, apr_pool_t *plog,
apr_pool_t *ptemp, server_rec *s)
{
@@ -1288,6 +1410,15 @@
AP_INIT_TAKE1("CacheLastModifiedFactor", set_cache_factor, NULL, RSRC_CONF,
"The factor used to estimate Expires date from "
"LastModified date"),
+ AP_INIT_FLAG("CacheLock", set_cache_lock,
+ NULL, RSRC_CONF,
+ "Enable or disable the thundering herd lock."),
+ AP_INIT_TAKE1("CacheLockPath", set_cache_lock_path, NULL, RSRC_CONF,
+ "The thundering herd lock path. Defaults to the '"
+ DEFAULT_CACHE_LOCKPATH "' directory in the system "
+ "temp directory."),
+ AP_INIT_TAKE1("CacheLockMaxAge", set_cache_lock_maxage, NULL, RSRC_CONF,
+ "Maximum age of any thundering herd lock."),
{NULL}
};
Modified: httpd/httpd/branches/2.2.x/modules/cache/mod_cache.h
URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/cache/mod_cache.h?rev=917013&r1=917012&r2=917013&view=diff
==============================================================================
--- httpd/httpd/branches/2.2.x/modules/cache/mod_cache.h (original)
+++ httpd/httpd/branches/2.2.x/modules/cache/mod_cache.h Sat Feb 27 18:54:40 2010
@@ -24,7 +24,7 @@
*/
#ifndef MOD_CACHE_H
-#define MOD_CACHE_H
+#define MOD_CACHE_H
#define CORE_PRIVATE
@@ -86,9 +86,13 @@
#define DEFAULT_CACHE_MAXEXPIRE MSEC_ONE_DAY
#define DEFAULT_CACHE_EXPIRE MSEC_ONE_HR
#define DEFAULT_CACHE_LMFACTOR (0.1)
+#define DEFAULT_CACHE_MAXAGE 5
+#define DEFAULT_CACHE_LOCKPATH "/mod_cache-lock"
+#define CACHE_LOCKNAME_KEY "mod_cache-lockname"
+#define CACHE_LOCKFILE_KEY "mod_cache-lockfile"
-/* Create a set of PROXY_DECLARE(type), PROXY_DECLARE_NONSTD(type) and
- * PROXY_DECLARE_DATA with appropriate export and import tags for the platform
+/* Create a set of CACHE_DECLARE(type), CACHE_DECLARE_NONSTD(type) and
+ * CACHE_DECLARE_DATA with appropriate export and import tags for the platform
*/
#if !defined(WIN32)
#define CACHE_DECLARE(type) type
@@ -134,7 +138,7 @@
int factor_set;
/** ignore the last-modified header when deciding to cache this request */
int no_last_mod_ignore_set;
- int no_last_mod_ignore;
+ int no_last_mod_ignore;
/** ignore client's requests for uncached responses */
int ignorecachecontrol;
int ignorecachecontrol_set;
@@ -159,6 +163,13 @@
#define CACHE_IGNORE_SESSION_ID_SET 1
#define CACHE_IGNORE_SESSION_ID_UNSET 0
int ignore_session_id_set;
+ /* thundering herd lock */
+ int lock;
+ int lock_set;
+ const char *lockpath;
+ int lockpath_set;
+ apr_time_t lockmaxage;
+ int lockmaxage_set;
} cache_server_conf;
/* cache info information */
@@ -173,11 +184,11 @@
/* cache handle information */
-/* XXX TODO On the next structure change/MMN bump,
+/* XXX TODO On the next structure change/MMN bump,
* count must become an apr_off_t, representing
* the potential size of disk cached objects.
* Then dig for
- * "XXX Bad Temporary Cast - see cache_object_t notes"
+ * "XXX Bad Temporary Cast - see cache_object_t notes"
*/
typedef struct cache_object cache_object_t;
struct cache_object {
@@ -206,7 +217,7 @@
apr_status_t (*store_headers)(cache_handle_t *h, request_rec *r, cache_info *i);
apr_status_t (*store_body)(cache_handle_t *h, request_rec *r, apr_bucket_brigade *b);
apr_status_t (*recall_headers) (cache_handle_t *h, request_rec *r);
- apr_status_t (*recall_body) (cache_handle_t *h, apr_pool_t *p, apr_bucket_brigade *bb);
+ apr_status_t (*recall_body) (cache_handle_t *h, apr_pool_t *p, apr_bucket_brigade *bb);
int (*create_entity) (cache_handle_t *h, request_rec *r,
const char *urlkey, apr_off_t len);
int (*open_entity) (cache_handle_t *h, request_rec *r,
@@ -260,6 +271,45 @@
CACHE_DECLARE(int) ap_cache_check_freshness(cache_handle_t *h, request_rec *r);
/**
+ * Try obtain a cache wide lock on the given cache key.
+ *
+ * If we return APR_SUCCESS, we obtained the lock, and we are clear to
+ * proceed to the backend. If we return APR_EEXISTS, the the lock is
+ * already locked, someone else has gone to refresh the backend data
+ * already, so we must return stale data with a warning in the mean
+ * time. If we return anything else, then something has gone pear
+ * shaped, and we allow the request through to the backend regardless.
+ *
+ * This lock is created from the request pool, meaning that should
+ * something go wrong and the lock isn't deleted on return of the
+ * request headers from the backend for whatever reason, at worst the
+ * lock will be cleaned up when the request is dies or finishes.
+ *
+ * If something goes truly bananas and the lock isn't deleted when the
+ * request dies, the lock will be trashed when its max-age is reached,
+ * or when a request arrives containing a Cache-Control: no-cache. At
+ * no point is it possible for this lock to permanently deny access to
+ * the backend.
+ */
+CACHE_DECLARE(apr_status_t) ap_cache_try_lock(cache_server_conf *conf,
+ request_rec *r, char *key);
+
+/**
+ * Remove the cache lock, if present.
+ *
+ * First, try to close the file handle, whose delete-on-close should
+ * kill the file. Otherwise, just delete the file by name.
+ *
+ * If no lock name has yet been calculated, do the calculation of the
+ * lock name first before trying to delete the file.
+ *
+ * If an optional bucket brigade is passed, the lock will only be
+ * removed if the bucket brigade contains an EOS bucket.
+ */
+CACHE_DECLARE(apr_status_t) ap_cache_remove_lock(cache_server_conf *conf,
+ request_rec *r, char *key, apr_bucket_brigade *bb);
+
+/**
* Merge in cached headers into the response
* @param h cache_handle_t
* @param r request_rec
@@ -330,8 +380,8 @@
#define CACHE_DECLARE_DATA __declspec(dllimport)
#endif
-APR_DECLARE_OPTIONAL_FN(apr_status_t,
- ap_cache_generate_key,
+APR_DECLARE_OPTIONAL_FN(apr_status_t,
+ ap_cache_generate_key,
(request_rec *r, apr_pool_t*p, char**key ));
Re: svn commit: r917013 - in /httpd/httpd/branches/2.2.x: CHANGES
STATUS docs/manual/mod/mod_cache.xml modules/cache/cache_storage.c modules/cache/cache_util.c
modules/cache/mod_cache.c modules/cache/mod_cache.h
Posted by Rainer Jung <ra...@kippdata.de>.
On 27.02.2010 19:54, minfrin@apache.org wrote:
> Author: minfrin
> Date: Sat Feb 27 18:54:40 2010
> New Revision: 917013
>
> URL: http://svn.apache.org/viewvc?rev=917013&view=rev
> Log:
> mod_cache: Introduce the thundering herd lock, a mechanism to keep
> the flood of requests at bay that strike a backend webserver as
> a cached entity goes stale.
> +1: minfrin, jim, pgollucci
>
> Modified:
> httpd/httpd/branches/2.2.x/CHANGES
> httpd/httpd/branches/2.2.x/STATUS
> httpd/httpd/branches/2.2.x/docs/manual/mod/mod_cache.xml
> httpd/httpd/branches/2.2.x/modules/cache/cache_storage.c
> httpd/httpd/branches/2.2.x/modules/cache/cache_util.c
> httpd/httpd/branches/2.2.x/modules/cache/mod_cache.c
> httpd/httpd/branches/2.2.x/modules/cache/mod_cache.h
>
> Modified: httpd/httpd/branches/2.2.x/modules/cache/cache_util.c
> URL: http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/cache/cache_util.c?rev=917013&r1=917012&r2=917013&view=diff
> ==============================================================================
> --- httpd/httpd/branches/2.2.x/modules/cache/cache_util.c (original)
> +++ httpd/httpd/branches/2.2.x/modules/cache/cache_util.c Sat Feb 27 18:54:40 2010
> @@ -22,6 +22,8 @@
>
> /* -------------------------------------------------------------- */
>
> +extern APR_OPTIONAL_FN_TYPE(ap_cache_generate_key) *cache_generate_key;
> +
> extern module AP_MODULE_DECLARE_DATA cache_module;
>
> /* Determine if "url" matches the hostname, scheme and port and path
> @@ -164,9 +166,182 @@
> return apr_time_sec(current_age);
> }
>
> +/**
> + * Try obtain a cache wide lock on the given cache key.
> + *
> + * If we return APR_SUCCESS, we obtained the lock, and we are clear to
> + * proceed to the backend. If we return APR_EEXISTS, then the lock is
> + * already locked, someone else has gone to refresh the backend data
> + * already, so we must return stale data with a warning in the mean
> + * time. If we return anything else, then something has gone pear
> + * shaped, and we allow the request through to the backend regardless.
> + *
> + * This lock is created from the request pool, meaning that should
> + * something go wrong and the lock isn't deleted on return of the
> + * request headers from the backend for whatever reason, at worst the
> + * lock will be cleaned up when the request dies or finishes.
> + *
> + * If something goes truly bananas and the lock isn't deleted when the
> + * request dies, the lock will be trashed when its max-age is reached,
> + * or when a request arrives containing a Cache-Control: no-cache. At
> + * no point is it possible for this lock to permanently deny access to
> + * the backend.
> + */
> +CACHE_DECLARE(apr_status_t) ap_cache_try_lock(cache_server_conf *conf,
> + request_rec *r, char *key) {
> + apr_status_t status;
> + const char *lockname;
> + const char *path;
> + char dir[5];
> + apr_time_t now = apr_time_now();
> + apr_finfo_t finfo;
> + apr_file_t *lockfile;
> + void *dummy;
> +
> + finfo.mtime = 0;
> +
> + if (!conf || !conf->lock || !conf->lockpath) {
> + /* no locks configured, leave */
> + return APR_SUCCESS;
> + }
> +
> + /* lock already obtained earlier? if so, success */
> + apr_pool_userdata_get(&dummy, CACHE_LOCKFILE_KEY, r->pool);
> + if (dummy) {
> + return APR_SUCCESS;
> + }
> +
> + /* create the key if it doesn't exist */
> + if (!key) {
> + cache_generate_key(r, r->pool,&key);
> + }
> +
> + /* create a hashed filename from the key, and save it for later */
> + lockname = ap_cache_generate_name(r->pool, 0, 0, key);
> +
> + /* lock files represent discrete just-went-stale URLs "in flight", so
> + * we support a simple two level directory structure, more is overkill.
> + */
> + dir[0] = '/';
> + dir[1] = lockname[0];
> + dir[2] = '/';
> + dir[3] = lockname[1];
> + dir[4] = 0;
> +
> + /* make the directories */
> + path = apr_pstrcat(r->pool, conf->lockpath, dir, NULL);
> + if (APR_SUCCESS != (status = apr_dir_make_recursive(path,
> + APR_UREAD|APR_UWRITE|APR_UEXECUTE, r->pool))) {
> + ap_log_error(APLOG_MARK, APLOG_ERR, status, r->server,
> + "Could not create a cache lock directory: %s",
> + path);
> + return status;
> + }
> + lockname = apr_pstrcat(r->pool, path, "/", lockname, NULL);
> + apr_pool_userdata_set(lockname, CACHE_LOCKNAME_KEY, NULL, r->pool);
> +
> + /* is an existing lock file too old? */
> + status = apr_stat(&finfo, lockname,
> + APR_FINFO_MTIME | APR_FINFO_NLINK, r->pool);
> + if (!(APR_STATUS_IS_ENOENT(status))&& APR_SUCCESS != status) {
> + ap_log_error(APLOG_MARK, APLOG_ERR, APR_EEXIST, r->server,
> + "Could not stat a cache lock file: %s",
> + lockname);
> + return status;
> + }
> + if (APR_SUCCESS == status&& ((now - finfo.mtime)> conf->lockmaxage)
> + || (now< finfo.mtime)) {
Compiler warns:
cache_util.c: In function 'ap_cache_try_lock':
cache_util.c:253: warning: suggest parentheses around && within ||
and indeed trunk has additional parentheses surrounding the "or" clause:
If ((status == APR_SUCCESS) && (((now - finfo.mtime) > conf->lockmaxage)
|| (now < finfo.mtime))) {
It seems the backport missed something.
> + ap_log_error(APLOG_MARK, APLOG_INFO, status, r->server,
> + "Cache lock file for '%s' too old, removing: %s",
> + r->uri, lockname);
> + apr_file_remove(lockname, r->pool);
> + }
> +
> + /* try obtain a lock on the file */
> + if (APR_SUCCESS == (status = apr_file_open(&lockfile, lockname,
> + APR_WRITE | APR_CREATE | APR_EXCL | APR_DELONCLOSE,
> + APR_UREAD | APR_UWRITE, r->pool))) {
> + apr_pool_userdata_set(lockfile, CACHE_LOCKFILE_KEY, NULL, r->pool);
> + }
> + return status;
> +
> +}
Regards,
Rainer