You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2010/11/22 21:09:45 UTC
DO NOT REPLY [Bug 50317] New: Possible error in mod_cache's
"Avoiding the Thundering Herd"
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
Summary: Possible error in mod_cache's "Avoiding the Thundering
Herd"
Product: Apache httpd-2
Version: 2.2.17
Platform: PC
OS/Version: Linux
Status: NEW
Severity: normal
Priority: P2
Component: mod_cache
AssignedTo: bugs@httpd.apache.org
ReportedBy: jules@neofonie.de
Created an attachment (id=26333)
--> (https://issues.apache.org/bugzilla/attachment.cgi?id=26333)
Proposed fix
Hello.
I configured apache (2.2.17) as written in
http://httpd.apache.org/docs/current/mod/mod_cache.html#thunderingherd
Nevertheless, more than one request hit the backend. The Backend sends neither
a ETag nor Last-Modified header but it sends an Expires header. I took a look
into mod_cache and made the attached patch.
The problem was in cache_storage.c:
irv = cache->provider->remove_url(h, r->pool);
Because of this line, there is no stale content left to serve, while getting a
newer version from the backend. There is no use in deleting this entry anyway,
because at this point, there is still interest in this url and the cache entry
will be regenerated very soon.
While investigating this, I found:
rv = ap_cache_try_lock(conf, r, NULL);
in mod_cache.c. This seems useless, because the lockfile is already created
in cache_util.c ap_cache_check_freshness(). Because of the NULL, it uses
the same lockfile for different URLs. I am unsure about the intention of
this second lock.
With kind regards,
Julius Gehr
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #51 from Ruediger Pluem <rp...@apache.org> ---
(In reply to jkaluza from comment #50)
> In trunk, only patch from Comment 41 is needed. I've committed it in
> r1597533.
I guess this one is not needed on trunk since key management does not seem to
use the keys generated by the providers any longer. So I would tend to revert
it. Let it sit for 1 or 2 days and see if we get comments from other people who
know the cache code well (especially Graham) to see what is the correct thing
to do.
>
> For 2.2.x, I will create proposal for merged "Covener's patch" from this bug
> + the one from Comment 41 + my one. Does it sound OK?
Sounds great.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #54 from jkaluza@redhat.com ---
(In reply to Yann Ylavic from comment #53)
> (In reply to jkaluza from comment #52)
> > Proposed for 2.2.x in r1597706, please vote :).
>
> Got my vote, but not yours ;)
> (same for your commit r1572092 in 2.4.x/STATUS)
>
> BTW, thanks all for this fix, nice team work!
Hm, I didn't know I can vote for my own patches...
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #41 from Ruediger Pluem <rp...@apache.org> ---
(In reply to jkaluza from comment #38)
> No, it doesn't. "key" is still NULL even when "accept-encodingURL" is
> locked, so ap_cache_try_lock(...) generates new key without
> "accept-encoding" leading to second lock and second request to backend.
Can you please try the following patch instead?
Index: modules/cache/mod_cache.c
===================================================================
--- modules/cache/mod_cache.c (revision 1595827)
+++ modules/cache/mod_cache.c (working copy)
@@ -113,7 +113,27 @@
if (rv != OK) {
if (rv == DECLINED) {
if (!lookup) {
+ char *key;
+ cache_handle_t *h;
+ /*
+ * Try to use the key of a possible open but stall cache
+ * entry if we have one.
+ */
+ if (cache->handle != NULL) {
+ h = cache->handle;
+ }
+ else {
+ h = cache->stale_handle;
+ }
+ if ((h != NULL) &&
+ (h->cache_obj != NULL) &&
+ (h->cache_obj->key != NULL)) {
+ key = apr_pstrdup(r->pool, h->cache_obj->key);
+ }
+ else {
+ key = NULL;
+ }
/* try to obtain a cache lock at this point. if we succeed,
* we are the first to try and cache this url. if we fail,
* it means someone else is already trying to cache this
@@ -121,7 +141,7 @@
* backend without any attempt to cache. this stops
* duplicated simultaneous attempts to cache an entity.
*/
- rv = ap_cache_try_lock(conf, r, NULL);
+ rv = ap_cache_try_lock(conf, r, key);
if (APR_SUCCESS == rv) {
/*
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
macbogucki@gmail.com changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |macbogucki@gmail.com
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #26 from macbogucki@gmail.com ---
Here is the bug in the Red Hat Bugzilla -
https://bugzilla.redhat.com/show_bug.cgi?id=1094990
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #15 from Jim Riggs <ji...@riggs.me> ---
(In reply to macbogucki from comment #14)
> I have just attached 3 files
> * access_log
> * error_log
> * tomcat_log (application server log)
>
>
> * At 07:31:40 I ran the command '/usr/bin/GET -des http://apache-app1/url'
> ** There is nothing in the cache so the response was inserted in the cache
>
> * At 07:32:25 I ran the command '/usr/bin/ab -k -c 2 -n 4
> http://apache-app1/url'
> ** All the responses was served from the cache
>
> * At 07:32:37 I ran the command '/usr/bin/ab -k -c 2 -n 4
> http://apache-app1/url'
> ** 3 responses was served from the cache but 3 was serverd from the
> application server.
>
> I don't understand why only 3 responses were served from the cache in the
> third case.
I don't understand why you are seeing different results. I just ran a test with
your configuration (except I set max-age=10). I then ran `ab -k -c 2 -t 30'
(30-second test), and only 3 requests went to the backend as expected.
Covener - What are you seeing?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
Jim Riggs <ji...@riggs.me> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |jim@riggs.me
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #39 from jkaluza@redhat.com ---
Relevant part from log after your patch + debug log to log "key" before
ap_cache_try_lock(...):
[Thu May 22 07:50:47 2014] [debug] obtained key:
accept-encodinghttp://localhost:80/examples/jsp/jsp2/el/basic-arithmetic.jsp?
[Thu May 22 07:50:47 2014] [debug] cache_util.c(604): Cache lock obtained for
stale cached URL, revalidating entry:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Thu May 22 07:50:47 2014] [debug] cache_storage.c(272): Cached response for
/examples/jsp/jsp2/el/basic-arithmetic.jsp isn't fresh. Adding/replacing
conditional request headers.
[Thu May 22 07:50:47 2014] [debug] cache: cache_url_handler uses key '(null)'
[Thu May 22 07:50:47 2014] [debug] mod_cache.c(156): Adding CACHE_SAVE filter
for /examples/jsp/jsp2/el/basic-arithmetic.jsp
>From first line you see that lock is obtained with key with "accept-encoding,
then the code you've patched is executed and NULL is used as a next key. Second
lock is obtained and CACHE_SAVE filter added.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #30 from jkaluza@redhat.com ---
Created attachment 31643
--> https://issues.apache.org/bugzilla/attachment.cgi?id=31643&action=edit
error_log2 with more debugging
This error log shows one second when running "ab" and tomcat with patched
httpd. It shows that two requests have been stored into the cache in the same
second. Note that this is not "Initial caching of an entry" phase.
Logging has been enhanced to log ap_cache_remove_lock calls and special calls
of these two methods commented with "proactively remove the lock as soon as we
see the eos bucket" in the mod_cache.c
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #52 from jkaluza@redhat.com ---
Proposed for 2.2.x in r1597706, please vote :).
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 50317] Possible error in mod_cache's "Avoiding the
Thundering Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
Rein Tollevik <re...@basefarm.no> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |rein@basefarm.no
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #13 from macbogucki@gmail.com ---
Created attachment 31524
--> https://issues.apache.org/bugzilla/attachment.cgi?id=31524&action=edit
tomcat_log
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #28 from jkaluza@redhat.com ---
OK, probably Murphy's law, but I've reproduced it right when I sent the
previous comment. I will dig deeper to find out what's causing this.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #40 from jkaluza@redhat.com ---
It's strange that ap_cache_check_freshness() returned 0 in this case right
after the cached file update. Can't it be some sort of race condition between
locking and cached file being written on disk?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
Eric Covener <co...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--- Comment #9 from Eric Covener <co...@gmail.com> ---
(In reply to macbogucki from comment #8)
> Thank You very much for the patch but it doesn't works. When I'm doing ab
> (/usr/bin/ab -k -c 5 -n 10 http://host/url) test the application get more
> than one request
>
hard to review without Age headers and LogLevel debug output (* before/after)
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #23 from Yann Ylavic <yl...@gmail.com> ---
(In reply to macbogucki from comment #22)
> The problem is when I combine mod_proxy, mod_deflate, mod_cache
> * When I disable mod_deflate it works
Does it work when you let mod_deflate enabled but use "DeflateAlterETag
NoChange" from r1586542?
Just to know whether the Thundering Herd fix is misbehaving or the "-gzip"
appended ETag is the issue.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
macbogucki@gmail.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Version|2.2.17 |2.2.26
--- Comment #5 from macbogucki@gmail.com ---
Changed to 2.2.26 bug.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #53 from Yann Ylavic <yl...@gmail.com> ---
(In reply to jkaluza from comment #52)
> Proposed for 2.2.x in r1597706, please vote :).
Got my vote, but not yours ;)
(same for your commit r1572092 in 2.4.x/STATUS)
BTW, thanks all for this fix, nice team work!
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #49 from Ruediger Pluem <rp...@apache.org> ---
(In reply to Yann Ylavic from comment #48)
> apr/include.apr_file_io.h :
> /**
> * Rename the specified file.
> * [...]
> * @warning If a file exists at the new location, then it will be
> * overwritten. [...]
> */
> APR_DECLARE(apr_status_t) apr_file_rename(const char *from_path,
> const char *to_path,
> apr_pool_t *pool);
>
> The warning is not platform specific, so it should work for all.
Sounds good.
@Jan: Care to create a merged patch (my part and your part) and commit to
trunk? I guess further tuning can happen there. Or do we face this problem only
in 2.2.x?
In this case a merged backport proposal patch would be great :-)
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #16 from Jim Riggs <ji...@riggs.me> ---
(In reply to Jim Riggs from comment #15)
> I don't understand why you are seeing different results. I just ran a test
> with your configuration (except I set max-age=10). I then ran `ab -k -c 2 -t
> 30' (30-second test), and only 3 requests went to the backend as expected.
And that was 50,000 requests. 3 went to the backend.
I do notice, however, that when we serve stale content we are not setting an
Age header. It seems that RFC2616[1][2] requires this for 1.1: "An HTTP/1.1
server that includes a cache MUST include an Age header field in every response
generated from its own cache." So should we setting an Age header "in every
response", even when we are serving stale cache? If so, what value do we use
for the age? The real (now expired) age? Does anyone know what real-world
affect that would have downstream?
[1] http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.6
[2] http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.2.3
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #19 from macbogucki@gmail.com ---
Created attachment 31533
--> https://issues.apache.org/bugzilla/attachment.cgi?id=31533&action=edit
httpd.conf-notworkaround
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
Jim Riggs <ji...@riggs.me> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #31503|0 |1
is obsolete| |
--- Comment #7 from Jim Riggs <ji...@riggs.me> ---
Created attachment 31504
--> https://issues.apache.org/bugzilla/attachment.cgi?id=31504&action=edit
Covener's patch
This is the covener's patch posted to dev@ that is only slightly different from
mine, but closer to trunk/2.4.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #35 from Ruediger Pluem <rp...@apache.org> ---
(In reply to jkaluza from comment #34)
> OK, any idea what to do about it or do you think it's OK? This leads to two
> requests being passed to the backend server instead of one caused by
> multiple (in this case two) keys used for locking.
>
> If the key used in cache providers differs from the original one, shouldn't
> we generate the fresh key everytime when creating the lock to have
> consistent key across all ap_cache_try_lock(...) calls (basically what my
> change does)?
>
> Or do you think this is OK, because we get most likely (at least that's how
> I understand it) just maximally 2 requests to the backend server (one with
> original key and one with the key from cache-provider)?
I think 2 requests would be ok, but possibly we find out why we have two
different keys anyway, because after opening the cache entity we should only
have the one from the cache provider. How does this behave on trunk / 2.4.x?
Currently no time to dig further.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #47 from Ruediger Pluem <rp...@apache.org> ---
(In reply to jkaluza from comment #46)
> I have good results with following patch (on top of the patch in Comment 41):
>
> diff --git a/modules/cache/mod_disk_cache.c b/modules/cache/mod_disk_cache.c
> index 13d6c8b..305fa6b 100644
> --- a/modules/cache/mod_disk_cache.c
> +++ b/modules/cache/mod_disk_cache.c
> @@ -962,15 +962,6 @@ static apr_status_t store_headers(cache_handle_t *h,
> request_rec *r, cache_info
>
> apr_file_close(dobj->hfd); /* flush and close */
>
> - /* Remove old file with the same name. If remove fails, then
> - * perhaps we need to create the directory tree where we are
> - * about to write the new headers file.
> - */
> - rv = apr_file_remove(dobj->hdrsfile, r->pool);
> - if (rv != APR_SUCCESS) {
> - mkdir_structure(conf, dobj->hdrsfile, r->pool);
> - }
> -
> rv = safe_file_rename(conf, dobj->tempfile, dobj->hdrsfile, r->pool);
> if (rv != APR_SUCCESS) {
> ap_log_error(APLOG_MARK, APLOG_WARNING, rv, r->server,
>
> I think the race condition exists between this apr_file_remove and
> safe_file_rename. There is no cached file between these two calls.
> safe_file_rename should overwrite any existing file with the same name, so
> there's no need for apr_file_remove imho.
Agree for Linux (possibly all UNIX systems). Looking at the APR Windows code I
think that this should work as well (having an atomic operation if the target
file already exists and causing it to be replaced by the source file).
Can someone of the Windows guys please crosscheck?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #25 from Yann Ylavic <yl...@gmail.com> ---
(In reply to Yann Ylavic from comment #23)
> Just to know whether the Thundering Herd fix is misbehaving or the "-gzip"
> appended ETag is the issue.
Sorry my bad, mod_deflate does not append "-gzip" to ETag since 2.2.12, please
forget my comments.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #11 from macbogucki@gmail.com ---
Created attachment 31522
--> https://issues.apache.org/bugzilla/attachment.cgi?id=31522&action=edit
access_log
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #42 from jkaluza@redhat.com ---
With this patch it generates proper key in most situations, but when the
problem happens, I still see "NULL" key in cache_url_handler(). I will attach
log from that.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #50 from jkaluza@redhat.com ---
In trunk, only patch from Comment 41 is needed. I've committed it in r1597533.
For 2.2.x, I will create proposal for merged "Covener's patch" from this bug +
the one from Comment 41 + my one. Does it sound OK?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #33 from Ruediger Pluem <rp...@apache.org> ---
(In reply to jkaluza from comment #31)
> After more digging, it looks like problem is with key generated for locking
> in ap_cache_try_lock():
>
> Some requests come with
> key="http://localhost:80/examples/jsp/jsp2/el/basic-arithmetic.jsp?"
>
> While another come with
> key="accept-encodinghttp://localhost:80/examples/jsp/jsp2/el/basic-
> arithmetic.jsp?"
>
> Any ideas where that "accept-encoding" comes from?
This is because the cache entry varies on the accept-encoding header and the
original key gets regenerated by regen_key in mod_disk_cache.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #48 from Yann Ylavic <yl...@gmail.com> ---
apr/include.apr_file_io.h :
/**
* Rename the specified file.
* [...]
* @warning If a file exists at the new location, then it will be
* overwritten. [...]
*/
APR_DECLARE(apr_status_t) apr_file_rename(const char *from_path,
const char *to_path,
apr_pool_t *pool);
The warning is not platform specific, so it should work for all.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 50317] Possible error in mod_cache's "Avoiding the
Thundering Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #1 from Rein Tollevik <re...@basefarm.no> 2011-09-28 11:50:51 UTC ---
The problem reported here appear to be fixed in cache_storage.c revision
1023398, which is in trunk and 2.3. It should imo be backported to 2.2, as
thundering herd protection now effectively requires either Last-Modified or
ETag headers in the cached response. Headers which are seldom present in
dynamically generated content, which happen to be exactly the type of content I
find it useful to cache :-( I found this report while investigating why the
thundering herd protection didn't work as expected.
The comment about the cache lock in the bug report appear to be bogus. The
second call to ap_cache_try_lock is for the case when there are no cached
content. The lock is only acquired by ap_cache_check_freshness when it finds
stale content in the cache. ap_cache_try_lock will generate the cache key based
on the requested URL when its third argument is NULL, and it will notice that
the lock is already held if called for a second time for the same request.
--
Rein Tollevik
Basefarm AS
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #24 from Yann Ylavic <yl...@gmail.com> ---
(In reply to Yann Ylavic from comment #23)
> Can you still reproduce without DEFLATE or by applying r1586542 and using
> the suitable DeflateAlterETag value (NoChange/Suppress)?
Bug 45023 is where this commit is proposed.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #29 from jkaluza@redhat.com ---
Ok, when reproducing it, I see following in tomcat log:
::1 - - [21/May/2014:03:06:05 -0400] "GET
/examples/jsp/jsp2/el/basic-arithmetic.jsp HTTP/1.0" 200 2152
::1 - - [21/May/2014:03:06:05 -0400] "GET
/examples/jsp/jsp2/el/basic-arithmetic.jsp HTTP/1.0" 200 2152
and following is relevant part of httpd error_log with patch from Comment 7:
[Wed May 21 03:06:05 2014] [debug] cache_util.c(596): Cache lock obtained for
stale cached URL, revalidating entry:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] cache_storage.c(272): Cached response for
/examples/jsp/jsp2/el/basic-arithmetic.jsp isn't fresh. Adding/replacing
conditional request headers.
[Wed May 21 03:06:05 2014] [debug] cache_util.c(604): Cache already locked for
stale cached URL, pretend it is fresh:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] cache_util.c(604): Cache already locked for
stale cached URL, pretend it is fresh:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] cache_util.c(604): Cache already locked for
stale cached URL, pretend it is fresh:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] cache_util.c(604): Cache already locked for
stale cached URL, pretend it is fresh:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] cache_util.c(604): Cache already locked for
stale cached URL, pretend it is fresh:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] cache_util.c(604): Cache already locked for
stale cached URL, pretend it is fresh:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] cache_util.c(604): Cache already locked for
stale cached URL, pretend it is fresh:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] cache_util.c(604): Cache already locked for
stale cached URL, pretend it is fresh:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] cache_util.c(604): Cache already locked for
stale cached URL, pretend it is fresh:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] cache_util.c(604): Cache already locked for
stale cached URL, pretend it is fresh:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] cache_util.c(604): Cache already locked for
stale cached URL, pretend it is fresh:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] cache_util.c(604): Cache already locked for
stale cached URL, pretend it is fresh:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] cache_util.c(604): Cache already locked for
stale cached URL, pretend it is fresh:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] cache_util.c(604): Cache already locked for
stale cached URL, pretend it is fresh:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] mod_cache.c(767): cache: Caching url:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] mod_cache.c(990): cache: proactively
removing the lock #2
[Wed May 21 03:06:05 2014] [debug] cache_util.c(310): Cache lock removed for
stale cached URL, /examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] mod_cache.c(404): cache: proactively
removing the lock #1
[Wed May 21 03:06:05 2014] [debug] mod_cache.c(767): cache: Caching url:
/examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] mod_cache.c(990): cache: proactively
removing the lock #2
[Wed May 21 03:06:05 2014] [debug] cache_util.c(310): Cache lock removed for
stale cached URL, /examples/jsp/jsp2/el/basic-arithmetic.jsp
[Wed May 21 03:06:05 2014] [debug] mod_cache.c(404): cache: proactively
removing the lock #1
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #37 from Ruediger Pluem <rp...@apache.org> ---
Does the patch below fix the two keys issue?
Index: mod_cache.c
===================================================================
--- mod_cache.c (revision 1595827)
+++ mod_cache.c (working copy)
@@ -113,7 +113,19 @@
if (rv != OK) {
if (rv == DECLINED) {
if (!lookup) {
+ char *key;
+ /*
+ * Try to use the key of a possible open but stall cache
+ * entry if we have one.
+ */
+ if ((cache->handle != NULL) &&
+ (cache->handle->cache_obj != NULL)) {
+ key = cache->handle->cache_obj->key;
+ }
+ else {
+ key = NULL;
+ }
/* try to obtain a cache lock at this point. if we succeed,
* we are the first to try and cache this url. if we fail,
* it means someone else is already trying to cache this
@@ -121,7 +133,7 @@
* backend without any attempt to cache. this stops
* duplicated simultaneous attempts to cache an entity.
*/
- rv = ap_cache_try_lock(conf, r, NULL);
+ rv = ap_cache_try_lock(conf, r, key);
if (APR_SUCCESS == rv) {
/*
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #12 from macbogucki@gmail.com ---
Created attachment 31523
--> https://issues.apache.org/bugzilla/attachment.cgi?id=31523&action=edit
error_log
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
Jim Riggs <ji...@riggs.me> changed:
What |Removed |Added
----------------------------------------------------------------------------
Attachment #26333|0 |1
is obsolete| |
--- Comment #6 from Jim Riggs <ji...@riggs.me> ---
Created attachment 31503
--> https://issues.apache.org/bugzilla/attachment.cgi?id=31503&action=edit
2.2 thundering herd fix
The attached patched was discussed and looked at by myself, covener, and mrumph
today at ApacheCon. This fixes the thundering herd problem in 2.2. It also
addresses r572626 (https://issues.apache.org/bugzilla/show_bug.cgi?id=30370)
that created this regression. So, this patch fixes both the original problem in
30370 as well as this issue. It applies only to 2.2; >= 2.3 was fixed in
r1023398.
To summarize: without this patch, 2.2 has NO protection from the thundering
herd, as the stale content is removed before being refreshed. Thus, all
subsequent herd requests have no stale content to serve and flow through to the
backend.
Should the documentation be updated also to alert users to this? That is that
"2.2.x before 2.2.__ is not protected from the thundering herd"?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
macbogucki@gmail.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Version|2.2.26 |2.2.27
--- Comment #8 from macbogucki@gmail.com ---
Thank You very much for the patch but it doesn't works. When I'm doing ab
(/usr/bin/ab -k -c 5 -n 10 http://host/url) test the application get more than
one request
1.1.1.1 - - [14/Apr/2014:14:01:58 +0200] "GET /url HTTP/1.0" 200 42398
9A68DBA96CED90DC517F7D6302F5A748.gpi-app1 1163 1163
1.1.1.1 - - [14/Apr/2014:14:02:05 +0200] "GET /url HTTP/1.0" 200 42398
D378685BBD4FB87C63A3A867ABFAFB3E.gpi-app1 2931 2930
1.1.1.1 - - [14/Apr/2014:14:02:05 +0200] "GET /url HTTP/1.0" 200 42398
8B77A0C68FC6F16E0BA3A89C7A614E1A.gpi-app1 2992 2991
1.1.1.1 - - [14/Apr/2014:14:02:05 +0200] "GET /url HTTP/1.0" 200 42398
57A48B49FB6C52E28F1FA97DDFCDC0C8.gpi-app1 3007 3006
1.1.1.1 - - [14/Apr/2014:14:02:05 +0200] "GET /url HTTP/1.0" 200 42398
71573080388181B3C55E88CB4BFAB890.gpi-app1 3051 3051
1.1.1.1 - - [14/Apr/2014:14:02:06 +0200] "GET /url HTTP/1.0" 200 42398
38DA8533D4F9B4046A2F607071652E94.gpi-app1 1412 1412
Here are more information how to reproduce it.
Compilation
cd /tmp
svn co http://svn.apache.org/repos/asf/httpd/httpd/branches/2.2.x
cd 2.2.x/
svn co http://svn.apache.org/repos/asf/apr/apr/branches/1.4.x srclib/apr
svn co http://svn.apache.org/repos/asf/apr/apr-util/branches/1.4.x
srclib/apr-util
./buildconf
./configure --prefix=/etc/httpd --exec-prefix=/usr --bindir=/usr/bin
--sbindir=/usr/sbin --mandir=/usr/share/man --libdir=/usr/lib64
--sysconfdir=/etc/httpd/conf --includedir=/usr/include/httpd
--libexecdir=/usr/lib64/httpd/modules --datadir=/var/www
--with-installbuilddir=/usr/lib64/httpd/build --with-mpm=prefork
--with-apr=/usr --with-apr-util=/usr --enable-suexec --with-suexec
--with-suexec-caller=apache --with-suexec-docroot=/var/www
--with-suexec-logfile=/var/log/httpd/suexec.log
--with-suexec-bin=/usr/sbin/suexec --with-suexec-uidmin=500
--with-suexec-gidmin=100 --enable-pie --with-pcre
--enable-mods-shared=all --enable-ssl --with-ssl --enable-proxy
--enable-cache --enable-disk-cache --enable-ldap --enable-authnz-ldap
--enable-cgid --enable-authn-anon --enable-authn-alias
--disable-imagemap
patch -p0 < /root/rpmbuild/SOURCES/httpd-2.2.x-thunder.patch
make
make install
Configuration
<VirtualHost host:80>
...
...
## Cache
CacheRoot /tmp/cache
CacheEnable disk /
CacheDisable /static/
CacheMinFileSize 0
CacheMaxFileSize 1048576
CacheDirLevels 2
CacheDirLength 2
CacheLock on
CacheLockPath /tmp/mod_cache-lock
CacheLockMaxAge 5
CacheIgnoreHeaders ETag Set-Cookie
Header unset Expires
Header unset Cache-Control
Header always set Cache-Control "max-age=30,stale-while-revalidate=15"
</VirtualHost>
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #22 from macbogucki@gmail.com ---
I have just attached 3 files
* httpd.conf-bad - use this file to reproduce the problem
* httpd.conf-notworkaround - 2 vhosts (there is a problem with such a
configuration)
** first with mod_cache and proxy to the second
** second with deflate and proxy to the app server
* httpd.conf-workaround - 2 vhosts (it a workaround for my problem)
** first with deflate and proxy to the second
** second with mod_cache and proxy to the app server
The problem is when I combine mod_proxy, mod_deflate, mod_cache
* When I disable mod_deflate it works
* When I serve the content from local apache then it works (there is no
mod_proxy in such a case)
I have also tried to use http proxy instead of ajp but without success.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
macbogucki@gmail.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |NEW
--- Comment #14 from macbogucki@gmail.com ---
I have just attached 3 files
* access_log
* error_log
* tomcat_log (application server log)
* At 07:31:40 I ran the command '/usr/bin/GET -des http://apache-app1/url'
** There is nothing in the cache so the response was inserted in the cache
* At 07:32:25 I ran the command '/usr/bin/ab -k -c 2 -n 4
http://apache-app1/url'
** All the responses was served from the cache
* At 07:32:37 I ran the command '/usr/bin/ab -k -c 2 -n 4
http://apache-app1/url'
** 3 responses was served from the cache but 3 was serverd from the application
server.
I don't understand why only 3 responses were served from the cache in the third
case.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #17 from Eric Covener <co...@gmail.com> ---
> Covener - What are you seeing?
I can't get it to misbehave, i trigger a slow refresh and hammer it with ab. I
do see the Age: quirk where it seems to set the Age relative to the refresh
time rather than the real age.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #46 from jkaluza@redhat.com ---
I have good results with following patch (on top of the patch in Comment 41):
diff --git a/modules/cache/mod_disk_cache.c b/modules/cache/mod_disk_cache.c
index 13d6c8b..305fa6b 100644
--- a/modules/cache/mod_disk_cache.c
+++ b/modules/cache/mod_disk_cache.c
@@ -962,15 +962,6 @@ static apr_status_t store_headers(cache_handle_t *h,
request_rec *r, cache_info
apr_file_close(dobj->hfd); /* flush and close */
- /* Remove old file with the same name. If remove fails, then
- * perhaps we need to create the directory tree where we are
- * about to write the new headers file.
- */
- rv = apr_file_remove(dobj->hdrsfile, r->pool);
- if (rv != APR_SUCCESS) {
- mkdir_structure(conf, dobj->hdrsfile, r->pool);
- }
-
rv = safe_file_rename(conf, dobj->tempfile, dobj->hdrsfile, r->pool);
if (rv != APR_SUCCESS) {
ap_log_error(APLOG_MARK, APLOG_WARNING, rv, r->server,
I think the race condition exists between this apr_file_remove and
safe_file_rename. There is no cached file between these two calls.
safe_file_rename should overwrite any existing file with the same name, so
there's no need for apr_file_remove imho.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #10 from Jim Riggs <ji...@riggs.me> ---
Is this test with a primed cache? The first time through (when there is nothing
in the cache), the thundering herd always goes through, so you SHOULD see
multiple requests on the backend. (There is nothing stale to serve.)
Once the cache has content, though, stale content will get served while a
refresh/revalidate request is processing.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #45 from Ruediger Pluem <rp...@apache.org> ---
(In reply to jkaluza from comment #44)
> From what we see I think this happens:
>
> 1. Request is updating in a cache and therefore it's not returned by
> mod_disk_cache. Lock with proper name (the one with accept-encoding is used
> to guard that state).
> 2. New request arrives, it is not found in a cache and therefore the proper
The question IMHO is: Why is it not found in the cache? It should be. I guess
this needs some further digging.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #43 from jkaluza@redhat.com ---
Created attachment 31652
--> https://issues.apache.org/bugzilla/attachment.cgi?id=31652&action=edit
Second request not found in stale cache right after being cached
Check "cache: cache_url_handler uses key" error log lines. The first one (line
9) shows proper key, the second one (line 68) is NULL.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #2 from piotrglow@gmail.com ---
Hi,
I am using apache 2.2.26 and I can see that thundering herd protection does not
work at all. I encountered the same sympthoms as described by Julius.
It seems that fix is only present in trunk and was not merged to 2.2.x branch.
Are there any plans to fix thundering herd effect in 2.2.x by merging the fix?
Br,
Piotr G
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 50317] Possible error in mod_cache's "Avoiding the
Thundering Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
Julius Gehr <ju...@neofonie.de> changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |jules@neofonie.de
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #31 from jkaluza@redhat.com ---
After more digging, it looks like problem is with key generated for locking in
ap_cache_try_lock():
Some requests come with
key="http://localhost:80/examples/jsp/jsp2/el/basic-arithmetic.jsp?"
While another come with
key="accept-encodinghttp://localhost:80/examples/jsp/jsp2/el/basic-arithmetic.jsp?"
Any ideas where that "accept-encoding" comes from?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #21 from Yann Ylavic <yl...@gmail.com> ---
Couldn't this be that confs from comment #18 and comment #19 hit the "-gzip"
ETag issue of bug 39727, whereas conf from comment #20 does not?
Can you still reproduce without DEFLATE or by applying r1586542 and using the
suitable DeflateAlterETag value (NoChange/Suppress)?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #34 from jkaluza@redhat.com ---
OK, any idea what to do about it or do you think it's OK? This leads to two
requests being passed to the backend server instead of one caused by multiple
(in this case two) keys used for locking.
If the key used in cache providers differs from the original one, shouldn't we
generate the fresh key everytime when creating the lock to have consistent key
across all ap_cache_try_lock(...) calls (basically what my change does)?
Or do you think this is OK, because we get most likely (at least that's how I
understand it) just maximally 2 requests to the backend server (one with
original key and one with the key from cache-provider)?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #38 from jkaluza@redhat.com ---
No, it doesn't. "key" is still NULL even when "accept-encodingURL" is locked,
so ap_cache_try_lock(...) generates new key without "accept-encoding" leading
to second lock and second request to backend.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
Eric Covener <co...@gmail.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Version|2.0-HEAD |2.2.17
--- Comment #4 from Eric Covener <co...@gmail.com> ---
restore version so it doesn't look like a 2.0 bug.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #44 from jkaluza@redhat.com ---
>From what we see I think this happens:
1. Request is updating in a cache and therefore it's not returned by
mod_disk_cache. Lock with proper name (the one with accept-encoding is used to
guard that state).
2. New request arrives, it is not found in a cache and therefore the proper key
name (with accept-encoding) can't be obtained. Key without "accept-encoding" is
used and second request is sent to backend.
If this really happens, I don't see a way how to get the proper key name
without:
a) getting the regen_key output in cache_url_handler (which is not easy or
maybe even possible)
b) or locking with the default key (without accept-encoding) in cache_util.c
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #36 from jkaluza@redhat.com ---
The problem is that key is generated without opening cached entity in this
place:
http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/cache/mod_cache.c?revision=1497099&view=markup#l124
So in this case "accept-encoding" is not in URL, but in the following case,
cached entity is opened and accept-encoding is present:
svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/modules/cache/cache_util.c?revision=1053600&view=markup#l648
In httpd-2.4, the code has been changed a bit in svn.apache.org/r1004220, but
as far as I understand it, the logic remains the same (so the problem could
exist in 2.4.x too). I will try to reproduce it in 2.4.x to be 100% sure.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #18 from macbogucki@gmail.com ---
Created attachment 31532
--> https://issues.apache.org/bugzilla/attachment.cgi?id=31532&action=edit
httpd.conf-bad - use this to reproduce
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #27 from jkaluza@redhat.com ---
I was able to reproduce this bug using config from Comment 18 and the patch
from Comment 7 fixed the issue for me. Is that what others see too?
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
macbogucki@gmail.com changed:
What |Removed |Added
----------------------------------------------------------------------------
Version|2.2.17 |2.0-HEAD
--- Comment #3 from macbogucki@gmail.com ---
Hello,
Is there any chance to fix this error?
Best regards
Maciej Bogucki
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #32 from jkaluza@redhat.com ---
Following change fixes the issue completely for me, but it's not the fix for
the root of this problem. However, it could give idea to someone with better
knowledge to find out what's going on:
diff --git a/modules/cache/cache_util.c b/modules/cache/cache_util.c
index cb3d565..84edcd2 100644
--- a/modules/cache/cache_util.c
+++ b/modules/cache/cache_util.c
@@ -585,7 +598,7 @@ CACHE_DECLARE(int) ap_cache_check_freshness(cache_handle_t
*h,
* A lock that exceeds a maximum age will be deleted, and another
* request gets to make a new lock and try again.
*/
- status = ap_cache_try_lock(conf, r, (char *)h->cache_obj->key);
+ status = ap_cache_try_lock(conf, r, NULL);
if (APR_SUCCESS == status) {
/* we obtained a lock, follow the stale path */
ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, r->server,
h->cache_obj->key is
"accept-encodinghttp://localhost:80/examples/jsp/jsp2/el/basic-arithmetic.jsp?"
while newly generated key is
"http://localhost:80/examples/jsp/jsp2/el/basic-arithmetic.jsp?".
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 50317] Possible error in mod_cache's "Avoiding the Thundering
Herd"
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=50317
--- Comment #20 from macbogucki@gmail.com ---
Created attachment 31534
--> https://issues.apache.org/bugzilla/attachment.cgi?id=31534&action=edit
httpd.conf-workaround
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org