You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2014/09/12 18:52:31 UTC
[Bug 56977] New: segfaults when using mod_mem_cache + mod_disk_cache
on a reverse proxy with mod_proxy + mod_proxy_http
https://issues.apache.org/bugzilla/show_bug.cgi?id=56977
Bug ID: 56977
Summary: segfaults when using mod_mem_cache + mod_disk_cache on
a reverse proxy with mod_proxy + mod_proxy_http
Product: Apache httpd-2
Version: 2.2.22
Hardware: PC
OS: Linux
Status: NEW
Severity: major
Priority: P2
Component: mod_mem_cache
Assignee: bugs@httpd.apache.org
Reporter: bpkroth@gmail.com
Created attachment 32013
--> https://issues.apache.org/bugzilla/attachment.cgi?id=32013&action=edit
A test PHP file to help in reproducing the crash.
We have an application (Moodle 2.6) that switches from sending "Cache-Control:
public" to "Cache-Control: private" headers for the same content depending upon
whether or not the user is logged in.
That application is served via a reverse proxy using mod_proxy/mod_proxy_http
to backends running mod_php5/mod_xsendfile (among others).
Traditionally we've also allowed the use of mod_disk_cache on the reverse
proxy.
Recently we also added mod_mem_cache *before* the mod_disk_cache confs so that
highly requested content could be served more rapidly from the in memory cache:
<IfModule mod_mem_cache.c>
CacheEnable mem /
</IfModule>
<IfModule mod_disk_cache.c>
CacheEnable disk /
</IfModule>
It was discovered that for that content that switches between public and
private Cache-Control headers, when the request is made for content that is in
the cache (originally via public Cache-Control headers), but has expired so
that a revalidation request is required, then the Apache process will segfault.
Based on the backtrace from the coredump, it appears that the segfault is while
mod_mem_cache is attempting to remove the now stale (since it now has
Cache-Control: private) entry from the cache.
If we remove mod_mem_cache from the mix, then this does not occurr.
mod_disk_cache appropriately stores, serves, and then removes the cache entry
as necessary.
Attached are a test php script (proxycaching-index.php) and proxy coredump
backtrace output (reverse-proxy-backtrace).
It was also reported to me, though I could not reproduce this other error case,
that aside from segfaults, sometimes the mod_mem_cache configuration would just
return "garbage data". My guess would be that it returned partial data or data
from an alternative Vary.
Also, I wasn't able to reliably reproduce this bug with content <
MCacheMaxObjectSize or without the "Accept-Ranges: bytes" header. I suspect
that those aspects are also tied up in the issue.
So, to test this I did the following:
- setup a reverse proxy with mod_mem_cache and mod_disk_cache (in that order).
- place proxycaching-index.php in /proxycaching/ of a vhost. Make sure
$private = 0 at the top of the file
- add "XSendFile On" in that backend vhost's .htacces (just a convinience for
sending the file)
- copy /usr/share/cups/data/default-testpage.pdf (or someother pdf) into
/proxycaching/default-testpage.pdf. It may or may not be important that the
size of that file is > MCacheMaxObjectSize
- run the following a few times to prime the cache in each worker mpm process
with "public content":
# curl -v -s -o /tmp/curl.out http://vhostaddress/proxycaching/index.php; file
/tmp/curl.out; ls -l /tmp/curl.out
- switch $private = 1 in the /proxycaching/index.php file
# rerun the curl command a few times
Let me know if you need any more details.
Thanks,
Brian
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 56977] segfaults when using mod_mem_cache + mod_disk_cache on a
reverse proxy with mod_proxy + mod_proxy_http
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56977
--- Comment #6 from bpkroth@gmail.com ---
(In reply to Yann Ylavic from comment #5)
> (In reply to bpkroth from comment #3)
> > (In reply to Ruediger Pluem from comment #2)
> > > Honest opinion: Don't use mod_mem_cache. It does not speed up things
> > > compared to mod_disk_cache. mod_mem_cache's cache is not shared between
> > > different httpd processes. So you waste more memory for getting less
> > > performance. Given that you have enough memory in your server mod_disk_cache
> > > content is kept in the buffer caches by the OS. If you don't use SSL stuff
> > > is send via sendfile which moves stuff from the buffer caches to the socket
> > > directly inside the kernel. If you are using SSL stuff will need to be MMAP
> > > which is still very fast.
> >
> > Yeah, I don't disagree. Under very high load there is a difference between
> > mod_mem_cache and mod_disk_cache in so far as the latter requires some extra
> > syscalls to the OS for file permissions and handles and the like, which can
> > be particularly expensive in a VM environment, but that is kind of an edge
> > case.
>
> A good alternative is also to use mod_disk_cache on a directory which a
> (mounted) ramdisk cache.
This is a little off topic from what the bug was actually about (errors in
mod_mem_cache), but I'll bite.
Someone correct me if I'm wrong, but using tmpfs/ramdisk won't avoid your file
open(), read()/sendfile(), write(), etc. syscalls from going through the OS to
access the cache files instead of just staying in the Apache process space when
doing cache lookups. I believe that context switch is what accounts for the
performance difference between mod_mem_cache and mod_disk_cache.
As Ruediger pointed out, if you have enough memory free, then the OS is already
going to do a good job of caching the dirents, inode and data blocks in the
page cache anyways, so you shouldn't be seeing any major read performance
differences between mod_disk_cache and mod_mem_cache aside from those calls to
do the lookups and get handles on the file. Even write performance to the
cache shouldn't be too bad given the OS will probably buffer that too and write
it out to disk in the background.
But I guess the only way to know for sure would be to test it :)
All that said, you don't have to convince me not to use mod_mem_cache anymore.
Consider this just a heads up that it's broken in some more edge cases.
Perhaps a warning to all future users who run across it :)
Cheers,
Brian
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 56977] segfaults when using mod_mem_cache + mod_disk_cache on a
reverse proxy with mod_proxy + mod_proxy_http
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56977
--- Comment #3 from bpkroth@gmail.com ---
(In reply to Ruediger Pluem from comment #2)
> Honest opinion: Don't use mod_mem_cache. It does not speed up things
> compared to mod_disk_cache. mod_mem_cache's cache is not shared between
> different httpd processes. So you waste more memory for getting less
> performance. Given that you have enough memory in your server mod_disk_cache
> content is kept in the buffer caches by the OS. If you don't use SSL stuff
> is send via sendfile which moves stuff from the buffer caches to the socket
> directly inside the kernel. If you are using SSL stuff will need to be MMAP
> which is still very fast.
Yeah, I don't disagree. Under very high load there is a difference between
mod_mem_cache and mod_disk_cache in so far as the latter requires some extra
syscalls to the OS for file permissions and handles and the like, which can be
particularly expensive in a VM environment, but that is kind of an edge case.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 56977] segfaults when using mod_mem_cache + mod_disk_cache on a
reverse proxy with mod_proxy + mod_proxy_http
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56977
bpkroth@gmail.com changed:
What |Removed |Added
----------------------------------------------------------------------------
CC| |bpkroth@gmail.com
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 56977] segfaults when using mod_mem_cache + mod_disk_cache on a
reverse proxy with mod_proxy + mod_proxy_http
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56977
--- Comment #1 from bpkroth@gmail.com ---
Created attachment 32014
--> https://issues.apache.org/bugzilla/attachment.cgi?id=32014&action=edit
The coredump backtrace output from the reverse proxy.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 56977] segfaults when using mod_mem_cache + mod_disk_cache on a
reverse proxy with mod_proxy + mod_proxy_http
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56977
--- Comment #2 from Ruediger Pluem <rp...@apache.org> ---
Honest opinion: Don't use mod_mem_cache. It does not speed up things compared
to mod_disk_cache. mod_mem_cache's cache is not shared between different httpd
processes. So you waste more memory for getting less performance. Given that
you have enough memory in your server mod_disk_cache content is kept in the
buffer caches by the OS. If you don't use SSL stuff is send via sendfile which
moves stuff from the buffer caches to the socket directly inside the kernel. If
you are using SSL stuff will need to be MMAP which is still very fast.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 56977] segfaults when using mod_mem_cache + mod_disk_cache on a
reverse proxy with mod_proxy + mod_proxy_http
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56977
--- Comment #5 from Yann Ylavic <yl...@gmail.com> ---
(In reply to bpkroth from comment #3)
> (In reply to Ruediger Pluem from comment #2)
> > Honest opinion: Don't use mod_mem_cache. It does not speed up things
> > compared to mod_disk_cache. mod_mem_cache's cache is not shared between
> > different httpd processes. So you waste more memory for getting less
> > performance. Given that you have enough memory in your server mod_disk_cache
> > content is kept in the buffer caches by the OS. If you don't use SSL stuff
> > is send via sendfile which moves stuff from the buffer caches to the socket
> > directly inside the kernel. If you are using SSL stuff will need to be MMAP
> > which is still very fast.
>
> Yeah, I don't disagree. Under very high load there is a difference between
> mod_mem_cache and mod_disk_cache in so far as the latter requires some extra
> syscalls to the OS for file permissions and handles and the like, which can
> be particularly expensive in a VM environment, but that is kind of an edge
> case.
A good alternative is also to use mod_disk_cache on a directory which a
(mounted) ramdisk cache.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
[Bug 56977] segfaults when using mod_mem_cache + mod_disk_cache on a
reverse proxy with mod_proxy + mod_proxy_http
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=56977
--- Comment #4 from Eric Covener <co...@gmail.com> ---
(In reply to bpkroth from comment #1)
> Created attachment 32014 [details]
> The coredump backtrace output from the reverse proxy.
I think this crash is the same as my recent question in dev@httpd thread
"mod_cache/mod_mem_cache questions" about the difference between remove_url and
remove_entity.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org