You are viewing a plain text version of this content. The canonical link for it is here.
Posted to bugs@httpd.apache.org by bu...@apache.org on 2008/04/04 15:16:23 UTC
DO NOT REPLY [Bug 44755] New: Piped logs use two file descriptors
per logfile (was 1:
1 with previous versions)
https://issues.apache.org/bugzilla/show_bug.cgi?id=44755
Summary: Piped logs use two file descriptors per logfile (was 1:1
with previous versions)
Product: Apache httpd-2
Version: 2.2.8
Platform: Sun
OS/Version: Solaris
Status: NEW
Severity: normal
Priority: P2
Component: Core
AssignedTo: bugs@httpd.apache.org
ReportedBy: owen.boyle@swx.com
Created an attachment (id=21781)
--> (https://issues.apache.org/bugzilla/attachment.cgi?id=21781)
Gzipped TAR containing pfiles and truss output demonstrating bug
When pipes to logfiles are used (eg, rotatelogs), apache 2.2.8 uses two file
descriptors for each log file, as compared to one fd per file for previous
versions (ie, 2.2.6 and below). This has consequences for servers with a large
number of logfiles because the file descriptor limit (eg, 255 on Solaris) may
be suddenly breached simply by upgrading and with no other changes to the
config.
To reproduce, set up a minimal config file with logging via pipes, eg:
TransferLog "|exec bin/rotatelogs logs/access_log 604800"
ErrorLog "|exec bin/rotatelogs logs/error_log 604800"
Run a server as version 2.2.6 and another as version 2.2.8 and compare the
"pfiles" output for each process. You will find that the 2.2.8 process uses an
extra two file descriptors. This scales with the number of logfiles - for
example, with 10 logs, 2.2.8 uses an extra 10 FDs relative to 2.2.6.
Possibly related to recent commit:
http://svn.apache.org/viewvc/httpd/httpd/tags/2.2.8/server/log.c?view=diff&r1=602466&r2=602467
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 44755] Piped logs use two file descriptors per
logfile (
was 1:1 with previous versions)
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=44755
--- Comment #9 from Owen Boyle <ow...@swx.com> 2008-04-25 05:37:15 PST ---
Created an attachment (id=21852)
--> (https://issues.apache.org/bugzilla/attachment.cgi?id=21852)
lsof output for various cases of logging
lsof output for various apache versions against various configs that
demonstrate how different number of pipes are used in each case.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 44755] Piped logs use two file descriptors per
logfile (
was 1:1 with previous versions)
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=44755
--- Comment #11 from Will Rowe <wr...@apache.org> 2008-06-20 07:29:29 PST ---
Owen, you tested with apr 1.3?
Note there was a change also to the way pipe inheritance works, and the
apr 1.3 is in sync with httpd 2.2.9, thus the change you observed, I think.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 44755] Piped logs use two file descriptors per
logfile (
was 1:1 with previous versions)
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=44755
--- Comment #3 from Owen Boyle <ow...@swx.com> 2008-04-14 05:17:54 PST ---
I applied the patch to a test case and can confirm that the number of pipes
opened is equal to that in version 2.2.6. So the patch looks good :-)
I will make a more comprehensive test to verify that all features of logging
are working and report back in a day or two.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 44755] Piped logs use two file descriptors per
logfile (
was 1:1 with previous versions)
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=44755
--- Comment #6 from Will Rowe <wr...@apache.org> 2008-04-24 09:23:14 PST ---
Owen, keep in mind that 70 handles are required simply for the output pipe,
which you already knew. In order to recover gracefully from a stopped logger,
the 70 read-ends of the pipes are required to pass this to a replacement
process.
That's 140 in your case. The other 37 handles? I don't know... but clearly
the
original 107 handles you cite were incorrect.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 44755] Piped logs use two file descriptors per
logfile (
was 1:1 with previous versions)
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=44755
Ruediger Pluem <rp...@apache.org> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |NEEDINFO
--- Comment #7 from Ruediger Pluem <rp...@apache.org> 2008-04-24 12:39:49 PST ---
(In reply to comment #6)
> Owen, keep in mind that 70 handles are required simply for the output pipe,
> which you already knew. In order to recover gracefully from a stopped logger,
> the 70 read-ends of the pipes are required to pass this to a replacement
We only keep the read ends for the CustomLogs / Transferlogs. We do not do the
same for the error logs. Provided the 70 piped loggers are equally distributed
over CustomLogs / Transferlogs and ErrorLogs this could explain Owens numbers
for 2.2.6:
70 write ends
35 read ends
---
105 fds in total.
The remaining four might be for other purposes (e.g. pipe of death). Of course
this applies only to the main process and not to its childs.
For 2.2.8 with the patches applied this would be
70 write ends
35 read ends
4 don't knows as in 2.2.6
68 additional ones we don't know.
Owen, could you please use lsof which is also available for Solaris and attach
the result filtered for the main process. This may help us a little bit if 68
unknown fd's are read or write end of pipes.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 44755] Piped logs use two file descriptors per
logfile (
was 1:1 with previous versions)
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=44755
Owen Boyle <ow...@swx.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|ASSIGNED |RESOLVED
Resolution| |FIXED
--- Comment #10 from Owen Boyle <ow...@swx.com> 2008-06-20 06:40:13 PST ---
I have just tested the recently-released 2.2.9 for this bug and can confirm
that it is fixed in this version.
Note that there are no changes to server/log.c between 2.2.8 and 2.2.9.
However, there has been quite a bit of work done in support/rotatelogs.c so it
looks like that was where the fix was done.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 44755] Piped logs use two file descriptors per
logfile (
was 1:1 with previous versions)
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=44755
--- Comment #1 from Joe Orton <jo...@redhat.com> 2008-04-11 01:54:20 PST ---
I've been comparing 2.2.x server/log.c with 2.2.6-level log.c:
For a config with two vhosts each with a piped logger, the 2.2.x code has no
less than SIX pipe fds open in the parent.
The 2.2.x code has two - expected behaviour.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 44755] Piped logs use two file descriptors per
logfile (
was 1:1 with previous versions)
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=44755
--- Comment #4 from Ruediger Pluem <rp...@apache.org> 2008-04-14 14:34:30 PST ---
(In reply to comment #2)
> Created an attachment (id=21815)
--> (https://issues.apache.org/bugzilla/attachment.cgi?id=21815) [details]
> fix for 2.2.8 piped logger fd leaks
>
> I've attached a patch which should fix this; a combined effort from Ruediger
> and I. Can you try this out?
>
Thanks for putting the patch here Joe. Actually the the call to
apr_file_inherit_unset is not directly linked to this report as it is no
regression. After some thoughts I think the following place would be better for
apr_file_inherit_unset as it would also solve the issue for all callers of
ap_open_piped_log (e.g. the access logs). Furthermore I guess this would also
improve security as CGI scripts would no longer inherit these file descriptors
and thus would be no longer able to write directly to the access logs.
Index: server/log.c
===================================================================
--- server/log.c (Revision 647395)
+++ server/log.c (Arbeitskopie)
@@ -307,6 +307,7 @@
if (rc == APR_SUCCESS) {
apr_pool_note_subprocess(p, procnew, APR_KILL_AFTER_TIMEOUT);
(*fpin) = procnew->in;
+ apr_file_inherit_unset(*fpin);
/* read handle to pipe not kept open, so no need to call
* close_handle_in_child()
*/
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 44755] Piped logs use two file descriptors per
logfile (
was 1:1 with previous versions)
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=44755
Owen Boyle <ow...@swx.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEEDINFO |ASSIGNED
--- Comment #8 from Owen Boyle <ow...@swx.com> 2008-04-25 05:27:04 PST ---
I was able to reproduce the various differences on a test set-up and can report
the following number of pipes used (nb, using "lsof -p `cat httpdTest.pid` |
grep PIPE | wc" to count the pipes):
(This table might not format properly after posting... :-)
| 2.2.6 | 2.2.8 unpatched | 2.2.8 patched |
| 7 | 9 | 7 | T+E logs in Main (2 logs)
| 5 | 9 | 7 | T+E logs in 1 VH (2 logs)
| 7 | 13 | 11 | T+E logs in 2 VH (4 logs)
Observations:
1) In 2.2.6, the number of pipes used in each case depends on whether the log
directives are in a VirtualHost or not. Adding a second VH (with 1 TransferLog
and 1 Errorlog) adds 2 pipes.
2) Apache 2.2.8, does not have a dependency on log directive location. However,
the number of pipes starts at the higher level compared to 2.2.6. Adding a
second VH adds 4 pipes.
3) Applying the patch to 2.2.8, in the simplest case (logs only in main config)
reduces the number of pipes to the same as 2.2.6 (this is why I originally
thought the patch was successful). However, in the case when the logs are in
the VHs, there are two extra pipes. Adding a second VH adds 4 pipes.
Conclusions:
Adding a VH with 1 TransferLog and 1 Errorlog to a 2.2.6 config adds 2 pipes to
the process. With 2.2.8 (patched or unpatched), each VH adds 4 extra pipes.
However, the patch does reduce the absolute number of pipes by two. So the bug
is still extant, it seems.
I don't know what to make of the directive location dependency in 2.2.6!
PS - Attached is a tar file containing the lsof output for each case mentioned
in the table.
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 44755] Piped logs use two file descriptors per
logfile (
was 1:1 with previous versions)
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=44755
--- Comment #2 from Joe Orton <jo...@redhat.com> 2008-04-11 03:10:58 PST ---
Created an attachment (id=21815)
--> (https://issues.apache.org/bugzilla/attachment.cgi?id=21815)
fix for 2.2.8 piped logger fd leaks
I've attached a patch which should fix this; a combined effort from Ruediger
and I. Can you try this out?
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 44755] Piped logs use two file descriptors per
logfile (
was 1:1 with previous versions)
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=44755
--- Comment #12 from Owen Boyle <ow...@swx.com> 2008-06-23 00:21:33 PST ---
Hi Will,
I always compile "with-included-apr" so, yes, httpd-2.2.9 is using apr-1.3.0.
I assume the implementation of the new function, apr_file_pipe_create_ex(), in
pipe.c and the refactoring in proc.c (where some case-statement logic has been
replaced by if-blocks) is what has done the trick.
So it looks like this bug is definitively resolved and thank you very much for
your assistance!
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org
DO NOT REPLY [Bug 44755] Piped logs use two file descriptors per
logfile (
was 1:1 with previous versions)
Posted by bu...@apache.org.
https://issues.apache.org/bugzilla/show_bug.cgi?id=44755
--- Comment #5 from Owen Boyle <ow...@swx.com> 2008-04-24 07:46:15 PST ---
I have tested various patches against a production config with 70 piped logs
and have to say the the problem has improved but is still present. For each
mix, I counted N, the number of S_IFIFO lines in the pfiles output (NB this is
Solaris 10). What I find, with identical config, is:
1) Version 2.2.6, N = 109
2) Version 2.2.8, unpatched, N = 247
3) Version 2.2.8, with original patch of Joe Orton, N = 177
4) Version 2.2.8, with patch of Orton and Pluem, N = 177
What seems to happen is, for M piped logs, the unpatched 2.2.8 has 2M-2 more
file descriptors than the reference 2.2.6. With either of the patches, the
excess goes down to M-2.
Ignoring the -2 term (probably because I have two pipes that are outside of
VirtualHost containers), it looks like the patches fix one leak of FDs.
However, there must be another leak somewhere...
--
Configure bugmail: https://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: bugs-unsubscribe@httpd.apache.org
For additional commands, e-mail: bugs-help@httpd.apache.org