You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@httpd.apache.org by Chris Boyce <ch...@berbee.com> on 2007/11/14 20:37:03 UTC

[users@httpd] Lingering children tying up file handles

Hello... We're using Apache 2.0.59 on Solaris.  I've noticed over the
last few bug/security releases, file handles being held open on rotated
log files when using graceful restarts.

>>From what I understand, graceful restarting is the best way to do log
rotation.  But over the course of a few nightly log rotations, there are
rotated logs that never have their file handles closed, causing df to
incorrectly report a full filesystem.  The only fix is to either a) kill
the offending server process (if identifiable) or b) do a hard restart.

In the ps output below, I was able to identify the offending server (see
Nov 11 process **):

  nobody 23698  7548  0 12:26:15 ? httpd -k start
  nobody 23769  7548  0 12:27:30 ? httpd -k start
  nobody 23508  7548  0 12:24:33 ? httpd -k start
  nobody 26870  7548  0   Nov 11 ? httpd -k start **
  nobody 23704  7548  0 12:26:39 ? httpd -k start
    root  7548     1  0   Nov 08 ? httpd -k start
  nobody 25972  7548  0 18:40:32 ? httpd -k start
  nobody 23766  7548  0 12:27:10 ? httpd -k start
  nobody 23572  7548  0 12:25:00 ? httpd -k start
  nobody 23636  7548  0 12:25:08 ? httpd -k start
  nobody 23430  7548  0 12:23:29 ? httpd -k start
  nobody 23436  7548  0 12:23:34 ? httpd -k start
  nobody 23703  7548  0 12:26:37 ? httpd -k start
  nobody 23767  7548  0 12:27:18 ? httpd -k start

The Nov 11 server was three days old (and was not the parent process),
so it stuck out, and killing it "freed up" the "used" space in df.

We use the worker MPM as follows:

StartServers           40
MaxClients            350
MinSpareThreads        40
MaxSpareThreads        75
ThreadsPerChild        25
MaxRequestsPerChild 10000

I tried setting MaxRequestsPerChild in an effort to force aging children
to die, but that hasn't helped.

Can anyone explain why these rogue children are not going away over
time?  Do I simply need to stop using "graceful" in my log rotations
(though less desirable)?

Thanks for reading.


-- 
Chris Boyce



---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

RE: [users@httpd] Lingering children tying up file handles

Posted by Axel-Stephane SMORGRAV <Ax...@europe.adp.com>.

I wonder whether this could be related to a discussion on the dev list that took place around the last August-September timeframe, about the handling of file handles across restarts on different OS platforms.

You can find the thread here: http://www.mail-archive.com/dev@httpd.apache.org/msg37412.html

The following is a quote from the STATUS file for 2.0.x:

RELEASE SHOWSTOPPERS:

* core log.c: Authored and Reviewed by both rplume and wrowe within
the same 10 minutes, share only a single apr_file_t/fd between the
stderr and server_main->error_log to prevent any lingering write
handles from hanging around in unexpected ways.
http://svn.apache.org/viewvc?view=rev&revision=580437
PR 43491, solution validated by reporter
+1: wrowe, rpluem

* core log.c: Work around possible solutions rejected by apr for
the old implementation of apr_proc_create(), and explicitly pass
the output and error channels to all log processes created.
This goes all the way back to piped logs failing to run on win32.
Not in or needed at trunk/, as apr 1.3.0 has the proper fix.
http://people.apache.org/~wrowe/httpd-2.0-2.2-procattr-bugfix-log.c.patch
+1: wrowe
rpluem says: Is this really the correct thing to do on UNIX? I am not sure
if all dup2 implementation notice that both fd's are the same. Otherwise
they close stdout/stderr first and dup a then closed fd in stdout/stderr,
leaving us without stdout/stderr in the child.

-ascs

-----Message d'origine-----
De : Chris Boyce [mailto:chris.boyce@berbee.com]
Envoyé : mercredi 14 novembre 2007 20:37
À : users@httpd.apache.org
Objet : [users@httpd] Lingering children tying up file handles

Hello... We're using Apache 2.0.59 on Solaris. I've noticed over the last few bug/security releases, file handles being held open on rotated log files when using graceful restarts.

>>From what I understand, graceful restarting is the best way to do log
rotation. But over the course of a few nightly log rotations, there are rotated logs that never have their file handles closed, causing df to incorrectly report a full filesystem. The only fix is to either a) kill the offending server process (if identifiable) or b) do a hard restart.

In the ps output below, I was able to identify the offending server (see Nov 11 process **):

nobody 23698 7548 0 12:26:15 ? httpd -k start
nobody 23769 7548 0 12:27:30 ? httpd -k start
nobody 23508 7548 0 12:24:33 ? httpd -k start
nobody 26870 7548 0 Nov 11 ? httpd -k start **
nobody 23704 7548 0 12:26:39 ? httpd -k start
root 7548 1 0 Nov 08 ? httpd -k start
nobody 25972 7548 0 18:40:32 ? httpd -k start
nobody 23766 7548 0 12:27:10 ? httpd -k start
nobody 23572 7548 0 12:25:00 ? httpd -k start
nobody 23636 7548 0 12:25:08 ? httpd -k start
nobody 23430 7548 0 12:23:29 ? httpd -k start
nobody 23436 7548 0 12:23:34 ? httpd -k start
nobody 23703 7548 0 12:26:37 ? httpd -k start
nobody 23767 7548 0 12:27:18 ? httpd -k start

The Nov 11 server was three days old (and was not the parent process), so it stuck out, and killing it "freed up" the "used" space in df.

We use the worker MPM as follows:

StartServers 40
MaxClients 350
MinSpareThreads 40
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 10000

I tried setting MaxRequestsPerChild in an effort to force aging children to die, but that hasn't helped.

Can anyone explain why these rogue children are not going away over time? Do I simply need to stop using "graceful" in my log rotations (though less desirable)?

Thanks for reading.

--
Chris Boyce

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org