You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Ben Welsh <be...@gmail.com> on 2009/05/20 08:00:15 UTC
[users@httpd] What is the best way to handle "too many open files" errors?
Hello,
I'm managing a pgsql --> django --> mod_python --> apache machine that
serves no media, but handles the postgres memcached and apache part of the
stack all by itself. It's a dedicated virtual server with Red Hat EL and 1GB
RAM.
As traffic to the site has ramped up over time, I've begun to encounter "too
many open files in the system" errors via Apache that look something like
this:
OperationalError: FATAL: could not open file <filename>: Too many open files
in systemThis has become something of a debilitating problem and I'm
struggling to come up with the best strategy for handling it. Some people
have told me that I can solve it by better configuring my existing machine
-- while others have insisted that I must upgrade my server rig by expanding
it beyond one DV.
My hopes in posting here is that y'all might give me some insight into how I
can answer the question about the better approach.
I've run the commands you see below and the conclusion that I'm drawing is
that the I simply have way too many httpd file objects open -- but I'm
unsure about the wisest way of reducing them -- or whether I can reasonably
expected them to go down much.
I'd very much appreciate any insight the list could provide into the
situation, even if it's just point me to a reference. I've trolled the lists
and the web and still am a Iittle unsure of the best approach, so I thought
I'd post. If I'm overlooking the most awesome tutorial or previous lister
post ever, please forgive my ignorance. I'd appreciate even just the link.
Thank you,
Ben.
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 137215
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
*postgres*.conf (max_files_per_process is commented out)
max_connections = 100
shared_buffers = 1000
$ httpd -l
Compiled in modules:
core.c
prefork.c
http_core.c
mod_so.c
httpd.conf...
Timeout 20
KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 5
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 4000
</IfModule>
$ cat /proc/sys/fs/file-nr
73320 0 262144
$ lsof | awk '{print $1}' | sort | uniq -c
1 COMMAND
10 awk
49 bash
17 crond
5154 httpd
9 init
25 lsof
39 memcached
807 postmaste
62 saslauthd
69 sendmail
28 sh
7 sort
183 sshd
13 syslogd
8 uniq
19 xinetd
Re: [users@httpd] What is the best way to handle "too many open
files" errors?
Posted by palewire <be...@gmail.com>.
awarnier wrote:
>
> I could not say that. Only you really know which modules you load with
> the LoadModule directives, and which ones you really need for your
> configuration and applications.
>
> Taking into account that libraries (stuff like the
> /usr/lib/libkrb5.so.3.3 for instance) are sometimes loaded very
> indirectly (by another library loaded by another library loaded by some
> innocent-looking module for example), I would nevertheless wonder why
> for example /usr/lib/libkrb5.so.3.3 is being loaded at all, unless one
> of your areas is protected by kerberos authentication. The same for
> ldap, the same for jpeg libs, and so on.
> But is is quite possible that this is really needed by some module in
> some application.
> I would say : look at each of your LoadModule directives, and at that
> module's description here :
> http://httpd.apache.org/docs/2.2/mod/
> (in "Other modules")
> and ask yourself if you really need this.
> If you think not, comment out that LoadModule, restart your server and
> check if anything breaks.
>
> Another thing : it looks from your lsof list, that you are using the
> Apache "prefork" model.
> I don't remember precisely your configuration or the kind of load or
> processes you are running, but you might try the "worker" (threaded)
> model instead. I am not entirely sure, but I believe that this will
> also reduce the total number of files opened on your system, as threads
> may share a lot more in that respect, than individual child processes do.
>
> But I would wait a few hours for a real expert to comment, which I'm
> sure one will do if I wrote something really stupid above.
>
Alright. Thank you so much for your opinions, Andre. I am grateful for your
time and attention.
--
View this message in context: http://www.nabble.com/-users%40httpd--What-is-the-best-way-to-handle-%22too-many-open-files%22-errors--tp23629492p23642169.html
Sent from the Apache HTTP Server - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What is the best way to handle "too many open files" errors?
Posted by Sean Conner <sp...@conman.org>.
It was thus said that the Great Ben Welsh once stated:
> When I study my lsof logs on httpd instances, I'm seeing the use of a number
> of image libraries, or instance, that I cannot imagine any use for in my
> applications. then when I look at the yum log the DV provisioned by my host
> I see those same libraries as default installs. To me there seem to be two
> possible explanations -- one, there are unnecessary libraries installed on
> the system that are racking up my open files needlessly;
First off, libraries being referenced aren't counted towards open files
[1]. The thing is, if you are running Apache under Linux, you can find the
process ID of Apache (one of many, probably), say, it's 32345. Then, within
a shell, do:
GenericRootPrompt> cd /proc/32345/fd
GenericRootPrompt> ls -l | more
And that will list all the open files in that particular process. From
there, you can figure out what you might need to do.
As to why Apache has so many linked libraries, one question is: do you
run PHP? If so, that includes a ton of libraries (in my experience with
typical distributions), but since they don't count against the open file
limit, don't bother looking there to solve this issue.
-spc
[1] I wrote a quick program to settle this. All the program does is
attempt to open as many files as possible at once and see were I
get. This is what I got:
[spc]lucy:/tmp/blah>../t
me: 21464
max files: 1024
01021.txt: Too many open files
At that point, the program pauses so I can check things out. In
/proc/21464/fd I see there are in fact 1,024 files open. The first
/file opened, 00000.txt, is actually the *fourth* file to be open,
the first three are:
lrwx------ 1 spc spc 64 May 26 17:37 0 -> /dev/pts/3
lrwx------ 1 spc spc 64 May 26 17:37 1 -> /dev/pts/3
lrwx------ 1 spc spc 64 May 26 17:37 2 -> /dev/pts/3
which correspond to STDIN, STDOUT and STDERR (what every process
starts with). Further more, 1,024 files are open and yet, when I
check to see what libraries are loaded:
[spc]lucy:/proc/21464>more maps
00777000-0078c000 r-xp 00000000 fd:00 18187218 /lib/ld-2.3.4.so
0078c000-0078d000 r--p 00015000 fd:00 18187218 /lib/ld-2.3.4.so
0078d000-0078e000 rw-p 00016000 fd:00 18187218 /lib/ld-2.3.4.so
00795000-008ba000 r-xp 00000000 fd:00 18187219 /lib/tls/libc-2.3.4.so
008ba000-008bc000 r--p 00124000 fd:00 18187219 /lib/tls/libc-2.3.4.so
008bc000-008be000 rw-p 00126000 fd:00 18187219 /lib/tls/libc-2.3.4.so
008be000-008c0000 rw-p 008be000 00:00 0
08048000-08049000 r-xp 00000000 fd:00 15942114 /tmp/t
08049000-0804a000 rw-p 00000000 fd:00 15942114 /tmp/t
b7ef1000-b7ef2000 rw-p b7ef1000 00:00 0
b7f11000-b7f13000 rw-p b7f11000 00:00 0
bfe3e000-c0000000 rw-p bfe3e000 00:00 0
ffffe000-fffff000 ---p 00000000 00:00 0
If these had any effect on open files, then I would have only been
able to create 1,013 files, not the 1,022 I did (I started from 0,
which is why there seems to be a discrpancy of one).
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] What is the best way to handle "too many open
files" errors?
Posted by palewire <be...@gmail.com>.
Mick,
Thanks for your expertise and time.
Here are my settings in httpd.conf, which I've tweaked in the last couple
days per some advice from my host.
<IfModule prefork.c>
StartServers 2
MinSpareServers 2
MaxSpareServers 5
ServerLimit 200
MaxClients 200
MaxRequestsPerChild 4000
</IfModule>
And here's the ulimit:
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 137215
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
How would I do the sort of math you propose?
Thanks,
Ben.
Mick Sheppard wrote:
>
> Hi,
>
> Just to throw a slight spanner in the works here. My understanding of
> 'open files' is open file descriptors. As far as a file descriptor is
> concerned there is no real difference between a physical file on disk
> and a socket (network connection). So eliminating physical files, whilst
> it might get you a little more headroom, is unlikely to solve the
> problem.
>
> I would look at the Apache configuration to see how many connections
> your server is configured to handle and compare that to the max files
> value from ulimit. I suspect that this is your issue.
>
> Remember that as you add connectivity to databases that this will
> increase the number of sockets, and therefore open files, used for each
> page served.
>
> Mick
>
> -----Original Message-----
> From: palewire [mailto:ben.welsh@gmail.com]
> Sent: 20 May 2009 23:55
> To: users@httpd.apache.org
> Subject: Re: [users@httpd] What is the best way to handle "too many open
> files" errors?
>
>
> Sean --
>
> Thanks for the lengthy and thoughtful response.
>
> What I've found most successful so far is hunting through the lsof |
> grep
> http and finding ways to eliminate unnecessary files. And I'm also
> exploring
> an expansion of my production resources to include a second tier machine
> that houses the database.
>
> If either of these ideas are stupid -- or there's something more
> efficient
> -- I'd really appreciate the pointers.
>
> Thank you.
>
> Ben.
> --
> View this message in context:
> http://www.nabble.com/-users%40httpd--What-is-the-best-way-to-handle-%22
> too-many-open-files%22-errors--tp23629492p23644950.html
> Sent from the Apache HTTP Server - Users mailing list archive at
> Nabble.com.
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server
> Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
> This is an email from the CPP Group Plc, Holgate Park, York, YO26 4GA;
> telephone +44 (0)1904 544500.
> This message may contain information that is confidential. If you are not
> the intended recipient,
> you may not peruse, use, disseminate, distribute or copy this message. If
> you have received this
> message in error, please notify the sender immediately by email, facsimile
> or telephone and either
> return or destroy the original message.
>
> The CPP Group Plc accepts no responsibility for any changes made to this
> message after it has been
> sent by the original author. This email has been scanned for all viruses
> by the MessageLabs
> Email Security System.
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
--
View this message in context: http://www.nabble.com/-users%40httpd--What-is-the-best-way-to-handle-%22too-many-open-files%22-errors--tp23629492p23662369.html
Sent from the Apache HTTP Server - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What is the best way to handle "too many open
files" errors?
Posted by Ben Welsh <be...@gmail.com>.
When I study my lsof logs on httpd instances, I'm seeing the use of a number
of image libraries, or instance, that I cannot imagine any use for in my
applications. then when I look at the yum log the DV provisioned by my host
I see those same libraries as default installs. To me there seem to be two
possible explanations -- one, there are unnecessary libraries installed on
the system that are racking up my open files needlessly; two, my app is
using these libraries for things I an unaware of. can you think of another
explanation?
On Sat, May 23, 2009 at 2:01 PM, Sean Conner <sp...@conman.org> wrote:
> It was thus said that the Great Ben Welsh once stated:
> > I just had something of a "slap your forehead" moment on this one. It had
> be
> > eating me about where all those extra lsof processes came from. And then
> it
> > hit me. It's the "developer tools" kit from CentOS. Duh.
> >
> >
> http://www.linuxtopia.org/online_books/centos_linux_guides/centos_linux_developer_tools_guide/
> >
> > Any tips for the smartest way to pull it apart after the fact?
>
> I don't quite understand your question here. Could you clarify?
>
> -spc
>
>
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] What is the best way to handle "too many open files" errors?
Posted by Sean Conner <sp...@conman.org>.
It was thus said that the Great Ben Welsh once stated:
> I just had something of a "slap your forehead" moment on this one. It had be
> eating me about where all those extra lsof processes came from. And then it
> hit me. It's the "developer tools" kit from CentOS. Duh.
>
> http://www.linuxtopia.org/online_books/centos_linux_guides/centos_linux_developer_tools_guide/
>
> Any tips for the smartest way to pull it apart after the fact?
I don't quite understand your question here. Could you clarify?
-spc
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What is the best way to handle "too many open
files" errors?
Posted by Ben Welsh <be...@gmail.com>.
I just had something of a "slap your forehead" moment on this one. It had be
eating me about where all those extra lsof processes came from. And then it
hit me. It's the "developer tools" kit from CentOS. Duh.
http://www.linuxtopia.org/online_books/centos_linux_guides/centos_linux_developer_tools_guide/
Any tips for the smartest way to pull it apart after the fact?
On Thu, May 21, 2009 at 5:55 PM, Sean Conner <sp...@conman.org> wrote:
> It was thus said that the Great Mick Sheppard once stated:
> > Hi,
> >
> > Just to throw a slight spanner in the works here. My understanding of
> > 'open files' is open file descriptors. As far as a file descriptor is
> > concerned there is no real difference between a physical file on disk
> > and a socket (network connection). So eliminating physical files, whilst
> > it might get you a little more headroom, is unlikely to solve the
> > problem.
>
> Yes and no. Yes, sockets are considered "opened files" and do count
> against the open file limit, that limit is *per process*, not *system
> wide*.
> I just checked one of our main web servers (serving up a few hundred sites)
> and found only three active sockets per process:
>
> 1. the listening socket on port 80, waiting for a connection
> 2. the listening socket on port 443, waiting for a connection
> 3. an actual connection to a client
>
> The main root process only has the first two sockets open. If you have
> Apache configured to have at a minimum 30 spare servers, no one Apache
> process will have 30 sockets open---sockets just don't work that way. What
> it does mean is you have at least 31 apache processes running, one as root
> (to bind to the proper ports [1]) and 30 ready to service a request, so at
> most, any Apache process will have one socket per Listen statement, and one
> socket only when actually processing a request.
>
> So, for example, let's say you have the following Apache configuration:
>
> Listen *:80
> StartServers 100
> MinSpareServers 100
> MaxSpareServers 900
> ServerLimit 1000
> MaxClients 1000
> MaxRequestsPerChild 4000
> CustomLog access_log
> ErrorLog error_log
>
> All your apache processes would have only seven files open (eight if
> serving
> a request), and they would look something like:
>
> 0 -> /dev/null (STDIN)
> 1 -> /dev/null (STDOUT)
> 2 -> error_log (STDERR)
> 3 -> listening socket
> 4 -> pipe (for CGI communications)
> 5 -> pipe (for CGI communications)
> 6 -> error_log
> 7 -> access_log
>
> The real concern are log files, not the number of connections in the
> Apache configuration. The only way the number of connections might be a
> consideration is if you are using the threading MPM, where each thread (in
> a
> single process) gets a connection.
>
> > Remember that as you add connectivity to databases that this will
> > increase the number of sockets, and therefore open files, used for each
> > page served.
>
> True, but you really need to look at this at the process level.
>
> -spc
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] What is the best way to handle "too many open files" errors?
Posted by Sean Conner <sp...@conman.org>.
It was thus said that the Great Mick Sheppard once stated:
> Hi,
>
> Just to throw a slight spanner in the works here. My understanding of
> 'open files' is open file descriptors. As far as a file descriptor is
> concerned there is no real difference between a physical file on disk
> and a socket (network connection). So eliminating physical files, whilst
> it might get you a little more headroom, is unlikely to solve the
> problem.
Yes and no. Yes, sockets are considered "opened files" and do count
against the open file limit, that limit is *per process*, not *system wide*.
I just checked one of our main web servers (serving up a few hundred sites)
and found only three active sockets per process:
1. the listening socket on port 80, waiting for a connection
2. the listening socket on port 443, waiting for a connection
3. an actual connection to a client
The main root process only has the first two sockets open. If you have
Apache configured to have at a minimum 30 spare servers, no one Apache
process will have 30 sockets open---sockets just don't work that way. What
it does mean is you have at least 31 apache processes running, one as root
(to bind to the proper ports [1]) and 30 ready to service a request, so at
most, any Apache process will have one socket per Listen statement, and one
socket only when actually processing a request.
So, for example, let's say you have the following Apache configuration:
Listen *:80
StartServers 100
MinSpareServers 100
MaxSpareServers 900
ServerLimit 1000
MaxClients 1000
MaxRequestsPerChild 4000
CustomLog access_log
ErrorLog error_log
All your apache processes would have only seven files open (eight if serving
a request), and they would look something like:
0 -> /dev/null (STDIN)
1 -> /dev/null (STDOUT)
2 -> error_log (STDERR)
3 -> listening socket
4 -> pipe (for CGI communications)
5 -> pipe (for CGI communications)
6 -> error_log
7 -> access_log
The real concern are log files, not the number of connections in the
Apache configuration. The only way the number of connections might be a
consideration is if you are using the threading MPM, where each thread (in a
single process) gets a connection.
> Remember that as you add connectivity to databases that this will
> increase the number of sockets, and therefore open files, used for each
> page served.
True, but you really need to look at this at the process level.
-spc
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] What is the best way to handle "too many open files" errors?
Posted by Mick Sheppard <Mi...@cpp.co.uk>.
Hi,
Just to throw a slight spanner in the works here. My understanding of
'open files' is open file descriptors. As far as a file descriptor is
concerned there is no real difference between a physical file on disk
and a socket (network connection). So eliminating physical files, whilst
it might get you a little more headroom, is unlikely to solve the
problem.
I would look at the Apache configuration to see how many connections
your server is configured to handle and compare that to the max files
value from ulimit. I suspect that this is your issue.
Remember that as you add connectivity to databases that this will
increase the number of sockets, and therefore open files, used for each
page served.
Mick
-----Original Message-----
From: palewire [mailto:ben.welsh@gmail.com]
Sent: 20 May 2009 23:55
To: users@httpd.apache.org
Subject: Re: [users@httpd] What is the best way to handle "too many open
files" errors?
Sean --
Thanks for the lengthy and thoughtful response.
What I've found most successful so far is hunting through the lsof |
grep
http and finding ways to eliminate unnecessary files. And I'm also
exploring
an expansion of my production resources to include a second tier machine
that houses the database.
If either of these ideas are stupid -- or there's something more
efficient
-- I'd really appreciate the pointers.
Thank you.
Ben.
--
View this message in context:
http://www.nabble.com/-users%40httpd--What-is-the-best-way-to-handle-%22
too-many-open-files%22-errors--tp23629492p23644950.html
Sent from the Apache HTTP Server - Users mailing list archive at
Nabble.com.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server
Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
This is an email from the CPP Group Plc, Holgate Park, York, YO26 4GA; telephone +44 (0)1904 544500.
This message may contain information that is confidential. If you are not the intended recipient,
you may not peruse, use, disseminate, distribute or copy this message. If you have received this
message in error, please notify the sender immediately by email, facsimile or telephone and either
return or destroy the original message.
The CPP Group Plc accepts no responsibility for any changes made to this message after it has been
sent by the original author. This email has been scanned for all viruses by the MessageLabs
Email Security System.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What is the best way to handle "too many open
files" errors?
Posted by palewire <be...@gmail.com>.
Sean --
Thanks for the lengthy and thoughtful response.
What I've found most successful so far is hunting through the lsof | grep
http and finding ways to eliminate unnecessary files. And I'm also exploring
an expansion of my production resources to include a second tier machine
that houses the database.
If either of these ideas are stupid -- or there's something more efficient
-- I'd really appreciate the pointers.
Thank you.
Ben.
--
View this message in context: http://www.nabble.com/-users%40httpd--What-is-the-best-way-to-handle-%22too-many-open-files%22-errors--tp23629492p23644950.html
Sent from the Apache HTTP Server - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What is the best way to handle "too many open files"
errors?
Posted by André Warnier <aw...@ice-sa.com>.
Sean Conner wrote:
>
>> But I would wait a few hours for a real expert to comment, which I'm
>> sure one will do if I wrote something really stupid above.
>
Thanks, Sean, for the clarification. Usweful for me too.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What is the best way to handle "too many open files" errors?
Posted by Sean Conner <sp...@conman.org>.
It was thus said that the Great Andr Warnier once stated:
>
> Another thing : it looks from your lsof list, that you are using the
> Apache "prefork" model.
> I don't remember precisely your configuration or the kind of load or
> processes you are running, but you might try the "worker" (threaded)
> model instead. I am not entirely sure, but I believe that this will
> also reduce the total number of files opened on your system, as threads
> may share a lot more in that respect, than individual child processes do.
Switching to the threaded model won't help if you are running out of open
files---the open file limit is *per process* (more on that below). And yes,
a child process will inherit all open files from the parent process, but
once the child process is created, it can close any open files without
affecting the parent (and by the same token, the parent can close files
without affecting any existing children).
> But I would wait a few hours for a real expert to comment, which I'm
> sure one will do if I wrote something really stupid above.
First off, "ulmit -n" will report back the number of open files a
*PROCESS* can have open---this isn't system wide, but per-proess. It's an
important distinction.
Second, lsof does report all files used by a process, but that isn't the
whole story. For instance, lsof (and one instance of apache from my
development server):
httpd 17155 apache cwd DIR 253,0 4096 2 /
httpd 17155 apache rtd DIR 253,0 4096 2 /
httpd 17155 apache txt REG 253,0 259488 5300352 /usr/sbin/httpd
httpd 17155 apache mem REG 253,0 50748 18187232 /lib/tls/librt-2.3.4.so
httpd 17155 apache mem REG 253,0 28544 18187230 /lib/libcrypt-2.3.4.so
httpd 17155 apache mem REG 253,0 1525004 18187219 /lib/tls/libc-2.3.4.so
httpd 17155 apache mem REG 253,0 81184 18187224 /lib/libresolv-2.3.4.so
httpd 17155 apache mem REG 253,0 213600 18187228 /lib/libssl.so.0.9.7a
httpd 17155 apache mem REG 253,0 7004 18186360 /lib/libcom_err.so.2.1
httpd 17155 apache mem REG 253,0 82320 5303493 /usr/lib/libsasl2.so.2.0.19
... a whole mess of output deleted
httpd 17155 apache mem REG 253,0 1261824 5345160 /usr/lib/perl5/5.8.5/i386-linux-thread-multi/CORE/libperl.so
httpd 17155 apache mem REG 253,0 804084 6424738 /usr/lib/libstdc++.so.6.0.3
httpd 17155 apache DEL REG 0,6 422353 /dev/zero
httpd 17155 apache DEL REG 0,6 37541 /dev/zero
httpd 17155 apache DEL REG 0,6 422347 /dev/zero
httpd 17155 apache 0r CHR 1,3 2029 /dev/null
httpd 17155 apache 1w CHR 1,3 2029 /dev/null
httpd 17155 apache 2w REG 253,0 8426 2360724 /var/log/httpd/error_log.1
httpd 17155 apache 3u IPv6 37481 TCP *:http (LISTEN)
httpd 17155 apache 4u IPv6 37483 TCP *:https (LISTEN)
httpd 17155 apache 5r FIFO 0,7 422346 pipe
httpd 17155 apache 6w FIFO 0,7 422346 pipe
httpd 17155 apache 7w REG 253,0 8426 2360724 /var/log/httpd/error_log.1
httpd 17155 apache 8w REG 253,0 262156 2360561 /var/log/httpd/access_log.1
httpd 17155 apache 9w REG 253,0 15126 11783673 /etc/httpd/www.roswell.area51
httpd 17155 apache 10w REG 253,0 151095 11782872 /etc/httpd/wwwtest.roswell.area51
httpd 17155 apache 11w REG 253,0 3046 11781743 /etc/httpd/s-secure.roswell.area51
httpd 17155 apache 12w REG 253,0 4263 11783866 /etc/httpd/secure.roswell.area51
It's 146 lines, but it's not 146 "open" files. Since Linux can page
directly from executables and libraries, technically, they're "open" in the
sense that yes, the kernel is reading from them as the program runs, but
they're not counted against the "ulimit -n" limit (unless I'm terribly
mistaken, and I could be, but bear with me for a bit). You can see such
memory mappings if you check the maps file in the appropriate /proc'
directory:
[root@lucy 17155]# pwd
/proc/17155
root@lucy 17155]# more maps
00111000-00119000 r-xp 00000000 fd:00 18187232 /lib/tls/librt-2.3.4.so
00119000-0011a000 r--p 00007000 fd:00 18187232 /lib/tls/librt-2.3.4.so
0011a000-0011b000 rw-p 00008000 fd:00 18187232 /lib/tls/librt-2.3.4.so
0011b000-00125000 rw-p 0011b000 00:00 0
00125000-0012a000 r-xp 00000000 fd:00 18187230 /lib/libcrypt-2.3.4.so
0012a000-0012b000 r--p 00004000 fd:00 18187230 /lib/libcrypt-2.3.4.so
0012b000-0012c000 rw-p 00005000 fd:00 18187230 /lib/libcrypt-2.3.4.so
0012c000-00153000 rw-p 0012c000 00:00 0
00153000-00278000 r-xp 00000000 fd:00 18187219 /lib/tls/libc-2.3.4.so
00278000-0027a000 r--p 00124000 fd:00 18187219 /lib/tls/libc-2.3.4.so
0027a000-0027c000 rw-p 00126000 fd:00 18187219 /lib/tls/libc-2.3.4.so
0027c000-0027e000 rw-p 0027c000 00:00 0
0027e000-0028d000 r-xp 00000000 fd:00 18187224 /lib/libresolv-2.3.4.so
0028d000-0028e000 r--p 0000f000 fd:00 18187224 /lib/libresolv-2.3.4.so
0028e000-0028f000 rw-p 00010000 fd:00 18187224 /lib/libresolv-2.3.4.so
0028f000-00291000 rw-p 0028f000 00:00 0
00291000-002c2000 r-xp 00000000 fd:00 18187228 /lib/libssl.so.0.9.7a
002c2000-002c5000 rw-p 00031000 fd:00 18187228 /lib/libssl.so.0.9.7a
002c5000-002c7000 r-xp 00000000 fd:00 18186360 /lib/libcom_err.so.2.1
002c7000-002c8000 rw-p 00001000 fd:00 18186360 /lib/libcom_err.so.2.1
... rest snipped
For files that are
actually "opened" (as in, via the open() system call), you can check the
proc file system, and here:
[root@lucy fd]# pwd
/proc/17155/fd
[root@lucy fd]# ll
total 13
lr-x------ 1 root root 64 May 20 17:46 0 -> /dev/null
l-wx------ 1 root root 64 May 20 17:46 1 -> /dev/null
l-wx------ 1 root root 64 May 20 17:46 10 -> /etc/httpd/wwwtest.roswell.area51
l-wx------ 1 root root 64 May 20 17:46 11 -> /etc/httpd/s-secure.roswell.area51
l-wx------ 1 root root 64 May 20 17:46 12 -> /etc/httpd/secure.roswell.area51
l-wx------ 1 root root 64 May 20 17:46 2 -> /var/log/httpd/error_log.1
lrwx------ 1 root root 64 May 20 17:46 3 -> socket:[37481]
lrwx------ 1 root root 64 May 20 17:46 4 -> socket:[37483]
lr-x------ 1 root root 64 May 20 17:46 5 -> pipe:[422346]
l-wx------ 1 root root 64 May 20 17:46 6 -> pipe:[422346]
l-wx------ 1 root root 64 May 20 17:46 7 -> /var/log/httpd/error_log.1
l-wx------ 1 root root 64 May 20 17:46 8 -> /var/log/httpd/access_log.1
l-wx------ 1 root root 64 May 20 17:46 9 -> /etc/httpd/www.roswell.area51
These are the files that have actually been "open()"ed by the program (and I
see that logrotate on the development system is still borked, but I
digress). Files 0 and 1 are STDIN and STDOUT respecitively and since this
is the webserver, they're remapped to '/dev/null'. File 2 is STDERR and
that's redirected to '/var/log/httpd/error_log.1' (should be error_log, but
like I said, there's something wrong with logrotate on that system). Files
3 and 4 are the listening sockets (port 80 and 443---if you cross reference
the lsof output, you can find out which sockets are listening to which
ports), and well, I can go on (the two pipes are probably there for CGIs,
and you hve the various config and log files) but I think you get the idea.
In any case, this is what you should be concerned with. And there are
several ways you can proceed:
1. You can increase the open files limit with ulimit.
2. You can consolidate all log files to two (access_log and error_log).
3. You can consolidate all configuration files into one large file.
4. Check Apache modules for open files and modify accordingly.
5. Some combination of the above.
But what you actually do depends upon what you are doing with your
webserver (whenever I've enountered this problem, I first consolidate error
logs, then increase ulimit).
-spc (Hope this helps some ... )
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What is the best way to handle "too many open
files" errors?
Posted by André Warnier <aw...@ice-sa.com>.
Ben Welsh wrote:
> Thanks for your thoughts Andre. As I recollect, I think some of the these
> errors started cropping up around the time I compiled the python module
> numpy on the server. And I think that fortran and a lot of those deadweight
> file objects are its children. Perhaps if I scaled by the server to no
> longer use that python module, it might reduce the problems.
>
> Are there other modules that look unnecessary to you?
>
I could not say that. Only you really know which modules you load with
the LoadModule directives, and which ones you really need for your
configuration and applications.
Taking into account that libraries (stuff like the
/usr/lib/libkrb5.so.3.3 for instance) are sometimes loaded very
indirectly (by another library loaded by another library loaded by some
innocent-looking module for example), I would nevertheless wonder why
for example /usr/lib/libkrb5.so.3.3 is being loaded at all, unless one
of your areas is protected by kerberos authentication. The same for
ldap, the same for jpeg libs, and so on.
But is is quite possible that this is really needed by some module in
some application.
I would say : look at each of your LoadModule directives, and at that
module's description here :
http://httpd.apache.org/docs/2.2/mod/
(in "Other modules")
and ask yourself if you really need this.
If you think not, comment out that LoadModule, restart your server and
check if anything breaks.
Another thing : it looks from your lsof list, that you are using the
Apache "prefork" model.
I don't remember precisely your configuration or the kind of load or
processes you are running, but you might try the "worker" (threaded)
model instead. I am not entirely sure, but I believe that this will
also reduce the total number of files opened on your system, as threads
may share a lot more in that respect, than individual child processes do.
But I would wait a few hours for a real expert to comment, which I'm
sure one will do if I wrote something really stupid above.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What is the best way to handle "too many open
files" errors?
Posted by Ben Welsh <be...@gmail.com>.
Thanks for your thoughts Andre. As I recollect, I think some of the these
errors started cropping up around the time I compiled the python module
numpy on the server. And I think that fortran and a lot of those deadweight
file objects are its children. Perhaps if I scaled by the server to no
longer use that python module, it might reduce the problems.
Are there other modules that look unnecessary to you?
On Wed, May 20, 2009 at 12:10 PM, André Warnier <aw...@ice-sa.com> wrote:
> palewire wrote:
>
>> For what it's worth, I've taken the additional step of running `lsof |grep
>> httpd > http.dump` and posting it online. My understanding is that this
>> shows all of the httpd open files that are adding up to my high totals.
>>
>> http://gist.github.com/114943
>>
>> If anyone who understands this better than me has a minute, I'd appreciate
>> whether you could say whether something is out of the ordinary.
>>
>> I probably don't understand this much better than you, but I will
> nevertheless offer some comments.
>
> The first thing that comes to mind, is that you probably want to use some
> of the lsof filtering options, so that your list really shows only open
> files on the filesystem. I have the impression that quite a lot of the
> stuff shown are not really open files, and may not really count against a
> ulimit.
> But then again, my knowledge of lsof is quite limited.
>
> The second thing I notice, is that you seem to have just about everything
> under the sun running under your Apache, so I'm not so surprised that you
> would have a lot of files open.
> I see things there that make me believe you have stuff running under Apache
> like, at least : python, ldap, sql, kerberos, imaging stuff, geos, fortran,
> xml parsing, and whatnot.
> As good as Apache is, I am sure there is a limit somewhere as to what one
> poor server can take as abuse.
> You might want to look around your "LoadModule" directives, and check what
> you really need.
>
>
>
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] What is the best way to handle "too many open
files" errors?
Posted by André Warnier <aw...@ice-sa.com>.
palewire wrote:
> For what it's worth, I've taken the additional step of running `lsof |grep
> httpd > http.dump` and posting it online. My understanding is that this
> shows all of the httpd open files that are adding up to my high totals.
>
> http://gist.github.com/114943
>
> If anyone who understands this better than me has a minute, I'd appreciate
> whether you could say whether something is out of the ordinary.
>
I probably don't understand this much better than you, but I will
nevertheless offer some comments.
The first thing that comes to mind, is that you probably want to use
some of the lsof filtering options, so that your list really shows only
open files on the filesystem. I have the impression that quite a lot of
the stuff shown are not really open files, and may not really count
against a ulimit.
But then again, my knowledge of lsof is quite limited.
The second thing I notice, is that you seem to have just about
everything under the sun running under your Apache, so I'm not so
surprised that you would have a lot of files open.
I see things there that make me believe you have stuff running under
Apache like, at least : python, ldap, sql, kerberos, imaging stuff,
geos, fortran, xml parsing, and whatnot.
As good as Apache is, I am sure there is a limit somewhere as to what
one poor server can take as abuse.
You might want to look around your "LoadModule" directives, and check
what you really need.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What is the best way to handle "too many open
files" errors?
Posted by palewire <be...@gmail.com>.
For what it's worth, I've taken the additional step of running `lsof |grep
httpd > http.dump` and posting it online. My understanding is that this
shows all of the httpd open files that are adding up to my high totals.
http://gist.github.com/114943
If anyone who understands this better than me has a minute, I'd appreciate
whether you could say whether something is out of the ordinary.
Thanks again,
Ben.
--
View this message in context: http://www.nabble.com/-users%40httpd--What-is-the-best-way-to-handle-%22too-many-open-files%22-errors--tp23629492p23639553.html
Sent from the Apache HTTP Server - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What is the best way to handle "too many open
files" errors?
Posted by palewire <be...@gmail.com>.
Igor Cicimov wrote:
>
> Have you considered increasing the open files limit? Have a look at the
> ulimit command. The default is sometimes not enough.
>
> Cheers,
>
> Igor
>
Thx for your time and attention.
I have considered fiddling with the open files liimits, but frankly before I
go too crazy I'd like to get a better idea of what sort of performance
changes I can expect or tradeoffs it might entails. And, honestly, just how
the whole thing is supposed to work.
First off, I'd really like to know why I'm getting so many lsof entries for
httpd. Is that something that's normal -- and a typical symptom of of high
traffic -- or could it be the result of some bad configuration settings or
application programming that I need to repair?
Also, I'm told that there can be "hard" and "soft" limits between the number
of open files the OS is set to handle and what the database or other
software is set to handle. Do you happen to know how I can sort all of that
out? From the errors messages, I suspect the limits being breached as the
"hard" limits of the operating system, would you agree?
And, let's say I double or triple the current limits, is that going to be
enough? Too much? I'm curious what how far I can reasonable push that before
I can expect other problems with my hardware.
If there's some documentation somewhere on this I can read, I'd be so very
pleased. I'm happy to randomly fiddle with the settings and see what
happens, but I'm really hoping for something more authoritative so I can
learn what's actually going on under the hood and make the wisest possible
choice.
Ben.
--
View this message in context: http://www.nabble.com/-users%40httpd--What-is-the-best-way-to-handle-%22too-many-open-files%22-errors--tp23629492p23637915.html
Sent from the Apache HTTP Server - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] What is the best way to handle "too many open
files" errors?
Posted by Igor Cicimov <ic...@gmail.com>.
Have you considered increasing the open files limit? Have a look at the
ulimit command. The default is sometimes not enough.
Cheers,
Igor
On Wed, May 20, 2009 at 4:00 PM, Ben Welsh <be...@gmail.com> wrote:
> Hello,
>
> I'm managing a pgsql --> django --> mod_python --> apache machine that
> serves no media, but handles the postgres memcached and apache part of the
> stack all by itself. It's a dedicated virtual server with Red Hat EL and 1GB
> RAM.
>
> As traffic to the site has ramped up over time, I've begun to encounter "too
> many open files in the system" errors via Apache that look something like
> this:
>
> OperationalError: FATAL: could not open file <filename>: Too many open
> files in systemThis has become something of a debilitating problem and I'm
> struggling to come up with the best strategy for handling it. Some people
> have told me that I can solve it by better configuring my existing machine
> -- while others have insisted that I must upgrade my server rig by expanding
> it beyond one DV.
>
> My hopes in posting here is that y'all might give me some insight into how
> I can answer the question about the better approach.
>
> I've run the commands you see below and the conclusion that I'm drawing is
> that the I simply have way too many httpd file objects open -- but I'm
> unsure about the wisest way of reducing them -- or whether I can reasonably
> expected them to go down much.
>
> I'd very much appreciate any insight the list could provide into the
> situation, even if it's just point me to a reference. I've trolled the lists
> and the web and still am a Iittle unsure of the best approach, so I thought
> I'd post. If I'm overlooking the most awesome tutorial or previous lister
> post ever, please forgive my ignorance. I'd appreciate even just the link.
>
> Thank you,
>
> Ben.
>
> $ ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> file size (blocks, -f) unlimited
> pending signals (-i) 1024
> max locked memory (kbytes, -l) 32
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> stack size (kbytes, -s) 10240
> cpu time (seconds, -t) unlimited
> max user processes (-u) 137215
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
> *postgres*.conf (max_files_per_process is commented out)
> max_connections = 100
> shared_buffers = 1000
>
> $ httpd -l
>
> Compiled in modules:
> core.c
> prefork.c
> http_core.c
> mod_so.c
>
> httpd.conf...
>
> Timeout 20
> KeepAlive Off
> MaxKeepAliveRequests 100
> KeepAliveTimeout 5
> <IfModule prefork.c>
> StartServers 8
> MinSpareServers 5
> MaxSpareServers 20
> ServerLimit 256
> MaxClients 256
> MaxRequestsPerChild 4000
> </IfModule>
>
> $ cat /proc/sys/fs/file-nr
> 73320 0 262144
>
> $ lsof | awk '{print $1}' | sort | uniq -c
> 1 COMMAND
> 10 awk
> 49 bash
> 17 crond
> 5154 httpd
> 9 init
> 25 lsof
> 39 memcached
> 807 postmaste
> 62 saslauthd
> 69 sendmail
> 28 sh
> 7 sort
> 183 sshd
> 13 syslogd
> 8 uniq
> 19 xinetd
>
>