You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Boyle Owen <Ow...@swx.com> on 2008/04/02 19:43:52 UTC
[users@httpd] Return of bug 14219 in version 2.2.8? (too many SSL VHs causes fopen to fail)
Greetings,
On recently upgrading to 2.2.8, I was rather surprised to find that my
new binary would not start on the production server. The error message
is:
[Wed Apr 02 19:25:18 2008] [error] Init: Unable to read server
certificate from file /path-to-certificates/cert99.crt
[Wed Apr 02 19:25:18 2008] [error] SSL Library Error: 33558552
error:02001018:system library:fopen:Too many open files
[Wed Apr 02 19:25:18 2008] [error] SSL Library Error: 537317378
error:2006D002:BIO routines:BIO_new_file:system lib
After trying all the usual suspects to do with increasing system
resources, I discovered that this was documented in an old bugzilla:
https://issues.apache.org/bugzilla/show_bug.cgi?id=14219
I commented out a few VHs and found that the problem only kicked in when
the number of VHs (hence log files) and SSL certs reached a certain
size. So it seems that the problem is indeed due to a file descriptor
exceeding 255, as the report describes.
The funny thing is, apache 2.2.6 starts perfectly when loaded with the
*exact* same config file (hence same number of files to handle)... So my
question is, does anybody know what has changed between 2.2.6 and 2.2.8
that could resurrect this rather limiting behaviour?
Rgds,
Owen Boyle
Disclaimer: Any disclaimer attached to this message may be ignored.
This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. If you receive this message in error, please notify the sender urgently and then immediately delete the message and any copies of it from your system. Please also immediately destroy any hardcopies of the message. The sender's company reserves the right to monitor all e-mail communications through their networks.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Return of bug 14219 in version 2.2.8? (too many SSL VHs causes fopen to fail)
Posted by Boyle Owen <Ow...@swx.com>.
> -----Original Message-----
> From: Eric Covener [mailto:covener@gmail.com]
> Sent: Thursday, April 03, 2008 3:35 PM
> To: users@httpd.apache.org
> Subject: Re: [users@httpd] Return of bug 14219 in version
> 2.2.8? (too many SSL VHs causes fopen to fail)
>
> On Thu, Apr 3, 2008 at 7:33 AM, Boyle Owen <Ow...@swx.com> wrote:
>
> http://developers.sun.com/solaris/articles/stdio_256.html
This is a great link - much insight revealed.... I will try the
extendedFILE facility when the server is a bit quieter.
>
> Any chance your 2.2.6 was 64-bit (-V) ?
All versions are definately 32-bit (checked with httpd -V).
> Or your config just barely
> breaks the 255 mark and 2.2.8 had a minor change that pushed you over
> the edge?
There is no change to the config between versions. But maybe, for given
number of VHs, 2.2.8 opens an extra couple of filehandles, compared to
2.2.6 - is that what you're wondering? That could be... I'll try to
count exactly the number of FHs in use in our config.
Rgds,
Owen Boyle
Disclaimer: Any disclaimer attached to this message may be ignored.
>
> --
> Eric Covener
> covener@gmail.com
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP
> Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. If you receive this message in error, please notify the sender urgently and then immediately delete the message and any copies of it from your system. Please also immediately destroy any hardcopies of the message. The sender's company reserves the right to monitor all e-mail communications through their networks.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Return of bug 14219 in version 2.2.8? (too manySSL VHs causes fopen to fail) - UNDERSTOOD (?)
Posted by Boyle Owen <Ow...@swx.com>.
> -----Original Message-----
> From: Joe Orton [mailto:jorton@redhat.com]
> Sent: Friday, April 04, 2008 12:28 PM
> To: Boyle Owen
> Cc: users@httpd.apache.org
> Subject: Re: [users@httpd] Return of bug 14219 in version
> 2.2.8? (too manySSL VHs causes fopen to fail) - UNDERSTOOD (?)
>
>
> Can you capture truss output ("truss -o foo bin/httpd" or something
> similar) and file a bug? It sounds like a regression in the piped
> logger code, it shouldn't be leaving any more open fds in the parent
> AFAIK.
Done (https://issues.apache.org/bugzilla/show_bug.cgi?id=44755)
Warmest thanks to all for contributing to this issue!
Rgds,
Owen Boyle
Disclaimer: Any disclaimer attached to this message may be ignored.
>
> joe
>
This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. If you receive this message in error, please notify the sender urgently and then immediately delete the message and any copies of it from your system. Please also immediately destroy any hardcopies of the message. The sender's company reserves the right to monitor all e-mail communications through their networks.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Return of bug 14219 in version 2.2.8? (too many
SSL VHs causes fopen to fail) - UNDERSTOOD (?)
Posted by Joe Orton <jo...@redhat.com>.
On Fri, Apr 04, 2008 at 11:52:15AM +0200, Boyle Owen wrote:
...
> As an aside, our apache listens on 88 ports and has 102 piped log files
> to write to. Add on STDIN/OUT/ERR and the Pid and Lock files and that's
> how we get to 195. Comparing the two versions, we see the only
> difference is in the number of pipes - 102 in 2.2.6 and 235 in 2.2.8.
> So something is doubling up the number of pipes (and then some...)
>
> I then trawled the code to see what has changed and found that there is
> a recent addition to server/log.c
> (http://svn.apache.org/viewvc/httpd/httpd/tags/2.2.8/server/log.c?view=d
> iff&r1=602466&r2=602467) that seems to redirect SDTOUT and STDERR to the
> access and error logs. Would this have the effect of doubling the number
> of pipes?
...
Can you capture truss output ("truss -o foo bin/httpd" or something
similar) and file a bug? It sounds like a regression in the piped
logger code, it shouldn't be leaving any more open fds in the parent
AFAIK.
joe
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Return of bug 14219 in version 2.2.8? (too many SSL VHs causes fopen to fail) - UNDERSTOOD (?)
Posted by Boyle Owen <Ow...@swx.com>.
> -----Original Message-----
> From: Eric Covener [mailto:covener@gmail.com]
> Sent: Thursday, April 03, 2008 3:35 PM
> To: users@httpd.apache.org
> Subject: Re: [users@httpd] Return of bug 14219 in version
> 2.2.8? (too many SSL VHs causes fopen to fail)
>
>
> Sun has some relatively recent changes in this area:
>
> http://developers.sun.com/solaris/articles/stdio_256.html
I implemented the extendedFILE facility, as recommended in the article,
and the server started without the fopen error :-)
Once I got a working instance of 2.2.8, I was able to peek inside and
discovered that, for an identical config file, 2.2.6 requires 195 file
handles, whereas 2.2.8 needs 328. So that was why 2.2.8 was tripping the
fopen limitation.
As an aside, our apache listens on 88 ports and has 102 piped log files
to write to. Add on STDIN/OUT/ERR and the Pid and Lock files and that's
how we get to 195. Comparing the two versions, we see the only
difference is in the number of pipes - 102 in 2.2.6 and 235 in 2.2.8.
So something is doubling up the number of pipes (and then some...)
I then trawled the code to see what has changed and found that there is
a recent addition to server/log.c
(http://svn.apache.org/viewvc/httpd/httpd/tags/2.2.8/server/log.c?view=d
iff&r1=602466&r2=602467) that seems to redirect SDTOUT and STDERR to the
access and error logs. Would this have the effect of doubling the number
of pipes?
I am now faced with two ways forward;
1) Start using the extendedFILE facility. Probably a good idea since the
255 limit is an accident waiting to happen - we are already at 80%
capacity, even with 2.2.6 and so a few more sites could push us over
anyway. However, there are two caveats in the article; namely that
apache should not directly dereference the _file member of the FILE
structure or use the fileno() macro. I grep'd the code for '_file' and
'fileno()' and it seems clean so we should be OK...
2) Be a nuisance and question *why* apache needs to duplicate the pipes
like this. Can we be the only users that this is causing problems for?
Incidentally, I will be at apachecon in Amsterdam next week so if anyone
wants to discuss this, look me up - beer's on me :-)
Rgds,
Owen Boyle
Disclaimer: Any disclaimer attached to this message may be ignored.
>
> --
> Eric Covener
> covener@gmail.com
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP
> Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. If you receive this message in error, please notify the sender urgently and then immediately delete the message and any copies of it from your system. Please also immediately destroy any hardcopies of the message. The sender's company reserves the right to monitor all e-mail communications through their networks.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Return of bug 14219 in version 2.2.8? (too many SSL VHs causes fopen to fail)
Posted by Eric Covener <co...@gmail.com>.
On Thu, Apr 3, 2008 at 7:33 AM, Boyle Owen <Ow...@swx.com> wrote:
> I didn't try the patch because of the discouraging comment 7 from Joe
> Orton that follows it :-) Also, by the author's own admission the patch
> is really a hack that I am uncomfortable applying to an
> enterprise-critical webserver.
Sun has some relatively recent changes in this area:
http://developers.sun.com/solaris/articles/stdio_256.html
Any chance your 2.2.6 was 64-bit (-V) ? Or your config just barely
breaks the 255 mark and 2.2.8 had a minor change that pushed you over
the edge?
--
Eric Covener
covener@gmail.com
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
RE: [users@httpd] Return of bug 14219 in version 2.2.8? (too many SSL VHs causes fopen to fail)
Posted by Boyle Owen <Ow...@swx.com>.
> -----Original Message-----
> From: Nick Kew [mailto:nick@webthing.com]
> Sent: Wednesday, April 02, 2008 7:52 PM
> To: users@httpd.apache.org
> Subject: Re: [users@httpd] Return of bug 14219 in version
> 2.2.8? (too many SSL VHs causes fopen to fail)
>
> On Wed, 2 Apr 2008 19:43:52 +0200
> "Boyle Owen" <Ow...@swx.com> wrote:
>
> > After trying all the usual suspects to do with increasing system
> > resources, I discovered that this was documented in an old bugzilla:
> > https://issues.apache.org/bugzilla/show_bug.cgi?id=14219
>
> There's a patch posted in Comment 6 to that PR. Is that applicable
> to you? Did you try it, and what happened?
I didn't try the patch because of the discouraging comment 7 from Joe
Orton that follows it :-) Also, by the author's own admission the patch
is really a hack that I am uncomfortable applying to an
enterprise-critical webserver.
Recall that the *identical* config works fine with apache 2.2.6 (and
every version prior). The vulnerability to this Solaris feature/bug only
emerges with the recent upgrade to 2.2.8.
Rgds,
Owen Boyle
Disclaimer: Any disclaimer attached to this message may be ignored.
>
> --
> Nick Kew
>
> Application Development with Apache - the Apache Modules Book
> http://www.apachetutor.org/
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP
> Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. If you receive this message in error, please notify the sender urgently and then immediately delete the message and any copies of it from your system. Please also immediately destroy any hardcopies of the message. The sender's company reserves the right to monitor all e-mail communications through their networks.
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Return of bug 14219 in version 2.2.8? (too many
SSL VHs causes fopen to fail)
Posted by Nick Kew <ni...@webthing.com>.
On Wed, 2 Apr 2008 19:43:52 +0200
"Boyle Owen" <Ow...@swx.com> wrote:
> After trying all the usual suspects to do with increasing system
> resources, I discovered that this was documented in an old bugzilla:
> https://issues.apache.org/bugzilla/show_bug.cgi?id=14219
There's a patch posted in Comment 6 to that PR. Is that applicable
to you? Did you try it, and what happened?
--
Nick Kew
Application Development with Apache - the Apache Modules Book
http://www.apachetutor.org/
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org