"AcceptMutex sysvsem" on high-volume Solaris servers

[Mark Mentovai <ma...@mentovai.com>]

This information applies to Apache running on Solaris with AcceptMutex 
set to sysvsem, and offers a method to improve Apache's performance.  
It may apply to other systems as well, if you replace some of the 
particulars (like "/etc/system") with values that make sense.

On Solaris, Apache makes available three methods for accept 
serialization: fcntl, sysvsem, and pthread.  "AcceptMutex fcntl" is the 
default and "safest" method in that is most likely to function as 
expected, but is comparitively a poor performer.  Using System 5 
semaphores by specifying "AcceptMutex sysvsem" offers superior 
performance, but as discussed in the past, does not function on high-
volume servers.  (See htdocs/manual/misc/perf-tuning.html for a 
discussion of accept serialization.)

On an untuned Solaris system, configuring "AcceptMutex sysvsem" will 
eventually cause Apache to exit, leaving the log peppered with:

accept_mutex_on: No space left on device
[Mon Jul  8 12:23:24 2002] [alert] Child 2319 returned a Fatal error... 
Apache is exiting!

This occurs once 30 child processes are present.  It causes the parent 
to exit, leaving 30 children without any pool size regulation: you're 
stuck with 30, unless the children exit (either cleanly because of a 
nonzero "MaxRequestsPerChild" setting or for some other reason), in 
which case you'll be stuck with fewer.  The parent process probably 
shouldn't exit in this manner, but that's probably another discussion 
for another forum.  Regardless, sysvsem locking will not work with more 
than 30 processes without tuning.

The problem is that by default, Solaris (at least through Solaris 8, I 
haven't yet tried 9) is configured to support a maximum of 30 semaphore 
undo structures.  Each child calls semop in accept_mutex_on_system with 
sem_flags & SEM_UNDO, and therefore accounts for one such structure.  
When the 30th child calls semop, it fails with ENOSPC (see "man semop") 
and the parent exits.  This is obviously insufficient.

By tuning the seminfo_semmnu parameter in the semsys driver, it is 
possible to avoid this problem.  Adding the following to /etc/system 
and rebooting will increase the number of undo structures from its 
default of 30:

set semsys:seminfo_semmnu = 1024

It might be useful if this information was included in htdocs/manual/
misc/perf-tuning.html or htdocs/manual/misc/performance.html.

On a related note, htdocs/manual/misc/perf-tuning.html indicates that 
"AcceptMutex pthread" only gives appropriate behavior under some 
Solaris configurations.  Which configurations are these?  Is this 
information even current?  The document refers somewhat incorrectly to 
the HAVE_*_SERIALIZED_ACCEPT macros in a way that makes it unclear 
whether or not it's been updated aside from a quick :s///g.  Using the 
pthread interface for accept serialization would avoid some of the 
drawbacks of semaphores, and could speed up the locking.

Mark

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: "AcceptMutex sysvsem" on high-volume Solaris servers

[Jeff Beard <je...@cyberxape.com>]

You should probably post this up on the httpd docs list. In fact, if 
you're really interested in having this information include, DIY then 
post a diff.

--Jeff

Mark Mentovai wrote:
> This information applies to Apache running on Solaris with AcceptMutex 
> set to sysvsem, and offers a method to improve Apache's performance.  
> It may apply to other systems as well, if you replace some of the 
> particulars (like "/etc/system") with values that make sense.
> 
> On Solaris, Apache makes available three methods for accept 
> serialization: fcntl, sysvsem, and pthread.  "AcceptMutex fcntl" is the 
> default and "safest" method in that is most likely to function as 
> expected, but is comparitively a poor performer.  Using System 5 
> semaphores by specifying "AcceptMutex sysvsem" offers superior 
> performance, but as discussed in the past, does not function on high-
> volume servers.  (See htdocs/manual/misc/perf-tuning.html for a 
> discussion of accept serialization.)
> 
> On an untuned Solaris system, configuring "AcceptMutex sysvsem" will 
> eventually cause Apache to exit, leaving the log peppered with:
> 
> accept_mutex_on: No space left on device
> [Mon Jul  8 12:23:24 2002] [alert] Child 2319 returned a Fatal error... 
> Apache is exiting!
> 
> This occurs once 30 child processes are present.  It causes the parent 
> to exit, leaving 30 children without any pool size regulation: you're 
> stuck with 30, unless the children exit (either cleanly because of a 
> nonzero "MaxRequestsPerChild" setting or for some other reason), in 
> which case you'll be stuck with fewer.  The parent process probably 
> shouldn't exit in this manner, but that's probably another discussion 
> for another forum.  Regardless, sysvsem locking will not work with more 
> than 30 processes without tuning.
> 
> The problem is that by default, Solaris (at least through Solaris 8, I 
> haven't yet tried 9) is configured to support a maximum of 30 semaphore 
> undo structures.  Each child calls semop in accept_mutex_on_system with 
> sem_flags & SEM_UNDO, and therefore accounts for one such structure.  
> When the 30th child calls semop, it fails with ENOSPC (see "man semop") 
> and the parent exits.  This is obviously insufficient.
> 
> By tuning the seminfo_semmnu parameter in the semsys driver, it is 
> possible to avoid this problem.  Adding the following to /etc/system 
> and rebooting will increase the number of undo structures from its 
> default of 30:
> 
> set semsys:seminfo_semmnu = 1024
> 
> It might be useful if this information was included in htdocs/manual/
> misc/perf-tuning.html or htdocs/manual/misc/performance.html.
> 
> On a related note, htdocs/manual/misc/perf-tuning.html indicates that 
> "AcceptMutex pthread" only gives appropriate behavior under some 
> Solaris configurations.  Which configurations are these?  Is this 
> information even current?  The document refers somewhat incorrectly to 
> the HAVE_*_SERIALIZED_ACCEPT macros in a way that makes it unclear 
> whether or not it's been updated aside from a quick :s///g.  Using the 
> pthread interface for accept serialization would avoid some of the 
> drawbacks of semaphores, and could speed up the locking.
> 
> Mark
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org