more for perf.html

[Dean Gaudet <dg...@arctic.org>]

Here's the latest changes I've made to
<http://www.arctic.org/~dgaudet/apache/perf>. 

Dean

--- perf.html	1997/09/12 22:41:05	1.2
+++ perf.html	1997/09/23 07:41:46	1.4
@@ -38,9 +38,7 @@
 enough".  This causes users to hit stop and reload, further increasing
 the load.  You can, and should, control the <code>MaxClients</code>
 setting so that your server does not spawn so many children it starts
-swapping.  A large <code>KeepAliveTimeout</code> can lock up many children
-longer than necessary, setting it higher than 60 seconds is not
-recommended (see also <a href="ftp://ds.internic.net/internet-drafts/draft-ietf-http-connection-00.txt">draft-ietf-http-connection-00.txt</a>).
+swapping.
 
 <p>Beyond that the rest is mundane:  get a fast enough CPU, a fast enough
 network card, and fast enough disks, where "fast enough" is something
@@ -149,11 +147,63 @@
 
 where you list the most common choice first.
 
+<h4>Process Creation</h4>
+
+<p>Prior to Apache 1.3 the <code>MinSpareServers</code>,
+<code>MaxSpareServers</code>, and <code>StartServers</code> settings
+all had drastic effects on benchmark results.  In particular, Apache
+required a "ramp-up" period in order to reach a number of children
+sufficient to serve the load being applied.  After the initial
+spawning of <code>StartServers</code> children, only one child per
+second would be created to satisfy the <code>MinSpareServers</code>
+setting.  So a server being accessed by 100 simultaneous clients,
+using the default <code>StartServers</code> of 5 would take on
+the order 95 seconds to spawn enough children to handle the load.  This
+works fine in practice on real-life servers, because they aren't restarted
+frequently.  But does really poorly on benchmarks which might only run
+for ten minutes.
+
+<p>The one-per-second rule was implemented in an effort to avoid
+swamping the machine with the startup of new children.  If the machine
+is busy spawning children it can't service requests.  But it has such
+a drastic effect on the perceived performance of Apache that it had
+to be replaced.  As of Apache 1.3,
+the code will relax the one-per-second rule.  It
+will spawn one, wait a second, then spawn two, wait a second, then spawn
+four, and it will continue exponentially until it is spawning 32 children
+per second.  It will stop whenever it satisfies the
+<code>MinSpareServers</code> setting.
+
+<p>This appears to be responsive enough that it's
+almost unnecessary to twiddle the <code>MinSpareServers</code>,
+<code>MaxSpareServers</code> and <code>StartServers</code> knobs.  When
+more than 4 children are spawned per second, a message will be emitted
+to the <code>ErrorLog</code>.  If you see a lot of these errors then
+consider tuning these settings.  Use the <code>mod_status</code> output
+as a guide.
+
+<p>Related to process creation is process death induced by the
+<code>MaxRequestsPerChild</code> setting.  By default this is 30, which
+is probably far too low unless your server is using a module such as
+<code>mod_perl</code> which causes children to have bloated memory
+images.  If your server is serving mostly static pages then consider
+raising this value to something like 10000.  The code is robust enough
+that this shouldn't be a problem.
+
+<p>When keep-alives are in use, children will be kept busy
+doing nothing waiting for more requests on the already open
+connection.  The default <code>KeepAliveTimeout</code> of
+15 seconds attempts to minimise this effect.  The tradeoff
+here is between network bandwidth and server resources.
+In no event should you raise this above about 60 seconds, as <a
+href="http://www.research.digital.com/wrl/techreports/abstracts/95.4.html">
+most of the benefits are lost</a>.
+
 <h3>Compile-Time Configuration Issues</h3>
 
 <h4>mod_status and Rule STATUS=yes</h4>
 
-<p>This is a compile time issue.  If you include <code>mod_status</code>
+<p>If you include <code>mod_status</code>
 and you also set <code>Rule STATUS=yes</code> when building
 Apache, then on every request Apache will perform two calls to
 <code>gettimeofday(2)</code> (or <code>times(2)</code> depending
@@ -204,8 +254,10 @@
 that multiple children execute this loop at the same time, and so multiple
 children will block at <code>select</code> when they are in between
 requests.  All those blocked children will awaken and return from
-<code>select</code> when a single request appears on any socket.  Note
-that they will all then fall down into the loop and try to <code>accept</code>
+<code>select</code> when a single request appears on any socket
+(the number of children which awaken varies depending on the operating
+system and timing issues).
+They will all then fall down into the loop and try to <code>accept</code>
 the connection.  But only one will succeed (assuming there's still only
 one connection ready), the rest will be <i>blocked</i> in <code>accept</code>.
 This effectively locks those children into serving requests from that
@@ -407,7 +459,9 @@
 disable this feature you can define <code>NO_LINGCLOSE</code>, but
 this is not recommended at all.  In particular, as HTTP/1.1 pipelined
 persistant connections come into use <code>lingering_close</code>
-is an absolute necessity (and pipelined connections are faster, so you
+is an absolute necessity (and
+<a href="http://www.w3.org/Protocols/HTTP/Performance/Pipeline.html">
+pipelined connections are faster</a>, so you
 want to support them).
 
 <h4>Scoreboard File</h4>
@@ -511,7 +565,8 @@
 This is caused by the implementation of graceful restarts.  When the
 parent receives a <code>SIGUSR1</code> it sends a <code>SIGUSR1</code>
 to all of its children (and it also increments a "generation counter"
-in shared memory).  Any children that are idle will immediately die
+in shared memory).  Any children that are idle (between connections)
+will immediately die
 off when they receive the signal.  Any children that are in keep-alive
 connections, but are in between requests will die off immediately.  But
 any children that have a connection and are still waiting for the first
@@ -696,3 +751,47 @@
 
 That's 19 system calls, of which 4 remain relatively easy to remove,
 but don't seem worth the effort.
+
+<h3>Appendix: The Pre-Forking Model</h3>
+
+<p>Apache (on Unix) is a <i>pre-forking</i> model server.  The
+<i>parent</i> process is responsible only for forking <i>child</i>
+processes, it does not serve any requests or service any network
+sockets.  The child processes actually process connections, they serve
+multiple connections (one at a time) before dying.
+The parent spawns new or kills off old
+children in response to changes in the load on the server (it does so
+by monitoring a scoreboard which the children keep up to date).
+
+<p>This model for servers offers a robustness that other models do
+not.  In particular, the parent code is very simple, and with a high
+degree of confidence the parent will continue to do its job without
+error.  The children are complex, and when you add in third party
+code via modules, you risk segmentation faults and other forms of
+corruption.  Even should such a thing happen, it only affects one
+connection and the server continues serving requests.  The parent
+quickly replaces the dead child.
+
+<p>Pre-forking is also very portable across dialects of Unix.
+Historically this has been an important goal for Apache, and it continues
+to remain so.
+
+<p>The pre-forking model comes under criticism for various
+performance aspects.  Of particular concern are the overhead
+of forking a process, the overhead of context switches between
+processes, and the memory overhead of having multiple processes.
+Furthermore it does not offer as many opportunities for data-caching
+between requests (such as a pool of <code>mmaped</code> files).
+Various other models exist and extensive analysis can be found in the
+<a href="http://www.cs.wustl.edu/~jxh/research/research.html"> papers
+of the JAWS project</a>.  In practice all of these costs vary drastically
+depending on the operating system.
+
+<p>Apache's core code is already multithreaded aware, and Apache version
+1.3 is multithreaded on NT.  There have been at least two other experimental
+implementations of threaded Apache (one using the 1.3 code base on DCE,
+and one using a custom user-level threads package and the 1.0 code base,
+neither are available publically).  Part of our redesign for version 2.0
+of Apache will include abstractions of the server model so that we
+can continue to support the pre-forking model, and also support various
+threaded models.