[users@httpd] Root cause for drop in connections being served - performance issue or normal queueing behaviour?

[Marc Peters <ko...@gmail.com>]

Dear Apache user Community,

I searched apache-related user forums but still could not find a
comprehensible explanation whether the following behaviour was caused
by the maxclient-setting being reached & subsequent queueing or
performance issues of the webserver. The attached diagramm is a
visualization of the httpd-access log.

It shows the amount of requests per httpd-code per minute.

The webserver experienced a sharp rise in requests until it hit 3000
requests per minute. Then suddenly there
is a dramatic drop in served requests and for about two hours the
number of requests is low and there are 503 errors.
After two hours the number of requests rises again until it slowly fades out.

In the error log there is only one reference to maxclients:
[Mon Apr 15 23:50:44 2012][error] server reached MaxClients setting,
consider raising the MaxClients setting

In the access log at this time there appear 503-errors for about 2
hours, and the total amount of requests
in the protocoll is significantly lower than before. After two hours
the 503-errors disappear and the amount of requests
grows again.

What is the root cause in sudden drop of served requests?

In the apache2.conf MaxClients is sent to 10, MaxThreadsPerChild 25.

Is this caused by MaxClients being reached and subsequent requests being queued?
Is this caused by the server/hardware being unable to handle the load
of requests?
How often is Apache writing a MaxClients-error in its logfile - just
once or for every conncetion exceeding
the parameter setting?

Kind regards,

Marc Peters