[users@httpd] send mmap timed out

[Martijn <sw...@googlemail.com>]

Hello.

This is a bit of a long shot. I've done a bit of Google-search, but
that only gives a small number of hits, most of which are unanswered
questions on mailing lists or message boards.

Our web server occasionally gives the following 'errors' in the error_log file:

[Mon Apr 23 12:41:40 2007] [info] [client 12.34.56.78] send mmap timed out

and, more frequently:

[Mon Apr 23 12:41:40 2007] [info] [client 12.34.56.78] (32)Broken
pipe: client stopped connection before send mmap completed

This usually isn't a problem and it never worried me (it is, after
all, marked as "info"), until a number of the former errors, followed
by a number of the latter errors (about 30 of each) caused the server
to crash. The errors were all related to the same IP address and, as
far as I can see, this belongs to a genuine user.

I did, however, find in the access log files that their results
returned httpd status '206' (partial content) and the number of bytes
on large files, was always a nice rounded number, like 32768 (which is
2^15). It suggests that the problem has to do with the user
downloading only parts of larger files (pdfs/gifs) and, I would guess,
leaving an httpd process waiting for it to finish? At least the number
of httpd processes had reached its maximum (which is why the server
crashed).

But I don't know if my guesses are correct. And even if they are, what
has really happened (for instance: is this really a problem with the
client or with the server). And, of course, how to make sure this
won't happen again.

Martijn

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] send mmap timed out

[Martijn <sw...@googlemail.com>]

Sorry for taking so long to reply. I had given the problem a lower
priority, assuming (hoping) it was just one of these mysterious
incidents that only occurs once, when -of course- it happened again.
And -of course- again when I wasn't looking.

> Was this just a denial-of-service attack? Did you use a tool like
> netstat to identify the client connections and track down the origin?

No. I have scrutinized the access and error log files several times,
and I don't see anything suspicious: no sudden increase of traffic, no
increase in requests to specific files, not even more errors than
usual. (Apart from the one in the subject, but I've started to become
more convinced that that isn't the problem. Today, it had occurred
only once just before the problems started.)

Only the number of httpd processes increases suddenly: I have been
monitoring the number of processes on a minute by minute basis, and it
really is normal one minute and at its maximum the next one. I had
been thinking it might be an infinite loop somewhere in a script, but
then why would this script be called so many times at once? And why
doesn't the error occur most of the time? And why doesn't Apache
realise that there is a problem and 'kill' some processes itself?

Questions.. questions... and I realise I probably am not giving a lot
of clues that might lead to a solution. But really, if only I had such
clues...

But perhaps someone here knows what it could be. And if not, thanks
for your time anyway. :)

Martijn.

PS a little bit form httpd.conf:

Timeout 300

KeepAlive Off
MaxKeepAliveRequests 100
KeepAliveTimeout 5

MinSpareServers 25
MaxSpareServers 30
StartServers 30
MaxClients 85
MaxRequestsPerChild 10000

(I just realised that I had changed MaxClients fro 75 to 85 the other
day -and restarted apache twice since- but the number of processes
still doesn't get higher than 75. Weird.)

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] send mmap timed out

[Joshua Slive <jo...@slive.ca>]

On 4/23/07, Martijn <sw...@googlemail.com> wrote:
> > In all likelihood, those messages are only tangentially related to
> > your problem. They simply indicate that the client end of the
> > connection disappeared, which is a perfectly normal thing for a busy
> > webserver. The 206 responses are probably caused by a very impolite
> > download accelerator making multiple overlapping requests. When it has
> > received what it wants, it drops all the requests, resulting in the
> > error messages.
> >
> > But then you say the server "crashed". What exactly does that mean?
> > Did it dump core? Did it simply get very slow? It is not normal for a
> > server to "crash" when it hits MaxClients. You need to be more
> > detailed about what actually happened.
>
> Thank you for your reply.
>
> It did not just reach MaxClients, but the number of httpd processes
> stayed at this maximum for hours, until someone (not me) finally
> restarted apache. (Of course, this problem happened on Sunday night,
> so I wasn't there to see what really happened. Or to sort out things
> myself.) Both the access_log file and the error_log file weren't
> written to from the time the aforementioned mmap-info messages
> appeared until apache was restarted hours later. There is, indeed, no
> sign that apache really 'crashed', so it probably, didn't but the
> effect was the same. People who were awake at that time told me that
> they couldn't reach the site, but the server did ping.

Was this just a denial-of-service attack? Did you use a tool like
netstat to identify the client connections and track down the origin?

If apache was really "frozen" (not serving a connection and not
answering new connections, even after the Timeout has expired), then
you should attach to some process to get a backtrace:
http://httpd.apache.org/dev/debugging.html#backtrace

Joshua.

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] send mmap timed out

[Martijn <sw...@googlemail.com>]

> In all likelihood, those messages are only tangentially related to
> your problem. They simply indicate that the client end of the
> connection disappeared, which is a perfectly normal thing for a busy
> webserver. The 206 responses are probably caused by a very impolite
> download accelerator making multiple overlapping requests. When it has
> received what it wants, it drops all the requests, resulting in the
> error messages.
>
> But then you say the server "crashed". What exactly does that mean?
> Did it dump core? Did it simply get very slow? It is not normal for a
> server to "crash" when it hits MaxClients. You need to be more
> detailed about what actually happened.

Thank you for your reply.

It did not just reach MaxClients, but the number of httpd processes
stayed at this maximum for hours, until someone (not me) finally
restarted apache. (Of course, this problem happened on Sunday night,
so I wasn't there to see what really happened. Or to sort out things
myself.) Both the access_log file and the error_log file weren't
written to from the time the aforementioned mmap-info messages
appeared until apache was restarted hours later. There is, indeed, no
sign that apache really 'crashed', so it probably, didn't but the
effect was the same. People who were awake at that time told me that
they couldn't reach the site, but the server did ping.

I hope this helps.

Cheers.

Martijn

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org

Re: [users@httpd] send mmap timed out

[Joshua Slive <jo...@slive.ca>]

On 4/23/07, Martijn <sw...@googlemail.com> wrote:

> [Mon Apr 23 12:41:40 2007] [info] [client 12.34.56.78] send mmap timed out
>
> and, more frequently:
>
> [Mon Apr 23 12:41:40 2007] [info] [client 12.34.56.78] (32)Broken
> pipe: client stopped connection before send mmap completed
>
> This usually isn't a problem and it never worried me (it is, after
> all, marked as "info"), until a number of the former errors, followed
> by a number of the latter errors (about 30 of each) caused the server
> to crash. The errors were all related to the same IP address and, as
> far as I can see, this belongs to a genuine user.
>
> I did, however, find in the access log files that their results
> returned httpd status '206' (partial content) and the number of bytes
> on large files, was always a nice rounded number, like 32768 (which is
> 2^15). It suggests that the problem has to do with the user
> downloading only parts of larger files (pdfs/gifs) and, I would guess,
> leaving an httpd process waiting for it to finish? At least the number
> of httpd processes had reached its maximum (which is why the server
> crashed).

In all likelihood, those messages are only tangentially related to
your problem. They simply indicate that the client end of the
connection disappeared, which is a perfectly normal thing for a busy
webserver. The 206 responses are probably caused by a very impolite
download accelerator making multiple overlapping requests. When it has
received what it wants, it drops all the requests, resulting in the
error messages.

But then you say the server "crashed". What exactly does that mean?
Did it dump core? Did it simply get very slow? It is not normal for a
server to "crash" when it hits MaxClients. You need to be more
detailed about what actually happened.

Joshua.

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org