Serialising accepts (was Re: apache_0.7.3h comments)

[David Robinson <dr...@ast.cam.ac.uk>]

>From the Sun guy:
} On Solaris, you need to use it for single processors also and I suspect in
} all library based Socket interface implementations like Unixware etc.
      ~~~~~~~~~~~~~~~~~~~~
} The reason is that accept() is not an atomic operation but just a call in
} the ABI and any other call to the same fd at the same time can cause
} unpredictable behaviour.

In other words, relying on the kernel to serialise accepts may simply not
work on several unixes, not just multiprocessor OSes. For example, I would
suspect Linux might fall into this case. (Hence some of the problems reported.)

I think we should bite the bullet and admit that use of multiple accepts is not
portable, and that we have to re-work it to explicitly serialise the accepts
in a portable manner.

The alternative is to have ad hoc serialisation for each OS that needs it,
possibly in a manner that is OS minor-version-number specific. For these
OS's (and how will we know if an OS breaks until the users complain?) the
ad hoc serialisation will probably be no faster than some portable schemes
we could implement. For example, lockf() is actually implemented via IPC
to a lockd daemon; IPC to the parent httpd would not be any slower.

So why not have a queue managed by the parent of the children waiting
to accept? Queue management occurs after a child has serviced a request,
and so does not impact on the server response time. (Unlike the NCSA approach,
where it is done between accept() returning and the child being passed the
file-descriptor.)

So:
 The parent has a pipe to each child.
 It sends a short message to an idle child saying: 'you accept next'.
 On completing a request a child sends a message to the parent saying
 'I am idle', and waits for a response message.
 The parent has some algorithm for deciding which idle child should be
 given the next connection. Round-robin would be cache-unfriendly; instead,
 send it to the most recently used child.

 David.