[PATCH] sys_epoll Support for apr_pollset

[Paul Querna <ch...@force-elite.com>]

Hello, 
	Attached is a patch that implements a sys_epoll backend for
apr_pollset_*. For some info on epoll in general check out:
http://lse.sourceforge.net/epoll/
http://www.xmailserver.org/linux-patches/nio-improve.html

I had to change one of the tests to make them all pass.  In the original
test it had an invalid requirement on the order of the result set.  The
modified test allows it to pass if result set comes back in a different
order.

The code should work on a Linux Machine with a 2.6 kernel, and an
updated glibc.  I have been able to test it with Lunar-Linux on x86, and
I will hopefully test it on Sparc64/Gentoo Linux later tonight.

Questions and comments are welcome, as this is the first code I have
written for epoll :)

-Paul Querna

Re: [PATCH] sys_epoll Support for apr_pollset

[Joe Orton <jo...@manyfish.co.uk>]

On Sun, May 30, 2004 at 02:52:08PM -0700, Paul Querna wrote:
> On Sun, 2004-05-30 at 13:26 +0100, Joe Orton wrote:
> > Interesting... some nits:
> > 
> > - C++ // comments bad ;)
> > - CPP #directives must not have leading whitespace before the #
> 
> Fixed in the updated patch.

Good stuff...

> > - needs to cope with the fact that glibc implements epoll* returning 
> > ENOSYS or whatever on 2.4 kernels IIRC, via autoconf or runtime checks
> 
> I would prefer doing this in autoconf, unless people are willing to
> accept a much larger patch that could make the pollset back end
> selectable at runtime.  The other issue with this is if APR was built on
> a machine running 2.6, it might not be usable on a machine with 2.4.
> That would be the other advantage of making the pollset back end runtime
> selectable.  

Yeah, this kind of thing is a little annoying, but autoconf checks are
really the only way, indeed.  I guess by the time there's a stable httpd
release based on APR 1.0, the 2.6 kernel will be old hat anyway :)

> I did a simple benchmark with an epoll enabled HTTPd 2.1 yesterday.
> [mostly to see if it all worked correctly :) ]
> 
> It did a few transactions a second _faster_ than a normal poll() based
> HTTPd 2.1.  Not significantly better or worse for Apache's common case
> of not very many sockets.

Cool.

> Maybe later this week I can make up some more official benchmarks. 
> 
> > I wondered whether apr_poll() should remain using poll() with it's low
> > constant overhead, and the apr_pollset_* interface alone should use the
> > high-constant-overhead interfaces like epoll or /dev/poll?
> 
> Maybe.  I don't think poll's overhead is much less than epoll.  We will
> have to benchmark it to find a real answer.

Well, epoll is about scaling better; it means you can set up a
descriptor array with two thousand fds, and not have to copy that array
between kernel and userspace each time you call epoll_wait(), as you
would with poll().  That's the slow bit (AIUI anyway).

But for apr_poll(), that doesn't make any difference; you're setting up
a new fd array for each call into APR anyway.  So it may as well use
poll(); in fact, it is probably *worse* to use epoll for apr_poll(),
because it just has more overhead (with more system calls), and no
advantages.

Does that make sense?

joe

Re: [PATCH] sys_epoll Support for apr_pollset

[Paul Querna <ch...@force-elite.com>]

On Sun, 2004-05-30 at 13:26 +0100, Joe Orton wrote:
> Interesting... some nits:
> 
> - C++ // comments bad ;)
> - CPP #directives must not have leading whitespace before the #

Fixed in the updated patch.

> - needs to cope with the fact that glibc implements epoll* returning 
> ENOSYS or whatever on 2.4 kernels IIRC, via autoconf or runtime checks

I would prefer doing this in autoconf, unless people are willing to
accept a much larger patch that could make the pollset back end
selectable at runtime.  The other issue with this is if APR was built on
a machine running 2.6, it might not be usable on a machine with 2.4.
That would be the other advantage of making the pollset back end runtime
selectable.  

Another goal of mine is to make a KQueue() based back end for FreeBSD
sooner or later.

> An issue I mentioned before w.r.t. removal of apr_poll() in HEAD: this
> type of implementation will surely end up being slower than using poll()
> for polling on small numbers of fds (a not uncommon case), due to the
> fact that it requires three rather than one system call?  Some
> benchmarks might be interesting.

I did a simple benchmark with an epoll enabled HTTPd 2.1 yesterday.
[mostly to see if it all worked correctly :) ]

It did a few transactions a second _faster_ than a normal poll() based
HTTPd 2.1.  Not significantly better or worse for Apache's common case
of not very many sockets.

Maybe later this week I can make up some more official benchmarks. 

> I wondered whether apr_poll() should remain using poll() with it's low
> constant overhead, and the apr_pollset_* interface alone should use the
> high-constant-overhead interfaces like epoll or /dev/poll?

Maybe.  I don't think poll's overhead is much less than epoll.  We will
have to benchmark it to find a real answer.

-Paul Querna

Re: [PATCH] sys_epoll Support for apr_pollset

[Joe Orton <jo...@manyfish.co.uk>]

On Sat, May 29, 2004 at 03:04:58PM -0700, Paul Querna wrote:
> Hello, 
> 	Attached is a patch that implements a sys_epoll backend for
> apr_pollset_*. For some info on epoll in general check out:
> http://lse.sourceforge.net/epoll/
> http://www.xmailserver.org/linux-patches/nio-improve.html

Interesting... some nits:

- C++ // comments bad ;)
- CPP #directives must not have leading whitespace before the #
- needs to cope with the fact that glibc implements epoll* returning 
ENOSYS or whatever on 2.4 kernels IIRC, via autoconf or runtime checks

An issue I mentioned before w.r.t. removal of apr_poll() in HEAD: this
type of implementation will surely end up being slower than using poll()
for polling on small numbers of fds (a not uncommon case), due to the
fact that it requires three rather than one system call?  Some
benchmarks might be interesting.

I wondered whether apr_poll() should remain using poll() with it's low
constant overhead, and the apr_pollset_* interface alone should use the
high-constant-overhead interfaces like epoll or /dev/poll?

joe