FreeBSD and threads was Re: compile 0.24.2 errors

[Justin Erenkrantz <ju...@erenkrantz.com>]

--On Saturday, June 21, 2003 3:02 PM -0500 Ben Collins-Sussman 
<su...@collab.net> wrote:

> Not at all.  I've been building apache/subversion almost every day on
> FreeBSD for the last three years.  I started at FreeBSD 4.2, and I
> have gradually "built world" through every release, and now I'm
> running 4.8.  And I've always configured just like you, with
> --enable-threads forced for apache/apr/apu.  Never had a problem.

Apache httpd's threaded MPMs (i.e. worker) do not even serve pages correctly 
on FreeBSD.  This is due to a number of problems residing in the core of 
libc_r pertaining to network code.  The APR developers have decided that 
FreeBSD's threading code is not stable enough to be turned on by default.

To the best of our knowledge, we've isolated it to problems in the pthread 
condition variables used with libc_r.  We've heard reports from some FreeBSD 
kernel developers that some later versions of FreeBSD (i.e. next major release 
after 5) might have a rewritten libc_r that may address some of these problems.

From the reports we've heard, FreeBSD 4.8 and 5.x are still broken and can not 
serve pages.  But, I've heard rumors that the 'right' people in FreeBSD land 
are aware that libc_r is just broken and strongly discourage its use.

If Subversion uses any APR thread-level locking code (which I don't think it 
does), don't expect it to work on FreeBSD if libc_r is linked in.  -- justin

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: [OT] Re: FreeBSD and threads was Re: compile 0.24.2 errors

["Bruce A. Mah" <bm...@freebsd.org>]

If memory serves me right, Justin Erenkrantz wrote:
> --On Sunday, June 22, 2003 12:17 PM -0400 Greg Hudson <gh...@MIT.EDU> 
> wrote:

> The pthread_cond_signal calls appear to be delayed in this wakeup.  A 
> request can not be serviced until *another* request comes in.  This is the 
> core problem, but we were also able to trivially kernel panic FreeBSD by 
> placing a load on a threaded MPM.  (I personally haven't tried 5.x/-CURRENT 
> in a while, but we know 4.8 still doesn't work.  I have lost access to 
> boxes that can run -CURRENT.)
> 
> We'll note that the worker code works correctly, at least, on Linux, 
> Solaris, Darwin, and I believe, AIX.  So, we don't believe we're doing 
> anything massively wrong.  From a conversation with Alfred (one of the core 
> FreeBSD kernel guys), libc_r never really was tested with network code (why 
> a rewrite was thought to be necessary).  sendfile() and threads were broken 
> until we came along and were able to talk with Alfred to get it fixed.
> 
> One reason to keep this on dev@svn is that I'm aware that more FreeBSD 
> developers read this than dev@httpd.  ;-)  -- justin

Heh.  I'm "a" FreeBSD developer, but probably not the one you want.
I'm one of the release engineers (the poor sod who writes most of the
release notes) and the new maintainer of the subversion port in the
ports collection.  I'm doing the latter mostly because I have a vested
interest in having an up-to-date subversion, and I was committing most
of the changes for the old maintainer anyways.

IIRC, Alfred (Perlstein, right?) is not really a libc_r person, but
more of a sendfile person.  Dan Eischen is probably the guy you want
to ask about thread-related issues.  I can point you to him if you are
interested.

Someone mentioned a new libc_r in FreeBSD 5.X.  Beginning with 5.1,
there are three threading libraries in various stages of development.
There's:  1) libc_r (which is similar if not identical to 4.X).  This
is the default threading library that's in use when you first install
a system.  2) libkse.  This is the M:N threading package that uses the
work-in-progress Kernel Scheduled Entities support in kernel (similar
to scheduler activations).  Dan Eischen (mentioned above) is probably
the best contact for this.  3) libthr.  This is a 1:1 threading
package.  Mike Makonnen is the main person working on this now.

It's my personal opinion that eventually libkse will become the
default threads package, but at least for 5.1-RELEASE we (the RE team)
still consider libkse and libthr to be experimental.  (The main
concerns from the RE perspective are stability and cross-platform
portability.)  That having been said I've seen some fairly encouraging
reports with users running libkse on recent 5.1-HEAD snapshots
(post-release).

Although this isn't quite my area, I'd be happy to try to work with
you (or any other concerned parties) and the appropriate FreeBSD
developers to make this situation (whatever it is!) better.

Bruce.

[OT] Re: FreeBSD and threads was Re: compile 0.24.2 errors

[Justin Erenkrantz <ju...@erenkrantz.com>]

--On Sunday, June 22, 2003 12:17 PM -0400 Greg Hudson <gh...@MIT.EDU> wrote:

> server/mpm/worker/fdqueue.c's second use is also questionable; it
> re-checks its condition once and returns an "interrupted" error code if
> the condition is still false.  Depending on how callers handle such an
> error, that could also cause problems.

The problem is that wakeups don't occur correctly. 
(server/mpm/worker/fdqueue.c is the relevant one for this topic.) 
Specifically, the pthread_cond_wait calls only return after two 
pthread_cond_signals.  The pattern is this:

- Request comes in.
- Request pushed onto stack by 'main' thread
- This triggers pthread_cond_signal in the main thread
  (At this point, all 'workers' are idle and are in pthread_cond_wait.
   So, pthread_cond_signal should wake up at least *one* of them...but...)
- ...Nothing happens until...
- Another request comes in.
- Request pushed onto queue by 'main' thread
- Call pthread_cond_signal
- One worker thread *now* is woken up and services the first request
...repeat pattern...

The pthread_cond_signal calls appear to be delayed in this wakeup.  A request 
can not be serviced until *another* request comes in.  This is the core 
problem, but we were also able to trivially kernel panic FreeBSD by placing a 
load on a threaded MPM.  (I personally haven't tried 5.x/-CURRENT in a while, 
but we know 4.8 still doesn't work.  I have lost access to boxes that can run 
-CURRENT.)

We'll note that the worker code works correctly, at least, on Linux, Solaris, 
Darwin, and I believe, AIX.  So, we don't believe we're doing anything 
massively wrong.  From a conversation with Alfred (one of the core FreeBSD 
kernel guys), libc_r never really was tested with network code (why a rewrite 
was thought to be necessary).  sendfile() and threads were broken until we 
came along and were able to talk with Alfred to get it fixed.

One reason to keep this on dev@svn is that I'm aware that more FreeBSD 
developers read this than dev@httpd.  ;-)  -- justin

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: FreeBSD and threads was Re: compile 0.24.2 errors

[Greg Hudson <gh...@MIT.EDU>]

(Sorry for the off-topic continuation here.  Feel free to forward this
to a relevant Apache list.)

On Sat, 2003-06-21 at 21:26, Justin Erenkrantz wrote:
> To the best of our knowledge, we've isolated it to problems in the pthread 
> condition variables used with libc_r.  We've heard reports from some FreeBSD 
> kernel developers that some later versions of FreeBSD (i.e. next major release 
> after 5) might have a rewritten libc_r that may address some of these problems.

Interesting.

I found three failures to use condition variables in the httpd source,
one in server/mpm/experimental/leader/leader.c and two in
srclib/apr-util/misc/apr_queue.c.  These uses are not properly wrapped
in a while loop checking the condition, so a spurious wakeup (which is
allowed by the POSIX specification) could cause bugs.  Perhaps FreeBSD's
condition variable code sometimes generates spurious wakeups, and other
platforms do not.

server/mpm/worker/fdqueue.c's second use is also questionable; it
re-checks its condition once and returns an "interrupted" error code if
the condition is still false.  Depending on how callers handle such an
error, that could also cause problems.


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org