You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apr.apache.org by Joe Orton <jo...@redhat.com> on 2004/06/14 12:41:11 UTC
RFC: posixsem/pthread mutexes considered harmful
Having taken the time to review some of the proc mutex code, it seems
both the process-shared pthread mutexes and POSIX semaphores differ from
the other locking types on Unix in two significant ways:
1) for both types, if a process segfaults whilst holding the lock, the
mutex is left in an undefined (i.e. probably locked) state [1]. For the
other lock types, the mutex is unlocked and safe to use.
2) destroying a mutex in one process leaves it in an undefined state in
the other processes which forked and inherited the apr_proc_mutex_t.
For the other lock types behaviour varies but is well-defined.
This trade-off against better performance for these two types doesn't
seem acceptable: e.g. for httpd/worker, a module segfault in a single
thread can cause a complete denial of service if the listener thread in
that process has the accept mutex locked.
So, I'm proposing that _POSIXSEM or _PROC_PTHREAD should never be made
the default locking mechanism. Nothing to stop those who understand the
trade-off making an informed choice, of course.
joe
[1] supposedly on Solaris, this is solved for pthread mutexes by setting
the "robust" attribute, but evidence in bugzilla from at least one
tester was that mod_include segfaults using worker on Solaris could hang
the server.
Re: RFC: posixsem/pthread mutexes considered harmful
Posted by Jeff Trawick <tr...@attglobal.net>.
Joe Orton wrote:
> On Mon, Jun 14, 2004 at 08:22:41AM -0400, Jeff Trawick wrote:
>
>>Joe Orton wrote:
>>
>>
>>>So, I'm proposing that _POSIXSEM or _PROC_PTHREAD should never be made
>>>the default locking mechanism. Nothing to stop those who understand the
>>>trade-off making an informed choice, of course.
>>
>>generally +1, though I'm concerned that Solaris may not have a good default
>>due to the sysdef+reboot requirements which are often required with the
>>alternative mechanisms
>
>
> Ah, good stuff, I forgot about that. That undo structure limit is
> suprisingly low by default as well IIRC, so maybe the Right Thing is to
> restore the fcntl default for Solaris as per 1.3? My preference to
> commit below - thanks Jeff.
>
> Index: configure.in
> ===================================================================
> RCS file: /home/cvs/apr/configure.in,v
> retrieving revision 1.588
> diff -u -r1.588 configure.in
> --- configure.in 14 Jun 2004 08:53:31 -0000 1.588
> +++ configure.in 14 Jun 2004 14:19:38 -0000
> @@ -1597,8 +1597,10 @@
> # See which lock mechanism we'll select by default on this system.
> # The last APR_DECIDE to execute sets the default.
> # At this stage, we match the ordering in Apache 1.3
> -# which is (highest to lowest): pthread -> posixsem -> sysvsem -> fcntl -> flock
> -#
> +# which is (highest to lowest): sysvsem -> fcntl -> flock.
> +# POSIX semaphores and cross-process pthread mutexes are not
> +# used by default since they have less desirable behaviour when
> +# e.g. a process holding the mutex segfaults.
+1
(though I don't see why we should rely on such a decision tree for platforms
with which we have a lot of experience, such as Linux)
> Index: build/apr_hints.m4
> ===================================================================
> RCS file: /home/cvs/apr/build/apr_hints.m4,v
> retrieving revision 1.62
> diff -u -r1.62 apr_hints.m4
> --- build/apr_hints.m4 11 Jun 2004 15:00:21 -0000 1.62
> +++ build/apr_hints.m4 14 Jun 2004 14:19:38 -0000
> @@ -198,6 +198,7 @@
> *-solaris2*)
> PLATOSVERS=`echo $host | sed 's/^.*solaris2.//'`
> APR_ADDTO(CPPFLAGS, [-DSOLARIS2=$PLATOSVERS -D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT])
> + APR_SETIFNULL(apr_lock_method, [USE_FCNTL_SERIALIZE])
+1
Re: RFC: posixsem/pthread mutexes considered harmful
Posted by Joe Orton <jo...@redhat.com>.
On Mon, Jun 14, 2004 at 08:22:41AM -0400, Jeff Trawick wrote:
> Joe Orton wrote:
>
> >So, I'm proposing that _POSIXSEM or _PROC_PTHREAD should never be made
> >the default locking mechanism. Nothing to stop those who understand the
> >trade-off making an informed choice, of course.
>
> generally +1, though I'm concerned that Solaris may not have a good default
> due to the sysdef+reboot requirements which are often required with the
> alternative mechanisms
Ah, good stuff, I forgot about that. That undo structure limit is
suprisingly low by default as well IIRC, so maybe the Right Thing is to
restore the fcntl default for Solaris as per 1.3? My preference to
commit below - thanks Jeff.
Index: configure.in
===================================================================
RCS file: /home/cvs/apr/configure.in,v
retrieving revision 1.588
diff -u -r1.588 configure.in
--- configure.in 14 Jun 2004 08:53:31 -0000 1.588
+++ configure.in 14 Jun 2004 14:19:38 -0000
@@ -1597,8 +1597,10 @@
# See which lock mechanism we'll select by default on this system.
# The last APR_DECIDE to execute sets the default.
# At this stage, we match the ordering in Apache 1.3
-# which is (highest to lowest): pthread -> posixsem -> sysvsem -> fcntl -> flock
-#
+# which is (highest to lowest): sysvsem -> fcntl -> flock.
+# POSIX semaphores and cross-process pthread mutexes are not
+# used by default since they have less desirable behaviour when
+# e.g. a process holding the mutex segfaults.
APR_BEGIN_DECISION([apr_lock implementation method])
APR_IFALLYES(func:flock define:LOCK_EX,
APR_DECIDE(USE_FLOCK_SERIALIZE, [4.2BSD-style flock()]))
@@ -1606,14 +1608,6 @@
APR_DECIDE(USE_FCNTL_SERIALIZE, [SVR4-style fcntl()]))
APR_IFALLYES(func:semget func:semctl define:SEM_UNDO,
APR_DECIDE(USE_SYSVSEM_SERIALIZE, [SysV IPC semget()]))
-APR_IFALLYES(header:semaphore.h func:sem_open func_sem_close dnl
- func_sem_unlink func:sem_post func_sem_wait,
- APR_DECIDE(USE_POSIXSEM_SERIALIZE, [POSIX sem_open()]))
-# note: the current APR use of shared mutex requires /dev/zero
-APR_IFALLYES(header:pthread.h define:PTHREAD_PROCESS_SHARED dnl
- func:pthread_mutexattr_setpshared dnl
- file:/dev/zero,
- APR_DECIDE(USE_PROC_PTHREAD_SERIALIZE, [pthread mutex]))
if test "x$apr_lock_method" != "x"; then
APR_DECISION_FORCE($apr_lock_method)
fi
Index: build/apr_hints.m4
===================================================================
RCS file: /home/cvs/apr/build/apr_hints.m4,v
retrieving revision 1.62
diff -u -r1.62 apr_hints.m4
--- build/apr_hints.m4 11 Jun 2004 15:00:21 -0000 1.62
+++ build/apr_hints.m4 14 Jun 2004 14:19:38 -0000
@@ -198,6 +198,7 @@
*-solaris2*)
PLATOSVERS=`echo $host | sed 's/^.*solaris2.//'`
APR_ADDTO(CPPFLAGS, [-DSOLARIS2=$PLATOSVERS -D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT])
+ APR_SETIFNULL(apr_lock_method, [USE_FCNTL_SERIALIZE])
;;
*-sunos4*)
APR_ADDTO(CPPFLAGS, [-DSUNOS4])
Re: RFC: posixsem/pthread mutexes considered harmful
Posted by Jeff Trawick <tr...@attglobal.net>.
Joe Orton wrote:
> So, I'm proposing that _POSIXSEM or _PROC_PTHREAD should never be made
> the default locking mechanism. Nothing to stop those who understand the
> trade-off making an informed choice, of course.
generally +1, though I'm concerned that Solaris may not have a good default due
to the sysdef+reboot requirements which are often required with the alternative
mechanisms
> [1] supposedly on Solaris, this is solved for pthread mutexes by setting
> the "robust" attribute, but evidence in bugzilla from at least one
> tester was that mod_include segfaults using worker on Solaris could hang
> the server.
Yes, this cannot be considered solved by any stretch of the imagination.
Somehow the "robust" logic is prone to fail in production environments even
though it always works beautifully when I'm trussing the child processes to see
what happens. I've seen this failure at multiple customer sites.
But on the other hand Solaris has a special system configuration concern:
With a default Solaris install, SysV sems often fail, particularly with
unthreaded server, until tuning is made (not enough undo structures) and the
system rebooted.
With a default Solaris install, the default limit of outstanding fcntl lock
requests is 512 and requires a reboot to increase... this limit seems to be
system-wide across all locks. Less likely to hit this limit than the SysV sem
undo structure limit I suppose.
I guess the bright side is that we're less likely to hit these limits with a
threaded web server, and a threaded web server is where we have the good chance
of segfaulting while holding the accept mutex.
Any other info about sysvsem/fcntl concerns on Solaris?
I'm +0.5 for moving to a safer mutex mechanism on Solaris.
--/--
Meanwhile, we have the capability to pre-select the default lock method in
apr_hints.m4 where we have system-specific knowledge about what ought to be
used. No reason to leave the choice up to the whims of people trying to make
global decisions on mutex mechanisms based on various coolness factors when
there is a clear winner for the platform. For example, I'd be happy to see
APR_SETIFNULL(apr_lock_method, [USE_SYSVSEM_SERIALIZE])
in apr_hints.m4 for Linux.
Re: robust mutexes (was: RFC: posixsem/pthread mutexes considered harmful)
Posted by Joe Orton <jo...@redhat.com>.
On Tue, Jun 20, 2006 at 01:36:21AM +0900, Tsuyoshi SASAMOTO wrote:
> >[1] supposedly on Solaris, this is solved for pthread mutexes by setting
> >the "robust" attribute, but evidence in bugzilla from at least one
> >tester was that mod_include segfaults using worker on Solaris could hang
> >the server.
>
> I guess this was caused by bug 39833 (now fixed in rev. 415267 & 415268), that is,
> the configure script didn't detect usability of the robust attribute correctly,
> so robust mutexes weren't used.
> With the fix, robust mutexes will be used and should work.
Ah, no, those two things are not related - the configure code you fixed
is new this year (added in r382030) whereas the discussion you reference
happened two years ago.
joe
robust mutexes (was: RFC: posixsem/pthread mutexes considered harmful)
Posted by Tsuyoshi SASAMOTO <na...@miobox.jp>.
>[1] supposedly on Solaris, this is solved for pthread mutexes by setting
>the "robust" attribute, but evidence in bugzilla from at least one
>tester was that mod_include segfaults using worker on Solaris could hang
>the server.
I guess this was caused by bug 39833 (now fixed in rev. 415267 & 415268), that is,
the configure script didn't detect usability of the robust attribute correctly,
so robust mutexes weren't used.
With the fix, robust mutexes will be used and should work.
Tsuyoshi SASAMOTO
nazonazo@miobox.jp