You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by "Victor J. Orlikowski" <v....@gte.net> on 2000/12/13 19:25:17 UTC
[PATCH] 1.3.x Solaris performance and SERIALIZED_ACCEPT
Another patch for the brave :)
Pthreads bring great performance on Solaris, and don't bog down on
multiprocessors; but they have subtle races associated with them, and
cause apache to fall over on Solaris after a while.
SvsV semaphores are also great performers, but must be tuned properly
by the admin on Solaris for large numbers of clients for Apache (am
working on a patch to have the server die if the tuning numbers aren't
correct for the MaxClients setting), or else the server goes south
too.
The performance of fcntl on Solaris with large numbers of clients is
abysmal.
The following is a patch to use solaris native mutexes for locking, to
improve performance on Solaris. They are slightly slower than pthread
mutexes, but do not misbehave if unlocked twice, eliminating the race
resulting from pthread mutexes. Further, they do not need any special
configuration, unlike SysV sems and like pthread mutexes.
(Yeah, I know. The code looks almost exactly like the pthread
code. Trust me, the mutexes are a *lot* better behaved).
Try it out, see if it works for you, and make sure there are no
stability problems (something we are trying to eliminate and still
keep performance on Solaris :)
Index: src/include/ap_config.h
===================================================================
RCS file: /cvs/apache/apache-1.3/src/include/ap_config.h,v
retrieving revision 1.298
diff -u -d -r1.298 ap_config.h
--- ap_config.h 2000/11/14 09:56:59 1.298
+++ ap_config.h 2000/12/13 18:12:29
@@ -183,6 +183,7 @@
#undef NO_SETSID
#define bzero(a,b) memset(a,0,b)
#if !defined(USE_SYSVSEM_SERIALIZED_ACCEPT) && \
+ !defined(USE_SOLARIS_SERIALIZED_ACCEPT) && \
!defined(USE_PTHREAD_SERIALIZED_ACCEPT)
#define USE_FCNTL_SERIALIZED_ACCEPT
#endif
Index: src/main/http_main.c
===================================================================
RCS file: /cvs/apache/apache-1.3/src/main/http_main.c,v
retrieving revision 1.516
diff -u -d -r1.516 http_main.c
--- http_main.c 2000/11/22 02:46:27 1.516
+++ http_main.c 2000/12/13 18:12:34
@@ -683,6 +683,99 @@
}
}
+#elif defined (USE_SOLARIS_SERIALIZED_ACCEPT)
+
+/* This code only works on Solaris ... but it works really fast
+ * on Solaris. Note that mutexes are *NOT* released when a task
+ * dies ... the task has to free it itself. So we block signals and
+ * try to be nice about releasing the mutex.
+ */
+
+#include <thread.h>
+#include <synch.h>
+
+static mutex_t *accept_mutex = (void *)(caddr_t) -1;
+static sigset_t accept_block_mask;
+static sigset_t accept_previous_mask;
+
+static void accept_mutex_child_cleanup(void *foo)
+{
+ if (accept_mutex != (void *)(caddr_t)-1) {
+ mutex_unlock(accept_mutex);
+ }
+}
+
+static void accept_mutex_child_init(pool *p)
+{
+ ap_register_cleanup(p, NULL, accept_mutex_child_cleanup, ap_null_cleanup);
+}
+
+static void accept_mutex_cleanup(void *foo)
+{
+ if (accept_mutex != (void *)(caddr_t)-1
+ && munmap((caddr_t) accept_mutex, sizeof(*accept_mutex))) {
+ perror("munmap");
+ }
+ accept_mutex = (void *)(caddr_t)-1;
+}
+
+static void accept_mutex_init(pool *p)
+{
+ int fd;
+
+ fd = open("/dev/zero", O_RDWR);
+ if (fd == -1) {
+ perror("open(/dev/zero)");
+ exit(APEXIT_INIT);
+ }
+ accept_mutex = (mutex_t *) mmap((caddr_t) 0, sizeof(*accept_mutex),
+ PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
+ if (accept_mutex == (void *) (caddr_t) - 1) {
+ perror("mmap");
+ exit(APEXIT_INIT);
+ }
+ close(fd);
+ if ((errno = mutex_init(accept_mutex, USYNC_PROCESS, NULL))) {
+ perror("mutex_init");
+ exit(APEXIT_INIT);
+ }
+ sigfillset(&accept_block_mask);
+ sigdelset(&accept_block_mask, SIGHUP);
+ sigdelset(&accept_block_mask, SIGTERM);
+ sigdelset(&accept_block_mask, SIGUSR1);
+ ap_register_cleanup(p, NULL, accept_mutex_cleanup, ap_null_cleanup);
+}
+
+static void accept_mutex_on(void)
+{
+ int err;
+
+ if (sigprocmask(SIG_BLOCK, &accept_block_mask, &accept_previous_mask)) {
+ perror("sigprocmask(SIG_BLOCK)");
+ clean_child_exit(APEXIT_CHILDFATAL);
+ }
+ if ((err = mutex_lock(accept_mutex))) {
+ errno = err;
+ perror("mutex_lock");
+ clean_child_exit(APEXIT_CHILDFATAL);
+ }
+}
+
+static void accept_mutex_off(void)
+{
+ int err;
+
+ if ((err = mutex_unlock(accept_mutex))) {
+ errno = err;
+ perror("mutex_unlock");
+ clean_child_exit(APEXIT_CHILDFATAL);
+ }
+ if (sigprocmask(SIG_SETMASK, &accept_previous_mask, NULL)) {
+ perror("sigprocmask(SIG_SETMASK)");
+ clean_child_exit(1);
+ }
+}
+
#elif defined (USE_SYSVSEM_SERIALIZED_ACCEPT)
#include <sys/types.h>
Victor
--
Victor J. Orlikowski
======================
v.j.orlikowski@gte.net
vjo@raleigh.ibm.com
vjo@us.ibm.com
Re: [PATCH] 1.3.x Solaris performance and SERIALIZED_ACCEPT
Posted by dean gaudet <dg...@arctic.org>.
On Tue, 2 Jan 2001, Victor J. Orlikowski wrote:
> In the 2.4 kernels, the process limit is run-time configurable
> (i.e. echo > "100000" /proc/sys/kernel/max-threads does the
> adjustment). In 2.2 kernels, you have to edit
> /usr/src/linux/include/linux/tasks.h and re-compile.
>
> My apologies on the SEM_UNDO limit on linux. You're in the right
> there. However, what would we like done for those OSes which have this
> limit? Should we simply assume that the admin is clueful?
no i'd say use the slower fcntl/flock methods and let the clueful admin
recompile apache and their kernels. let those OS folks take the heat by
virtue of crap apache performance on their system.
-dean
Re: [PATCH] 1.3.x Solaris performance and SERIALIZED_ACCEPT
Posted by "Victor J. Orlikowski" <v....@gte.net>.
In the 2.4 kernels, the process limit is run-time configurable
(i.e. echo > "100000" /proc/sys/kernel/max-threads does the
adjustment). In 2.2 kernels, you have to edit
/usr/src/linux/include/linux/tasks.h and re-compile.
My apologies on the SEM_UNDO limit on linux. You're in the right
there. However, what would we like done for those OSes which have this
limit? Should we simply assume that the admin is clueful?
Victor
--
Victor J. Orlikowski
======================
v.j.orlikowski@gte.net
vjo@raleigh.ibm.com
vjo@us.ibm.com
Re: [PATCH] 1.3.x Solaris performance and SERIALIZED_ACCEPT
Posted by dean gaudet <dg...@arctic.org>.
On Mon, 1 Jan 2001, Victor J. Orlikowski wrote:
> The problem in the pthread code stems from a race that occurs right
> after unlock (which is documented in the code). Should the child
> process be killed right after the pthread unlock, the code as-is
> catches the signal, and unlocks the mutex again, resulting in a double
> unlock, which pthread mutexes do not catch but Solaris mutexes do.
oh ew. the problem then is that the cleanup should be
registered/unregistered on each acquisition/release of the mutex. maybe
with ap_block_alarms() protection.
> Whether such a patch is something we would want to include or not is
> another issue on the table: do we want the average user to have to
> re-compile the kernel on FreeBSD or Linux to set MaxClients as high as
> they need if SysV sems are set as the default synchronization mechanism?
linux doesn't have a static sysv semaphore SEM_UNDO limit. and i've
already configured sysv sems as the default on linux (as per request of
the linux-kernel folks). linux does have a static 256 or 512 per-user
process limit (it may actually be boot-time configurable lately i haven't
checked).
-dean
Re: [PATCH] 1.3.x Solaris performance and SERIALIZED_ACCEPT
Posted by "Victor J. Orlikowski" <v....@gte.net>.
The problem in the pthread code stems from a race that occurs right
after unlock (which is documented in the code). Should the child
process be killed right after the pthread unlock, the code as-is
catches the signal, and unlocks the mutex again, resulting in a double
unlock, which pthread mutexes do not catch but Solaris mutexes do.
Solaris has boot-time limits on SysV semaphores (modify desired values
in /etc/system, and reboot). Linux and FreeBSD have compile-time
limits. Holidays have sidetracked me from a patch to detect the limits
in the various OSes, and write errors to the log if the limits do not
correspond in a reasonable manner to MaxClients (especially since
determining the limits at run-time differs for each OS).
Whether such a patch is something we would want to include or not is
another issue on the table: do we want the average user to have to
re-compile the kernel on FreeBSD or Linux to set MaxClients as high as
they need if SysV sems are set as the default synchronization mechanism?
Victor
--
Victor J. Orlikowski
======================
v.j.orlikowski@gte.net
vjo@raleigh.ibm.com
vjo@us.ibm.com
Re: [PATCH] 1.3.x Solaris performance and SERIALIZED_ACCEPT
Posted by dean gaudet <dg...@arctic.org>.
Bill Stoddard asks:
Any objections setting default to USE_SYSVSEM_SERIALIZED_ACCEPT on
Solaris?
yeah i don't think they should be used. victor points out why a few weeks
later. they require the admin to reconfigure their system -- there's a
static boot-time limit, or some insanity. if you dig through the
new-httpd archives several years ago you'll find my analysis of the
various locking primitives.
On Wed, 13 Dec 2000, Victor J. Orlikowski wrote:
> The following is a patch to use solaris native mutexes for locking, to
> improve performance on Solaris. They are slightly slower than pthread
> mutexes, but do not misbehave if unlocked twice, eliminating the race
> resulting from pthread mutexes. Further, they do not need any special
> configuration, unlike SysV sems and like pthread mutexes.
this seems like a fine thing to try... hopefully you can get some folks
to try it live before putting it into 1.3.15 though.
so is it just a double-unlock problem? do we do something stupid like
release the semaphore during fork/exec?
i always thought it was because a process exiting without releasing the
mutex would leave it locked... because there's no kernel record of who
has the locks. whereas with fcntl() and sysvsems the kernel keeps track
of who has the locks and frees them at process exit (it's why both of
these are slow).
-dean