You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by "GUMMALAM,MOHAN (HP-Cupertino,ex2)" <mo...@hp.com> on 2001/07/14 06:42:29 UTC
[PATCH] Problems with MPM threaded
I propose the following patch [PATCH A]: It will partially fix the unwanted
child deaths problem (symptoms mentioned in the mails included below). It
fixes the problem by making sure that perform_idle_server_maintenance() does
not count the threads of the process that recd the POD, in the calculation
of idle_thread_count. To do that I have used
ap_scoreboard_image->parent[process_slot]->process_status field. I am
temporarily using an already defined value, SB_IDLE_DIE. If the general
idea is acceptable, I can work on solidifying the details. PATCH A is
attached below.
However this patch exposes another problem in the code - by this new fix,
although the untargetted childs do not get a POD, the targetted child
process does not die immediately either. Here is why that happens: In the
worker_thread() routine in threaded.c, at the instant when worker 1.0
(represented in the <process_slot>.<thread_slot> format, i.e, 1 is the
process_slot and 0 is the thread_slot) gets the POD, the remaining threads
of process 1 are all waiting at the apr_lock_acquire(accept_mutex). If the
web-server is really idle, the chances are slim that all the remaining
worker threads for process 1 would acquire the lock in a very short time.
As an effect, the remaining worker threads of process 1 do not die
immediately. To resolve this:
SOLUTION 1: I plan to temporarily implement a new
apr_lock_acquire_timeout() function, which would cause the threads waiting
on the mutex to give-up after sometime. The piece of code would look
something like this:
while ((rv = SAFE_ACCEPT(apr_lock_acquire_timeout(accept_mutex, timevalue)))
!= APR_SUCCESS) {
if (check_if_timer_popped)
if (workers_may_exit)
break;
else { /* apr_lock_acquire failed */
ap_log_error(....);
workers_may_exit = 1;
}
}
I know that this would cause some performance impact, but my guess is that
it would not be a lot, especially if we keep the timevalue reasonably high.
However, in order to get the functionality of
perform_idle_server_maintenance() working right, we _will_ have to implement
the above solution (or maybe something similar)!
SOLUTION 2: One could use a slightly more _involved_ approach, where the
dying thread could send a signal to its sibling threads, each of which will
then handle that signal with a graceful exit.
SOLUTION 3: We could turn our heads the other way by not worrying about this
situation at all, since these threads would eventually die (when more
requests are handled by the webserver). However, by ignoring the problem,
the purpose of perform_idle_server_maintenance() would be lost!!
Please respond with your thought, and if there are no objections, I will go
ahead and post a patch for SOLUTION 1 sometime soon.
Thanks,
Mohan
************************* Start PATCH A ********************************
--- server/mpm/threaded/threaded.c.orig Tue Jul 3 06:58:10 2001
+++ server/mpm/threaded/threaded.c Fri Jul 13 18:28:44 2001
@@ -495,7 +495,7 @@
}
/* Sets workers_may_exit if we received a character on the pipe_of_death */
-static void check_pipe_of_death(void)
+static void check_pipe_of_death(int process_slot)
{
apr_lock_acquire(pipe_of_death_mutex);
if (!workers_may_exit) {
@@ -511,6 +511,7 @@
else {
/* It won the lottery (or something else is very
* wrong). Embrace death with open arms. */
+ ap_scoreboard_image->parent[process_slot].process_status =
SB_IDLE_DIE;
workers_may_exit = 1;
}
}
@@ -584,7 +585,7 @@
if (event & APR_POLLIN) {
/* A process got a signal on the shutdown pipe. Check if
we're
* the lucky process to die. */
- check_pipe_of_death();
+ check_pipe_of_death(process_slot);
continue;
}
@@ -972,6 +973,9 @@
int status = SERVER_DEAD;
int any_dying_threads = 0;
int any_dead_threads = 0;
+
+ if (ap_scoreboard_image->parent[i].process_status == SB_IDLE_DIE)
+ continue;
if (i >= ap_max_daemons_limit && free_length == idle_spawn_rate)
break;
************************* End PATCH A ********************************
-----Original Message-----
From: GUMMALAM,MOHAN (HP-Cupertino,ex2) [mailto:mohan_gummalam@hp.com]
Sent: Tuesday, July 10, 2001 6:54 PM
To: 'new-httpd@apache.org'
Subject: RE: Problems, 2.0.20, and HP-UX
My httpd.conf file is as follows:
<IfModule threaded.c>
StartServers 6
MaxClients 8
MinSpareThreads 5
MaxSpareThreads 125
ThreadsPerChild 25
MaxRequestsPerChild 0
</IfModule>
If I set the MaxSpareThreads to 150 (which is equal to StartServers *
ThreadsPerChild), then everything works fine.. but whenever I set
MaxSpareThreads to 125 (or any value below that, say 100), I end up with
just two processes! It takes a while (like couple of minutes) for this to
take effect. Anyone seen this behavior yet? Any clues?
M
-----Original Message-----
From: GUMMALAM,MOHAN (HP-Cupertino,ex2) [mailto:mohan_gummalam@hp.com]
Sent: Tuesday, July 10, 2001 4:58 PM
To: 'new-httpd@apache.org'
Subject: Problems, 2.0.20, and HP-UX
Hi Apache2.0 devs: On HP-UX, we are facing a problem whose symptoms are
similar to the one seen with 2.0.19 release. Upon starting Apache, the
number of processes running reduces to 2. Upon further investigation, it
looked like the main process (watchdog process) went on a rampage and killed
all the child processes in the following piece of code:
static void perform_idle_server_maintenance(void)
{
.
.
if (idle_thread_count > max_spare_threads) {
/* Kill of one child */
.
.
}
By using tusc, we found that after the child processes were created, there
were a bunch of requests sent to kill the processes:
write(13, "! ", 1) ....................................... = 1
And although a lot of waitpid's were called earlier (each returning -1), the
following bunch of waitpid's succeeded after a while, all occuring together.
waitpid(-1, WIFEXITED(0), WNOHANG|WUNTRACED) ............. = 2122
waitpid(-1, WIFEXITED(0), WNOHANG|WUNTRACED) ............. = 2121
waitpid(-1, WIFEXITED(0), WNOHANG|WUNTRACED) ............. = 2120
waitpid(-1, WIFEXITED(0), WNOHANG|WUNTRACED) ............. = 2119
waitpid(-1, WIFEXITED(0), WNOHANG|WUNTRACED) ............. = 2118
waitpid(-1, WIFEXITED(0), WNOHANG|WUNTRACED) ............. = 2117
At the end of it, there are only two processes running. I tried this out
with different StartServers values, (4, 5 and 6). Each time I ended up with
the same final state (of two processes). The above output is in the case of
StartServers=6.
I'm surprised that no one else has run into this, when it consistently shows
up on HP-UX.
We'll be investigating this further here, and will get back to you. If
anyone else has run into this problem, do let me know.
Thanks,
M
Re: [PATCH] Problems with MPM threaded
Posted by "Roy T. Fielding" <fi...@ebuilt.com>.
> > That architecture was explored in detail by Netscape. It isn't reliable
> > and slows your web server to a crawl whenever dynamic content is produced.
> > It should only be used for static file servers and caching gateways, and
> > people implementing those might as well use an in-kernel server like TUX.
>
> i'm confused... what architecture has that problem with dynamic content?
>
> i can believe it if you're referring to a userland threading library,
> single process server.
>
> i can't believe it if the threads are scheduled by the kernel (either as
> 1:1 or 1:many).
>
> NSPR threading was probably to blame, no?
Does NSPR do userland threading on Solaris? I thought they were using native
threads. But you are right, the dynamic content problem was for libraries
that are not thread-safe and for platforms that had userland threads. They
ended up with some funky stuff in NSAPI that made the generic API slower
than mollasses (internal modules used another API).
....Roy
Re: [PATCH] Problems with MPM threaded
Posted by dean gaudet <dg...@arctic.org>.
On Sat, 14 Jul 2001, Roy T. Fielding wrote:
> > The "correct" fix, as I see it, is to kill off the interprocess
> > accept lock by removing the possibility of having other processes
> > in a *threaded* MPM. -- justin
>
> That architecture was explored in detail by Netscape. It isn't reliable
> and slows your web server to a crawl whenever dynamic content is produced.
> It should only be used for static file servers and caching gateways, and
> people implementing those might as well use an in-kernel server like TUX.
i'm confused... what architecture has that problem with dynamic content?
i can believe it if you're referring to a userland threading library,
single process server.
i can't believe it if the threads are scheduled by the kernel (either as
1:1 or 1:many).
NSPR threading was probably to blame, no?
-dean
Re: [PATCH] Problems with MPM threaded
Posted by "Roy T. Fielding" <fi...@ebuilt.com>.
> The "correct" fix, as I see it, is to kill off the interprocess
> accept lock by removing the possibility of having other processes
> in a *threaded* MPM. -- justin
That architecture was explored in detail by Netscape. It isn't reliable
and slows your web server to a crawl whenever dynamic content is produced.
It should only be used for static file servers and caching gateways, and
people implementing those might as well use an in-kernel server like TUX.
....Roy
Re: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Sat, Jul 14, 2001 at 01:06:28AM -0700, Aaron Bannert wrote:
> This looks like a job for condition variables. Using condition
> variables you can control exactly when threads get woken up to check
> on the condition. For example, when one thread returns from accept(),
> it can signal another thread to move to the accept() state (and as an
> optimization that I think Ryan suggested, now you don't need to have
> an intraprocess accept lock, since only one thread can be listening in
> accept() at a time). The other threads are blocked waiting for some
> condition, either allowing them to pass to the accept() state or the
> workers_may_exit state.
The problem, as I see it, is that you are mixing an *interprocess*
accept lock (otherwise, you'll have two processes call accept - no-no
on some platforms - threaded MPM, as it stands, has *multiple* child
processes *each* consisting of threads) and an *intraprocess*
destruction mechanism (workers_may_exit). The issue here is that the
accept_mutex may get passed onto another process which doesn't need to
be destroyed (hence, it won't give up the mutex for a while - forcing
the "doomed" child's threads to belay the request to die).
If condition variables allow you to handle this gracefully (the man
page doesn't seem to indicate this mixing of scopes), then maybe that
is a solution. But, remember, if the mutex stays within one process,
everything is happy and fine.
The "correct" fix, as I see it, is to kill off the interprocess
accept lock by removing the possibility of having other processes
in a *threaded* MPM. -- justin
Re: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Sat, Jul 14, 2001 at 12:42:29AM -0400, GUMMALAM,MOHAN (HP-Cupertino,ex2) wrote:
[snip]
> SOLUTION 1: I plan to temporarily implement a new
> apr_lock_acquire_timeout() function, which would cause the threads waiting
> on the mutex to give-up after sometime. The piece of code would look
> something like this:
>
> while ((rv = SAFE_ACCEPT(apr_lock_acquire_timeout(accept_mutex, timevalue)))
> != APR_SUCCESS) {
> if (check_if_timer_popped)
> if (workers_may_exit)
> break;
> else { /* apr_lock_acquire failed */
> ap_log_error(....);
> workers_may_exit = 1;
> }
> }
>
> I know that this would cause some performance impact, but my guess is that
> it would not be a lot, especially if we keep the timevalue reasonably high.
> However, in order to get the functionality of
> perform_idle_server_maintenance() working right, we _will_ have to implement
> the above solution (or maybe something similar)!
This looks like a job for condition variables. Using condition
variables you can control exactly when threads get woken up to check
on the condition. For example, when one thread returns from accept(),
it can signal another thread to move to the accept() state (and as an
optimization that I think Ryan suggested, now you don't need to have
an intraprocess accept lock, since only one thread can be listening in
accept() at a time). The other threads are blocked waiting for some
condition, either allowing them to pass to the accept() state or the
workers_may_exit state.
If this sounds like something you could use here, I can probably elaborate
on this some more, maybe even give some [pseudo]code.
-aaron
Re: Status Shared Memory
Posted by Ian Holsman <ia...@cnet.com>.
On 13 Jul 2001 23:58:12 -0700, Justin Erenkrantz wrote:
> On Sat, Jul 14, 2001 at 08:56:31AM +0200, Rivium wrote:
> > Hi,
> >
> > By Browsing through throught the source code, i notified several Anti-Shared
> > memory comments, as well a negative advices about using it.
>
> Shared memory at least on Unix has many limitations on it that hinder
> its effectiveness. It can only be used in very specific circumstances
> portably. (The issue is primarily about the size of the memory you can
> allocate with shmem.) Ideally, the new SMS code in APR will allow us
> to implement a shared-memory backbone for the allocation algorithms in
> httpd. That means that almost any data structure coded to use pools
> (aka SMS) can then be used with shared memory. Nobody is working on
> that yet though (no itches there yet - although Ian has talked about
> wanting it).
I think I said I couldn't see how it could be done via SMS so that the
requester could use it without explicitly knowing it was shared memory.
In general read-only shared memory structues scale pretty well, as you
only need 1 instance per machine, but you have to make concessions.
In the APR implmetaton for unix:
you can't expand the area
you can't attach to it from a seperate process (which isn't the
creator's child)
have a look at memory mapped files, or things like berkleyDB or TDB
if you are just wanting a methoud of passing control information between
processes then there are other better methouds that Justin mentioned.
if you want cross-machine shared information, have a look at spread
(www.spread.org)
>
> However, it would be impossible to have every allocation in httpd
> residing in shared memory due to all of the platform limitations.
>
> I'm wondering if you are talking about something other than shared
> memory as most of us Unix geeks know it. You're comments about
> Visual Studio .NET make me wonder what your definition of shared
> memory is. =-) -- justin
--
Ian Holsman
Performance Measurement & Analysis
CNET Networks - 415 364-8608
Re: Status Shared Memory
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
----- Original Message -----
From: "Justin Erenkrantz" <je...@ebuilt.com>
To: <ne...@apache.org>
Sent: Saturday, July 14, 2001 3:03 AM
Subject: Re: Status Shared Memory
> On Sat, Jul 14, 2001 at 09:40:59AM +0200, Rivium wrote:
> > But my Main Goal is to speed things -really- up. So i was talking about
> > segment of shared memory, on the same machine
> > (I love the current Memory prices!)
>
> Shared memory has major problems. Consider the fact that you can't
> store any pointers in shared memory nor can you expand the memory once
> you have allocated it (with most implementations AFAIK).
1. Yes, you can store offsets, rather than pointers. This makes it impossible
to store APR structures (e.g., an apr_table), but quite possible to store
your own data.
2. All clients need to re-map when the memory is expanded. The simplest
mechansim is a shmem section of memory, with a tag.
> The reason that you can't use pointers is that users of the shmem will *not*
> receive the memory at the same location in each process. So, about
> the only thing you can store is raw text or arrays of known size in
> shared memory (like the scoreboard which isn't that big and has known
> size at compile-time).
Very true. Stash the bounds, or size, inside the structure.
> IMHO, you're better off using true IPC mechanisms - especially when
> you are concerned about scalability (as you seem to be). Shared
> memory doesn't scale. If I'm wrong, please correct me. =) In
> your scenario, mmap() *may* provide a better alternative and remains
> file-based, and will be significantly faster than plain read()/write()
> (as long as you don't run out of address space). -- justin
MMAP shouldn't be significantly faster at run time, only during allocation.
Most systems have memory backed by a file, only it's a swap image, rather
than a shmem file.
Re: Status Shared Memory
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Sat, Jul 14, 2001 at 09:40:59AM +0200, Rivium wrote:
> But my Main Goal is to speed things -really- up. So i was talking about
> segment of shared memory, on the same machine
> (I love the current Memory prices!)
Shared memory has major problems. Consider the fact that you can't
store any pointers in shared memory nor can you expand the memory once
you have allocated it (with most implementations AFAIK). The reason
that you can't use pointers is that users of the shmem will *not*
receive the memory at the same location in each process. So, about
the only thing you can store is raw text or arrays of known size in
shared memory (like the scoreboard which isn't that big and has known
size at compile-time).
IMHO, you're better off using true IPC mechanisms - especially when
you are concerned about scalability (as you seem to be). Shared
memory doesn't scale. If I'm wrong, please correct me. =) In
your scenario, mmap() *may* provide a better alternative and remains
file-based, and will be significantly faster than plain read()/write()
(as long as you don't run out of address space). -- justin
Re: Status Shared Memory
Posted by Rivium <ri...@tser.org>.
Hi,
> However, it would be impossible to have every allocation in httpd
> residing in shared memory due to all of the platform limitations.
So the choice of the platform should be done with caution...
> I'm wondering if you are talking about something other than shared
> memory as most of us Unix geeks know it. You're comments about
> Visual Studio .NET make me wonder what your definition of shared
> memory is. =-)
A Piece of common used memory, where each client would get the same
information, if they looked up in that part of memory, not nessecary stored
on the same machine. (so for that purpose... a sql database , or something
done with a soap call could even be called shared memory hehehehe).
But my Main Goal is to speed things -really- up. So i was talking about
segment of shared memory, on the same machine
(I love the current Memory prices!)
My situation is that i'f got a bunch of users simultationously logged into
my module, and that they'r all chatting and babbling with each other, and
what to know where there friends hang around.
I used to have "OnlineTracking" Filesystem based access method for that, for
4 years now, But as more users log into, The negative drawbacks of that are
slowely popping up.
I'm going to try building an as empty possible module,that is just using
shared memory, and will see where i end. Hope you guys can comment a little
bit if i post " one and then " a version for comment.
Maybe it will not run on every system then, but at least on my target
platforms.
Regards,
R.
Re: Status Shared Memory
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Sat, Jul 14, 2001 at 08:56:31AM +0200, Rivium wrote:
> Hi,
>
> By Browsing through throught the source code, i notified several Anti-Shared
> memory comments, as well a negative advices about using it.
Shared memory at least on Unix has many limitations on it that hinder
its effectiveness. It can only be used in very specific circumstances
portably. (The issue is primarily about the size of the memory you can
allocate with shmem.) Ideally, the new SMS code in APR will allow us
to implement a shared-memory backbone for the allocation algorithms in
httpd. That means that almost any data structure coded to use pools
(aka SMS) can then be used with shared memory. Nobody is working on
that yet though (no itches there yet - although Ian has talked about
wanting it).
However, it would be impossible to have every allocation in httpd
residing in shared memory due to all of the platform limitations.
I'm wondering if you are talking about something other than shared
memory as most of us Unix geeks know it. You're comments about
Visual Studio .NET make me wonder what your definition of shared
memory is. =-) -- justin
Status Shared Memory
Posted by Rivium <ri...@tser.org>.
Hi,
By Browsing through throught the source code, i notified several Anti-Shared
memory comments, as well a negative advices about using it.
My Target Machine is a bsd Box, And i'm compiling Apache on Visual
Studio.Net.
How is the status now ? Should i get .21 If i want to use shared memory, or
could i implement it, just straight ahead ?
Not that there seems to be a straight example of how to use it :-)
Mod_aug_digest and the scoreboard file, seems to be using it a little bit
different.
Btw. Apache Compiles nice on the Microsoft Visual Studio .Net Platform :-)
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Sat, Jul 14, 2001 at 09:43:53PM +0200, Sander Striker wrote:
> Err, doesn't destruction of the mutex wake everyone up?
> Oh, wait, does every process share the same mutex?
Yup. accept_mutex is CROSS_PROCESS. See?
That's the rub. You can't destroy it. Otherwise, you'd screw the
other child processes which shouldn't be killed off.
Which is why I'm saying it is invalid to have multiple child process
in a threaded MPM.
> > And, destroying its parent pool does *NOT* destroy the thread.
> > Look at the code again.
>
> Well, it should, although it may not do so now.
That's impossible to do. Another thread can not indicate that a thread
should exit with the threading libraries (safely). It must have some
OOB way of doing so (which we have). The only problem is that our
thread is stuck in an *uninterruptable* operation. If that operation
were interruptable OR that thread received the mutex, then it'd check
the workers_may_exit flag. Then, it'd exit. -- justin
[PATCH] Add intraprocess mutex to threaded MPM WAS Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
> > Ah. Yes, that makes more sense. But, that's not what's there now.
> >
> > Shall I submit a patch to threaded MPM to do this? -- justin
>
> Please.
This also includes my POD patch. I can separate it out if you don't
want the POD code merged yet. -- justin
Index: threaded.c
===================================================================
RCS file: /home/cvs/httpd-2.0/server/mpm/threaded/threaded.c,v
retrieving revision 1.44
diff -u -r1.44 threaded.c
--- threaded.c 2001/07/03 13:58:10 1.44
+++ threaded.c 2001/07/14 20:48:04
@@ -143,10 +143,7 @@
char ap_coredump_dir[MAX_STRING_LEN];
-static apr_file_t *pipe_of_death_in = NULL;
-static apr_file_t *pipe_of_death_out = NULL;
-static apr_lock_t *pipe_of_death_mutex; /* insures that a child process only
- consumes one character */
+static ap_pod_t *pipe_of_death;
/* *Non*-shared http_main globals... */
@@ -178,7 +175,13 @@
static int worker_thread_count;
static apr_lock_t *worker_thread_count_mutex;
-/* Locks for accept serialization */
+/* Locks for accept serialization
+ * worker_accept_mutex ensures that only one thread in a child process
+ * may be in accept. It also ensures that when we have
+ * workers_may_exit=1 that we exit.
+ * accept_mutex is the cross-process mutex which all children have.
+ */
+static apr_lock_t *worker_accept_mutex;
static apr_lock_t *accept_mutex;
static apr_lockmech_e_np accept_lock_mech = APR_LOCK_DEFAULT;
static const char *lock_fname;
@@ -494,29 +497,6 @@
}
}
-/* Sets workers_may_exit if we received a character on the pipe_of_death */
-static void check_pipe_of_death(void)
-{
- apr_lock_acquire(pipe_of_death_mutex);
- if (!workers_may_exit) {
- apr_status_t ret;
- char pipe_read_char;
- apr_size_t n = 1;
-
- ret = apr_recv(listensocks[0], &pipe_read_char, &n);
- if (APR_STATUS_IS_EAGAIN(ret)) {
- /* It lost the lottery. It must continue to suffer
- * through a life of servitude. */
- }
- else {
- /* It won the lottery (or something else is very
- * wrong). Embrace death with open arms. */
- workers_may_exit = 1;
- }
- }
- apr_lock_release(pipe_of_death_mutex);
-}
-
static void * worker_thread(void * dummy)
{
proc_info * ti = dummy;
@@ -539,8 +519,8 @@
worker_thread_count++;
apr_lock_release(worker_thread_count_mutex);
- apr_poll_setup(&pollset, num_listensocks+1, tpool);
- for(n=0 ; n <= num_listensocks ; ++n)
+ apr_poll_setup(&pollset, num_listensocks, tpool);
+ for(n = 0; n < num_listensocks; ++n)
apr_poll_socket_add(pollset, listensocks[n], APR_POLLIN);
/* TODO: Switch to a system where threads reuse the results from earlier
@@ -553,7 +533,10 @@
(void) ap_update_child_status(process_slot, thread_slot, SERVER_READY,
(request_rec *) NULL);
- if ((rv = SAFE_ACCEPT(apr_lock_acquire(accept_mutex)))
+
+ apr_lock_acquire(worker_accept_mutex);
+ if (!workers_may_exit &&
+ (rv = SAFE_ACCEPT(apr_lock_acquire(accept_mutex)))
!= APR_SUCCESS) {
ap_log_error(APLOG_MARK, APLOG_EMERG, rv, ap_server_conf,
"apr_lock_acquire failed. Attempting to shutdown "
@@ -580,12 +563,8 @@
if (workers_may_exit) break;
- apr_poll_revents_get(&event, listensocks[0], pollset);
- if (event & APR_POLLIN) {
- /* A process got a signal on the shutdown pipe. Check if we're
- * the lucky process to die. */
- check_pipe_of_death();
- continue;
+ if (!ap_mpm_pod_check(pipe_of_death)) {
+ workers_may_exit = 1;
}
if (num_listensocks == 1) {
@@ -624,6 +603,12 @@
"process gracefully.");
workers_may_exit = 1;
}
+ if ((rv = apr_lock_release(worker_accept_mutex)) != APR_SUCCESS) {
+ ap_log_error(APLOG_MARK, APLOG_EMERG, rv, ap_server_conf,
+ "apr_lock_release failed. Attempting to shutdown "
+ "process gracefully.");
+ workers_may_exit = 1;
+ }
if (csd != NULL) {
process_socket(ptrans, csd, process_slot, thread_slot);
requests_this_child--;
@@ -637,6 +622,12 @@
"process gracefully.");
workers_may_exit = 1;
}
+ if ((rv = apr_lock_release(worker_accept_mutex)) != APR_SUCCESS) {
+ ap_log_error(APLOG_MARK, APLOG_EMERG, rv, ap_server_conf,
+ "apr_lock_release failed. Attempting to shutdown "
+ "process gracefully.");
+ workers_may_exit = 1;
+ }
break;
}
apr_pool_clear(ptrans);
@@ -780,12 +771,8 @@
}
/* Set up the pollfd array */
- listensocks = apr_pcalloc(pchild,
- sizeof(*listensocks) * (num_listensocks + 1));
-#if APR_FILES_AS_SOCKETS
- apr_socket_from_file(&listensocks[0], pipe_of_death_in);
-#endif
- for (lr = ap_listeners, i = 1; i <= num_listensocks; lr = lr->next, ++i)
+ listensocks = apr_pcalloc(pchild, sizeof(*listensocks) * num_listensocks);
+ for (lr = ap_listeners, i = 0; i < num_listensocks; lr = lr->next, ++i)
listensocks[i]=lr->sd;
/* Setup worker threads */
@@ -799,9 +786,8 @@
worker_thread_count = 0;
apr_lock_create(&worker_thread_count_mutex, APR_MUTEX, APR_INTRAPROCESS,
NULL, pchild);
- apr_lock_create(&pipe_of_death_mutex, APR_MUTEX, APR_INTRAPROCESS,
+ apr_lock_create(&worker_accept_mutex, APR_MUTEX, APR_INTRAPROCESS,
NULL, pchild);
-
ts = apr_palloc(pchild, sizeof(*ts));
apr_threadattr_create(&thread_attr, pchild);
@@ -894,29 +880,6 @@
return 0;
}
-/* If there aren't many connections coming in from the network, the child
- * processes may need to be awakened from their network i/o waits.
- * The pipe of death is an effective prod.
- */
-
-static void wake_up_and_die(void)
-{
- int i;
- char char_of_death = '!';
- apr_size_t one = 1;
- apr_status_t rv;
-
- for (i = 0; i < ap_daemons_limit;) {
- if ((rv = apr_file_write(pipe_of_death_out, &char_of_death, &one))
- != APR_SUCCESS) {
- if (APR_STATUS_IS_EINTR(rv)) continue;
- ap_log_error(APLOG_MARK, APLOG_WARNING, rv, ap_server_conf,
- "write pipe_of_death");
- }
- i++;
- }
-}
-
/* start up a bunch of children */
static void startup_children(int number_to_start)
{
@@ -955,8 +918,6 @@
int free_slots[MAX_SPAWN_RATE];
int last_non_dead;
int total_non_dead;
- apr_size_t one = 1;
- apr_status_t rv;
/* initialize the free_list */
free_length = 0;
@@ -1004,11 +965,7 @@
ap_max_daemons_limit = last_non_dead + 1;
if (idle_thread_count > max_spare_threads) {
- char char_of_death = '!';
- /* Kill off one child */
- if ((rv = apr_file_write(pipe_of_death_out, &char_of_death, &one)) != APR_SUCCESS) {
- ap_log_error(APLOG_MARK, APLOG_WARNING, rv, ap_server_conf, "write pipe_of_death");
- }
+ ap_mpm_pod_signal(pipe_of_death);
idle_spawn_rate = 1;
}
else if (idle_thread_count < min_spare_threads) {
@@ -1129,18 +1086,11 @@
pconf = _pconf;
ap_server_conf = s;
- rv = apr_file_pipe_create(&pipe_of_death_in, &pipe_of_death_out, pconf);
+ rv = ap_mpm_pod_open(pconf, &pipe_of_death);
if (rv != APR_SUCCESS) {
ap_log_error(APLOG_MARK, APLOG_ERR, rv,
- (const server_rec*) ap_server_conf,
- "apr_file_pipe_create (pipe_of_death)");
- exit(1);
- }
-
- if ((rv = apr_file_pipe_timeout_set(pipe_of_death_in, 0)) != APR_SUCCESS) {
- ap_log_error(APLOG_MARK, APLOG_ERR, rv,
(const server_rec*) ap_server_conf,
- "apr_file_pipe_timeout_set (pipe_of_death)");
+ "ap_mpm_pod_open");
exit(1);
}
@@ -1208,7 +1158,7 @@
/* Time to gracefully shut down:
* Kill child processes, tell them to call child_exit, etc...
*/
- wake_up_and_die();
+ ap_mpm_pod_killpg(pipe_of_death, ap_daemons_limit);
if (unixd_killpg(getpgrp(), SIGTERM) < 0) {
ap_log_error(APLOG_MARK, APLOG_WARNING, errno, ap_server_conf, "killpg SIGTERM");
@@ -1249,7 +1199,7 @@
update_scoreboard_global();
/* wake up the children...time to die. But we'll have more soon */
- wake_up_and_die();
+ ap_mpm_pod_killpg(pipe_of_death, ap_daemons_limit);
if (is_graceful) {
ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_NOTICE, 0, ap_server_conf,
Index: mpm.h
===================================================================
RCS file: /home/cvs/httpd-2.0/server/mpm/threaded/mpm.h,v
retrieving revision 1.5
diff -u -r1.5 mpm.h
--- mpm.h 2001/05/07 18:41:48 1.5
+++ mpm.h 2001/07/14 20:48:04
@@ -66,6 +66,7 @@
#define MPM_NAME "Threaded"
#define AP_MPM_NEEDS_RECLAIM_CHILD_PROCESSES 1
+#define AP_MPM_USES_POD 1
#define MPM_SYNC_CHILD_TABLE() (ap_sync_scoreboard_image())
#define MPM_CHILD_PID(i) (ap_scoreboard_image->parent[i].pid)
#define MPM_NOTE_CHILD_KILLED(i) (MPM_CHILD_PID(i) = 0)
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM
threaded
Posted by rb...@covalent.net.
On Sat, 14 Jul 2001, Justin Erenkrantz wrote:
> On Sat, Jul 14, 2001 at 01:29:35PM -0700, rbb@covalent.net wrote:
> > What you are missing is the history. All of the problems that we are
> > trying to solve were solved already in the original code. These problem
> > are solved most easily by using two mutexes, one for keeping only one
> > process in the inner loop, and the inner one for the threads in the
> > process. Then, when you are killing a process, you can easily wake up all
> > the threads in the process that are sitting on a lock.
>
> Ah. Yes, that makes more sense. But, that's not what's there now.
>
> Shall I submit a patch to threaded MPM to do this? -- justin
Please.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Sat, Jul 14, 2001 at 01:29:35PM -0700, rbb@covalent.net wrote:
> What you are missing is the history. All of the problems that we are
> trying to solve were solved already in the original code. These problem
> are solved most easily by using two mutexes, one for keeping only one
> process in the inner loop, and the inner one for the threads in the
> process. Then, when you are killing a process, you can easily wake up all
> the threads in the process that are sitting on a lock.
Ah. Yes, that makes more sense. But, that's not what's there now.
Shall I submit a patch to threaded MPM to do this? -- justin
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM
threaded
Posted by rb...@covalent.net.
> [bird's eye view observation follows:]
>
> I'm beginning to understand why I keep running across parts of APR that
> seem inconsistent, overly complicated, disjoint, not generally applicable,
> and sometimes seemingly suboptimal. They are holdovers from when they
> provided a solution in httpd. I can see how it would be difficult to
> form APR for general use when it started out just being code ripped out
> of apache. I do think that many of these problems are because APR is
> in some places still striving to support the needs of httpd directly,
> instead of just providing the basic building blocks that both httpd and
> other programs need.
>
> So where to start?
Comments like this really bother me. It shows that you haven't really
looked at the history. APR only took a few functions from Apache
originally. Most of APR was written from scratch by looking at what
Windows, Unix, and BeOS require. If you looked at APR when it was first
committed, it was less complex, but it also didn't do as much. Most of
the functions in APR were created because the web server required them,
but many were added because other programs required them. Subversion has
added many functions to APR.
If you think there are problems with complexity, then you should post
comments and patches to help reduce the complexity.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Sat, Jul 14, 2001 at 01:29:35PM -0700, rbb@covalent.net wrote:
> This is why the original code, in apache-apr, written by Manoj back at IBM
> used two mutexes. The first was cross-process, the second was
> cross-threads. It allowed us to handle this much cleaner. I didn't
> realize this had been removed.
>
[snip]
>
> What you are missing is the history. All of the problems that we are
> trying to solve were solved already in the original code. These problem
> are solved most easily by using two mutexes, one for keeping only one
> process in the inner loop, and the inner one for the threads in the
> process. Then, when you are killing a process, you can easily wake up all
> the threads in the process that are sitting on a lock.
[bird's eye view observation follows:]
I'm beginning to understand why I keep running across parts of APR that
seem inconsistent, overly complicated, disjoint, not generally applicable,
and sometimes seemingly suboptimal. They are holdovers from when they
provided a solution in httpd. I can see how it would be difficult to
form APR for general use when it started out just being code ripped out
of apache. I do think that many of these problems are because APR is
in some places still striving to support the needs of httpd directly,
instead of just providing the basic building blocks that both httpd and
other programs need.
So where to start?
-aaron
Standards compliance in Apache code WAS Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Sat, Jul 14, 2001 at 01:19:43PM -0700, Aaron Bannert wrote:
> > Async cancellation of threads is VERY bad ju-ju. We don't do it because
> > it causes most, if not all, thread libraries to leak. The cleanups are
> > called because the child pool is destroyed when all the threads die.
>
> Right, but we're about to kill the processes anyway, so who cares if
> it leaks on some platforms. Anyway, we're writing to a specification,
> not an implementation. If Apache isn't going to encourage OS vendors to
> properly implement their own APIs, who is?
Eh, it's probably the other way around. Otherwise, we'd have no need
for #ifdefs. =) Standards are only as good as their implementations.
Most users of Apache couldn't care less if their favorite OS didn't
implement a particular feature correctly. It's *our* problem to work
around it. Ideally, yes, we could just write to a spec, but that's in
fantasy land. I live in the real world with lots of sub-par
implementations flying around.
> (Just think how easy it would be for us if all these platforms fully supported
> a single API, be it posix or whatever. Heaven!)
>
> Anyway, to me this is what pools were designed for.
I'd still like to know how you are going to interrupt the cross-process
mutex.
It doesn't matter if it implemented Posix or whatever. That's not the
problem here. We're in the middle of an uninterruptable call and that
mutex can not be destroyed because it is *cross-process*. -- justin
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM
threaded
Posted by rb...@covalent.net.
On Sat, 14 Jul 2001, Aaron Bannert wrote:
> > Async cancellation of threads is VERY bad ju-ju. We don't do it because
> > it causes most, if not all, thread libraries to leak. The cleanups are
> > called because the child pool is destroyed when all the threads die.
>
> Right, but we're about to kill the processes anyway, so who cares if
> it leaks on some platforms. Anyway, we're writing to a specification,
That's not true. In ONE MPM we are about to die. APR does not support
async cancellation of threads, because platforms don't support it in
practice.
> not an implementation. If Apache isn't going to encourage OS vendors to
> properly implement their own APIs, who is?
When it is safe to cancel threads on all platforms, and it can be
implemented in APR without causing huge memory leaks, then we can add
support to APR for async thread cancellation. Since those requirements
aren't met today, this conversation is a non-starter.
> Anyway, to me this is what pools were designed for.
No, pools only help with memory allocated by the application. The memory
leaks we would be hitting would most often be in the kernel.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
> Async cancellation of threads is VERY bad ju-ju. We don't do it because
> it causes most, if not all, thread libraries to leak. The cleanups are
> called because the child pool is destroyed when all the threads die.
Right, but we're about to kill the processes anyway, so who cares if
it leaks on some platforms. Anyway, we're writing to a specification,
not an implementation. If Apache isn't going to encourage OS vendors to
properly implement their own APIs, who is?
(Just think how easy it would be for us if all these platforms fully supported
a single API, be it posix or whatever. Heaven!)
Anyway, to me this is what pools were designed for.
-aaron
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Sat, Jul 14, 2001 at 01:18:04PM -0700, Justin Erenkrantz wrote:
> On Sat, Jul 14, 2001 at 12:49:51PM -0700, Aaron Bannert wrote:
> > Doesn't pthread_mutex_acquire sit in sem_wait() or sigwait()? That'll
> > let it be cancel()ed.
>
> Nope.
>
> As I posted earlier, man cancellation on Solaris 8 says:
>
> A mutex is explicitly not a cancellation point and should
> be held for only the minimal essential time.
Hmmm..that sounds like we're using mutexes incorrectly.
-aaron
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Sat, Jul 14, 2001 at 12:49:51PM -0700, Aaron Bannert wrote:
> Doesn't pthread_mutex_acquire sit in sem_wait() or sigwait()? That'll
> let it be cancel()ed.
Nope.
As I posted earlier, man cancellation on Solaris 8 says:
A mutex is explicitly not a cancellation point and should
be held for only the minimal essential time.
pthread_cancel wouldn't work. And as Ryan pointed out, it is bad to try
and forcibly kill a thread. -- justin
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM
threaded
Posted by rb...@covalent.net.
On Sat, 14 Jul 2001, Aaron Bannert wrote:
> On Sat, Jul 14, 2001 at 12:27:05PM -0700, Justin Erenkrantz wrote:
> > And, you can't kick the thread out of the mutex acquire (pthread_cancel
> > or similar strategies don't cancel a mutex operation), so you are
> > screwed. And, destroying its parent pool does *NOT* destroy the
> > thread. Look at the code again. The only person that can call
> > pthread_exit() is the actual thread itself. You can't call
> > pthread_exit on behalf of another thread (i.e. from the thread that
> > knows it is doomed or a cleanup thread). It just doesn't work like
> > that.
>
> Here's what my box's pthread_cancel man page says:
>
> Cancellation points are those points in the program execu�
> tion where a test for pending cancellation requests is
> performed and cancellation is executed if positive. The
> following POSIX threads functions are cancellation points:
>
> pthread_join(3)
> pthread_cond_wait(3)
> pthread_cond_timedwait(3)
> pthread_testcancel(3)
> sem_wait(3)
> sigwait(3)
>
> Doesn't pthread_mutex_acquire sit in sem_wait() or sigwait()? That'll
> let it be cancel()ed.
>
>
> Anyway, yes the pool cleanup routines that are supposed to be calling
> apr_thread_exit() are broken because nothing in httpd is calling
> apr_thread_exit() (AFA-my-grep-says). So let's either let the thread
> creator (aka httpd) do the cleanup registration, or put it in apr
> (as an optional feature).
Async cancellation of threads is VERY bad ju-ju. We don't do it because
it causes most, if not all, thread libraries to leak. The cleanups are
called because the child pool is destroyed when all the threads die.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Sat, Jul 14, 2001 at 12:27:05PM -0700, Justin Erenkrantz wrote:
> And, you can't kick the thread out of the mutex acquire (pthread_cancel
> or similar strategies don't cancel a mutex operation), so you are
> screwed. And, destroying its parent pool does *NOT* destroy the
> thread. Look at the code again. The only person that can call
> pthread_exit() is the actual thread itself. You can't call
> pthread_exit on behalf of another thread (i.e. from the thread that
> knows it is doomed or a cleanup thread). It just doesn't work like
> that.
Here's what my box's pthread_cancel man page says:
Cancellation points are those points in the program execu
tion where a test for pending cancellation requests is
performed and cancellation is executed if positive. The
following POSIX threads functions are cancellation points:
pthread_join(3)
pthread_cond_wait(3)
pthread_cond_timedwait(3)
pthread_testcancel(3)
sem_wait(3)
sigwait(3)
Doesn't pthread_mutex_acquire sit in sem_wait() or sigwait()? That'll
let it be cancel()ed.
Anyway, yes the pool cleanup routines that are supposed to be calling
apr_thread_exit() are broken because nothing in httpd is calling
apr_thread_exit() (AFA-my-grep-says). So let's either let the thread
creator (aka httpd) do the cleanup registration, or put it in apr
(as an optional feature).
-aaron
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Sander Striker <st...@apache.org>.
> > The way I see it, each process has a single pool instance as the parent
> > for all the threads. Resetting or destroying that pool should
> effectively
> > kill all threads. What am I missing?
>
> As I see it, the problem is:
>
> [ Platforms with SAFE_ACCEPT == APR_SUCCESS rather than the lock don't
> apply here. ]
>
> All threads are stuck in the accept mutex except for one. It reads the
> POD. And, it is time to die. It sets workers_may_exit to 1. It then
> releases the accept mutex and exits. Another *child process* acquires
> the accept mutex. Depending upon how many child processes you have, it
> may take a while for all of the threads in the doomed child process to
> acquire the mutex, wake up, check workers_may_exit, release the mutex,
> and then exit. Only when all of the threads in the doomed child process
> are exited does any cleanup occur.
>
> These sibling threads are blocked until they receive the mutex. They
> aren't going to be going anywhere until the mutex is turned over to
> them. They can't check the value of workers_may_exit *until* they
> acquire the mutex. Condition variables help *somewhat*, but the
> problem is now of scope - you have an interprocess condition and a
> intraprocess condition to check. Does this work? I don't know.
>
> And, you can't kick the thread out of the mutex acquire (pthread_cancel
> or similar strategies don't cancel a mutex operation), so you are
> screwed.
Err, doesn't destruction of the mutex wake everyone up?
Oh, wait, does every process share the same mutex?
> And, destroying its parent pool does *NOT* destroy the thread.
> Look at the code again.
Well, it should, although it may not do so now.
> The only person that can call
> pthread_exit() is the actual thread itself. You can't call
> pthread_exit on behalf of another thread (i.e. from the thread that
> knows it is doomed or a cleanup thread). It just doesn't work like
> that.
>
> I just don't agree with Ryan's assessment here. But, it's possible
> I'm missing some major piece of code. -- justin
Could be I'm way off too.
Sander
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Sander Striker <st...@apache.org>.
> > The way I see it, each process has a single pool instance as the parent
> > for all the threads. Resetting or destroying that pool should
> effectively
> > kill all threads. What am I missing?
>
> As I see it, the problem is:
>
> [ Platforms with SAFE_ACCEPT == APR_SUCCESS rather than the lock don't
> apply here. ]
>
> All threads are stuck in the accept mutex except for one. It reads the
> POD. And, it is time to die. It sets workers_may_exit to 1. It then
> releases the accept mutex and exits. Another *child process* acquires
> the accept mutex. Depending upon how many child processes you have, it
> may take a while for all of the threads in the doomed child process to
> acquire the mutex, wake up, check workers_may_exit, release the mutex,
> and then exit. Only when all of the threads in the doomed child process
> are exited does any cleanup occur.
>
> These sibling threads are blocked until they receive the mutex. They
> aren't going to be going anywhere until the mutex is turned over to
> them. They can't check the value of workers_may_exit *until* they
> acquire the mutex. Condition variables help *somewhat*, but the
> problem is now of scope - you have an interprocess condition and a
> intraprocess condition to check. Does this work? I don't know.
>
> And, you can't kick the thread out of the mutex acquire (pthread_cancel
> or similar strategies don't cancel a mutex operation), so you are
> screwed.
Err, doesn't destruction of the mutex wake everyone up?
Oh, wait, does every process share the same mutex?
> And, destroying its parent pool does *NOT* destroy the thread.
> Look at the code again.
Well, it should, although it may not do so now.
> The only person that can call
> pthread_exit() is the actual thread itself. You can't call
> pthread_exit on behalf of another thread (i.e. from the thread that
> knows it is doomed or a cleanup thread). It just doesn't work like
> that.
>
> I just don't agree with Ryan's assessment here. But, it's possible
> I'm missing some major piece of code. -- justin
Could be I'm way off too.
Sander
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM
threaded
Posted by rb...@covalent.net.
On Sat, 14 Jul 2001, Justin Erenkrantz wrote:
> On Sat, Jul 14, 2001 at 09:13:08PM +0200, Sander Striker wrote:
> > The way I see it, each process has a single pool instance as the parent
> > for all the threads. Resetting or destroying that pool should effectively
> > kill all threads. What am I missing?
>
> As I see it, the problem is:
>
> [ Platforms with SAFE_ACCEPT == APR_SUCCESS rather than the lock don't
> apply here. ]
>
> All threads are stuck in the accept mutex except for one. It reads the
> POD. And, it is time to die. It sets workers_may_exit to 1. It then
> releases the accept mutex and exits. Another *child process* acquires
> the accept mutex. Depending upon how many child processes you have, it
> may take a while for all of the threads in the doomed child process to
> acquire the mutex, wake up, check workers_may_exit, release the mutex,
> and then exit. Only when all of the threads in the doomed child process
> are exited does any cleanup occur.
This is why the original code, in apache-apr, written by Manoj back at IBM
used two mutexes. The first was cross-process, the second was
cross-threads. It allowed us to handle this much cleaner. I didn't
realize this had been removed.
> And, you can't kick the thread out of the mutex acquire (pthread_cancel
> or similar strategies don't cancel a mutex operation), so you are
> screwed. And, destroying its parent pool does *NOT* destroy the
> thread. Look at the code again. The only person that can call
> pthread_exit() is the actual thread itself. You can't call
> pthread_exit on behalf of another thread (i.e. from the thread that
> knows it is doomed or a cleanup thread). It just doesn't work like
> that.
>
> I just don't agree with Ryan's assessment here. But, it's possible
> I'm missing some major piece of code. -- justin
What you are missing is the history. All of the problems that we are
trying to solve were solved already in the original code. These problem
are solved most easily by using two mutexes, one for keeping only one
process in the inner loop, and the inner one for the threads in the
process. Then, when you are killing a process, you can easily wake up all
the threads in the process that are sitting on a lock.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Sat, Jul 14, 2001 at 09:13:08PM +0200, Sander Striker wrote:
> The way I see it, each process has a single pool instance as the parent
> for all the threads. Resetting or destroying that pool should effectively
> kill all threads. What am I missing?
As I see it, the problem is:
[ Platforms with SAFE_ACCEPT == APR_SUCCESS rather than the lock don't
apply here. ]
All threads are stuck in the accept mutex except for one. It reads the
POD. And, it is time to die. It sets workers_may_exit to 1. It then
releases the accept mutex and exits. Another *child process* acquires
the accept mutex. Depending upon how many child processes you have, it
may take a while for all of the threads in the doomed child process to
acquire the mutex, wake up, check workers_may_exit, release the mutex,
and then exit. Only when all of the threads in the doomed child process
are exited does any cleanup occur.
These sibling threads are blocked until they receive the mutex. They
aren't going to be going anywhere until the mutex is turned over to
them. They can't check the value of workers_may_exit *until* they
acquire the mutex. Condition variables help *somewhat*, but the
problem is now of scope - you have an interprocess condition and a
intraprocess condition to check. Does this work? I don't know.
And, you can't kick the thread out of the mutex acquire (pthread_cancel
or similar strategies don't cancel a mutex operation), so you are
screwed. And, destroying its parent pool does *NOT* destroy the
thread. Look at the code again. The only person that can call
pthread_exit() is the actual thread itself. You can't call
pthread_exit on behalf of another thread (i.e. from the thread that
knows it is doomed or a cleanup thread). It just doesn't work like
that.
I just don't agree with Ryan's assessment here. But, it's possible
I'm missing some major piece of code. -- justin
Thread pools are isolated WAS Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
> Ah, ok, I'm getting the picture now. Well, for my purposes, the child
> pool creation doesn't have to be in apr_thread_create. There doesn't
> have to be child pool creation at all. But, I do want the worker fn
> to be able to create a child pool from the pool passed into
> apr_thread_create. So, we need a way to pass in the data to the
> worker fn.
No. I thought we were going to per-thread pools. And, this is why we
were going this way. This removes all of the notions of cleanups from
a parent trying to forcibly cleanup the thread. IMHO, there should be
no relationship between the parent SMS and its threads' SMS. -- justin
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
> > I have some use cases that just want a thread. Since no child-pool is
> > required in this case, it becomes unnecessary overhead (that is currently
> > broken, as the child-pool is only cleaned in apr_thread_exit() but that
> > whole thing is screwey).
>
> Again, that is incorrect. The THREAD-pool is cleaned whenever the parent
> pool is cleaned. Yes, there is some extra overhead, but remember that the
> problem with pools (as the originally existed, and are still implemented
> in the server), is that you can't have two threads working with the same
> pool.
You are correct, I failed to mention that it is not *only* cleaned in
apr_thread_exit(), but also when the parent pool is cleaned. It is,
however, inappropriate to assume that the parent pool will ever get
cleaned.
Right now, if my application were to have a global/root pool that
repeatedly created short-lived threads (as it does, BTW), it would
eventually run out of memory. So, like I said, if apr_thread_exit()
is not called (and in all of apr, apr-util, and httpd is is never
called, not even in the test* programs), then that application is wholly
dependent on the ability of the thread's parent pool to clean up the
memory.
> IF SMS's replace all pools, then that might change. As things stand
> today though, that hasn't happened, and may not happen quickly.
>
> > So if apr threads didn't know anything about child-pools, then they
> > would just simply create a thread that called my worker_function().
> > I could pass in my opaque data to that worker_function(), and do
> > the child-pool creations and destruction right there. I just don't see
> > how child-pool creation MUST be in apr_thread_create()
>
> Yes, you could do that, but then EVERYBODY who wanted to create a thread
> would have to do that, and why are we forcing people to duplicate that
> code everytime a thread is created?
I agree that this is a useful function *at some level*, and so I propose
that this particular utility be built on top of a lower-level set of APR
thread routines. I will even write these routines myself, just because
I believe they are useful.
-aaron
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
> > I have some use cases that just want a thread. Since no child-pool is
> > required in this case, it becomes unnecessary overhead (that is currently
> > broken, as the child-pool is only cleaned in apr_thread_exit() but that
> > whole thing is screwey).
>
> Again, that is incorrect. The THREAD-pool is cleaned whenever the parent
> pool is cleaned. Yes, there is some extra overhead, but remember that the
> problem with pools (as the originally existed, and are still implemented
> in the server), is that you can't have two threads working with the same
> pool.
You are correct, I failed to mention that it is not *only* cleaned in
apr_thread_exit(), but also when the parent pool is cleaned. It is,
however, inappropriate to assume that the parent pool will ever get
cleaned.
Right now, if my application were to have a global/root pool that
repeatedly created short-lived threads (as it does, BTW), it would
eventually run out of memory. So, like I said, if apr_thread_exit()
is not called (and in all of apr, apr-util, and httpd is is never
called, not even in the test* programs), then that application is wholly
dependent on the ability of the thread's parent pool to clean up the
memory.
> IF SMS's replace all pools, then that might change. As things stand
> today though, that hasn't happened, and may not happen quickly.
>
> > So if apr threads didn't know anything about child-pools, then they
> > would just simply create a thread that called my worker_function().
> > I could pass in my opaque data to that worker_function(), and do
> > the child-pool creations and destruction right there. I just don't see
> > how child-pool creation MUST be in apr_thread_create()
>
> Yes, you could do that, but then EVERYBODY who wanted to create a thread
> would have to do that, and why are we forcing people to duplicate that
> code everytime a thread is created?
I agree that this is a useful function *at some level*, and so I propose
that this particular utility be built on top of a lower-level set of APR
thread routines. I will even write these routines myself, just because
I believe they are useful.
-aaron
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Sander Striker <st...@apache.org>.
> > Why are we so desperate in opting out the child-pool creation?
> > I don't really have problems with a child pool for each thread.
> Actually,
> > it will make the dynamic locking a lot easier to implement if it stays.
>
> I have some use cases that just want a thread. Since no child-pool is
> required in this case, it becomes unnecessary overhead (that is currently
> broken, as the child-pool is only cleaned in apr_thread_exit() but that
> whole thing is screwey).
>
> So if apr threads didn't know anything about child-pools, then they
> would just simply create a thread that called my worker_function().
> I could pass in my opaque data to that worker_function(), and do
> the child-pool creations and destruction right there. I just don't see
> how child-pool creation MUST be in apr_thread_create()
Ah, ok, I'm getting the picture now. Well, for my purposes, the child
pool creation doesn't have to be in apr_thread_create. There doesn't
have to be child pool creation at all. But, I do want the worker fn
to be able to create a child pool from the pool passed into
apr_thread_create. So, we need a way to pass in the data to the
worker fn.
The only thing I need is something like this for the sms locking scheme:
struct thread_data
{
...
apr_sms_t *sms;
void (*worker_fn)(void *data);
void *data;
};
void thread_stub(void *data)
{
apr_sms_t *pms;
apr_os_thread_t thread;
struct thread_data *td = (struct thread_data *)data;
thread = apr_os_thread_current();
pms = td->sms;
apr_sms_thread_register(pms, thread);
td->worker_fn(td->data);
apr_sms_thread_unregister(pms, thread);
}
In apr_thread_create an instance of thread_data is created
and initialized, then instead of running the worker function
in the thread, we launch the stub which in turn calls the
worker.
Possibly we can use the stub for more than only this?
Sander
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Sander Striker <st...@apache.org>.
> > Why are we so desperate in opting out the child-pool creation?
> > I don't really have problems with a child pool for each thread.
> Actually,
> > it will make the dynamic locking a lot easier to implement if it stays.
>
> I have some use cases that just want a thread. Since no child-pool is
> required in this case, it becomes unnecessary overhead (that is currently
> broken, as the child-pool is only cleaned in apr_thread_exit() but that
> whole thing is screwey).
>
> So if apr threads didn't know anything about child-pools, then they
> would just simply create a thread that called my worker_function().
> I could pass in my opaque data to that worker_function(), and do
> the child-pool creations and destruction right there. I just don't see
> how child-pool creation MUST be in apr_thread_create()
Ah, ok, I'm getting the picture now. Well, for my purposes, the child
pool creation doesn't have to be in apr_thread_create. There doesn't
have to be child pool creation at all. But, I do want the worker fn
to be able to create a child pool from the pool passed into
apr_thread_create. So, we need a way to pass in the data to the
worker fn.
The only thing I need is something like this for the sms locking scheme:
struct thread_data
{
...
apr_sms_t *sms;
void (*worker_fn)(void *data);
void *data;
};
void thread_stub(void *data)
{
apr_sms_t *pms;
apr_os_thread_t thread;
struct thread_data *td = (struct thread_data *)data;
thread = apr_os_thread_current();
pms = td->sms;
apr_sms_thread_register(pms, thread);
td->worker_fn(td->data);
apr_sms_thread_unregister(pms, thread);
}
In apr_thread_create an instance of thread_data is created
and initialized, then instead of running the worker function
in the thread, we launch the stub which in turn calls the
worker.
Possibly we can use the stub for more than only this?
Sander
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with
MPM threaded
Posted by rb...@covalent.net.
On Sun, 15 Jul 2001, Aaron Bannert wrote:
> On Sun, Jul 15, 2001 at 07:16:35PM +0200, Sander Striker wrote:
> > > Fair enough. It's just that in order to opt-out of the child-pool creating
> > > process in apr_thread_create, we're going to have to add a parameter
> >
> > Why are we so desperate in opting out the child-pool creation?
> > I don't really have problems with a child pool for each thread. Actually,
> > it will make the dynamic locking a lot easier to implement if it stays.
>
> I have some use cases that just want a thread. Since no child-pool is
> required in this case, it becomes unnecessary overhead (that is currently
> broken, as the child-pool is only cleaned in apr_thread_exit() but that
> whole thing is screwey).
Again, that is incorrect. The THREAD-pool is cleaned whenever the parent
pool is cleaned. Yes, there is some extra overhead, but remember that the
problem with pools (as the originally existed, and are still implemented
in the server), is that you can't have two threads working with the same
pool.
IF SMS's replace all pools, then that might change. As things stand
today though, that hasn't happened, and may not happen quickly.
> So if apr threads didn't know anything about child-pools, then they
> would just simply create a thread that called my worker_function().
> I could pass in my opaque data to that worker_function(), and do
> the child-pool creations and destruction right there. I just don't see
> how child-pool creation MUST be in apr_thread_create()
Yes, you could do that, but then EVERYBODY who wanted to create a thread
would have to do that, and why are we forcing people to duplicate that
code everytime a thread is created?
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with
MPM threaded
Posted by rb...@covalent.net.
On Sun, 15 Jul 2001, Aaron Bannert wrote:
> On Sun, Jul 15, 2001 at 07:16:35PM +0200, Sander Striker wrote:
> > > Fair enough. It's just that in order to opt-out of the child-pool creating
> > > process in apr_thread_create, we're going to have to add a parameter
> >
> > Why are we so desperate in opting out the child-pool creation?
> > I don't really have problems with a child pool for each thread. Actually,
> > it will make the dynamic locking a lot easier to implement if it stays.
>
> I have some use cases that just want a thread. Since no child-pool is
> required in this case, it becomes unnecessary overhead (that is currently
> broken, as the child-pool is only cleaned in apr_thread_exit() but that
> whole thing is screwey).
Again, that is incorrect. The THREAD-pool is cleaned whenever the parent
pool is cleaned. Yes, there is some extra overhead, but remember that the
problem with pools (as the originally existed, and are still implemented
in the server), is that you can't have two threads working with the same
pool.
IF SMS's replace all pools, then that might change. As things stand
today though, that hasn't happened, and may not happen quickly.
> So if apr threads didn't know anything about child-pools, then they
> would just simply create a thread that called my worker_function().
> I could pass in my opaque data to that worker_function(), and do
> the child-pool creations and destruction right there. I just don't see
> how child-pool creation MUST be in apr_thread_create()
Yes, you could do that, but then EVERYBODY who wanted to create a thread
would have to do that, and why are we forcing people to duplicate that
code everytime a thread is created?
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Sun, Jul 15, 2001 at 07:16:35PM +0200, Sander Striker wrote:
> > Fair enough. It's just that in order to opt-out of the child-pool creating
> > process in apr_thread_create, we're going to have to add a parameter
>
> Why are we so desperate in opting out the child-pool creation?
> I don't really have problems with a child pool for each thread. Actually,
> it will make the dynamic locking a lot easier to implement if it stays.
I have some use cases that just want a thread. Since no child-pool is
required in this case, it becomes unnecessary overhead (that is currently
broken, as the child-pool is only cleaned in apr_thread_exit() but that
whole thing is screwey).
So if apr threads didn't know anything about child-pools, then they
would just simply create a thread that called my worker_function().
I could pass in my opaque data to that worker_function(), and do
the child-pool creations and destruction right there. I just don't see
how child-pool creation MUST be in apr_thread_create()
-aaron
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Sun, Jul 15, 2001 at 07:16:35PM +0200, Sander Striker wrote:
> > Fair enough. It's just that in order to opt-out of the child-pool creating
> > process in apr_thread_create, we're going to have to add a parameter
>
> Why are we so desperate in opting out the child-pool creation?
> I don't really have problems with a child pool for each thread. Actually,
> it will make the dynamic locking a lot easier to implement if it stays.
I have some use cases that just want a thread. Since no child-pool is
required in this case, it becomes unnecessary overhead (that is currently
broken, as the child-pool is only cleaned in apr_thread_exit() but that
whole thing is screwey).
So if apr threads didn't know anything about child-pools, then they
would just simply create a thread that called my worker_function().
I could pass in my opaque data to that worker_function(), and do
the child-pool creations and destruction right there. I just don't see
how child-pool creation MUST be in apr_thread_create()
-aaron
RE: DCEthreads (was Re: Pools in threads)
Posted by Sander Striker <st...@apache.org>.
>>> And, it would add 3.5 million lines of code to APR.
>>
>> Nah. It would add the lines to add thread cancellation. The
>> 3.5 million LOC is the _entire_ dce rpc codebase. Only part
>> of that is the dce threads library. And, I believe there are
>> some old dce/rpc team members in our midst who could know how
>> this all fit together. I could be wrong though.
>
> [warning: tangent -- i changed the subject appropriately]
>
> I must admit that it sounds tantalizing, but I have a comment and a
> question:
>
> - It appears that the Free DCE package quoted above is under the GPL
> and is therefore incompatible with the Apache License.
Yes. However, I wasn't suggesting using or copying code from the
free dce package. You can merely examine it and decide whether
implementing thread cancellation in such a way is something we want
in APR. Then, you write your own implementation.
Furthermore, I would like to compare the Free DCE threads implementation
with the DCE/RPC threads implementation. Then pick the best design
decisions from both.
> - What is meant by "*emulated* on top of POSIX threads"? What are the
> tradeoffs, if any, compared to a implementation that is native to
> the OS?
Sander
DCEthreads (was Re: Pools in threads)
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Mon, Jul 16, 2001 at 09:05:40AM +0200, Sander Striker wrote:
> > On Mon, Jul 16, 2001 at 01:54:14AM +0200, Luke Kenneth Casson
> > Leighton wrote:
> > > ah, yes, but it _is_ supported by DCEthreads - see
> > > http://sourceforge.net/projects/freedce which provides,
> > > horror-of-horrors, thread cancellation *emulated* on
> > > top of POSIX threads.
> > [snip, snip]
> > > ... but what i am basically saying is, on NT and Unix,
> > > thread cancellation _is_ possible, it's just that i
> > > think you might not be too happy about the coding-route
> > > you might have to take to _do_ it :) :)
> >
> > Possible, but not robust.
>
> DCE/RPC is _very_ robust. That being the case its underlying
> mechanisms are very likely to be robust too.
>
> > And, it would add 3.5 million lines of code to APR.
>
> Nah. It would add the lines to add thread cancellation. The
> 3.5 million LOC is the _entire_ dce rpc codebase. Only part
> of that is the dce threads library. And, I believe there are
> some old dce/rpc team members in our midst who could know how
> this all fit together. I could be wrong though.
[warning: tangent -- i changed the subject appropriately]
I must admit that it sounds tantalizing, but I have a comment and a
question:
- It appears that the Free DCE package quoted above is under the GPL
and is therefore incompatible with the Apache License.
- What is meant by "*emulated* on top of POSIX threads"? What are the
tradeoffs, if any, compared to a implementation that is native to
the OS?
-aaron
Re: Pools in threads
Posted by Luke Kenneth Casson Leighton <lk...@samba-tng.org>.
> Nah. It would add the lines to add thread cancellation. The
> 3.5 million LOC is the _entire_ dce rpc codebase.
which includes about 15 separate services such as DCE/DFS,
which is actually AFS [without the bugfixes and with a
new RPC mechanism]
which you don't need, as sander says.
> Only part
> of that is the dce threads library. And, I believe there are
> some old dce/rpc team members in our midst who could know how
> this all fit together. I could be wrong though.
i'll ask. might get a response tomorrow.
luke
RE: Pools in threads
Posted by Sander Striker <st...@apache.org>.
> On Mon, Jul 16, 2001 at 01:54:14AM +0200, Luke Kenneth Casson
> Leighton wrote:
> > ah, yes, but it _is_ supported by DCEthreads - see
> > http://sourceforge.net/projects/freedce which provides,
> > horror-of-horrors, thread cancellation *emulated* on
> > top of POSIX threads.
> [snip, snip]
> > ... but what i am basically saying is, on NT and Unix,
> > thread cancellation _is_ possible, it's just that i
> > think you might not be too happy about the coding-route
> > you might have to take to _do_ it :) :)
>
> Possible, but not robust.
DCE/RPC is _very_ robust. That being the case its underlying
mechanisms are very likely to be robust too.
> And, it would add 3.5 million lines of code to APR.
Nah. It would add the lines to add thread cancellation. The
3.5 million LOC is the _entire_ dce rpc codebase. Only part
of that is the dce threads library. And, I believe there are
some old dce/rpc team members in our midst who could know how
this all fit together. I could be wrong though.
Sander
Re: Pools in threads
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Mon, Jul 16, 2001 at 01:54:14AM +0200, Luke Kenneth Casson Leighton wrote:
> ah, yes, but it _is_ supported by DCEthreads - see
> http://sourceforge.net/projects/freedce which provides,
> horror-of-horrors, thread cancellation *emulated* on
> top of POSIX threads.
[snip, snip]
> ... but what i am basically saying is, on NT and Unix,
> thread cancellation _is_ possible, it's just that i
> think you might not be too happy about the coding-route
> you might have to take to _do_ it :) :)
Possible, but not robust. And, it would add 3.5 million lines
of code to APR. Well, we could the APROS in that many LOC. =)
Kind of like Quake - have an OS inside of your web-server.
Thread cancellation just opens up a can of worms not because the OS
doesn't support them, but because their semantics are ill-defined.
We'd have to add cancellation points to our code. Not going to
happen. Well, I'm not going to do it. I might even veto it out
of pure spite. =) -- justin
Re: Pools in threads
Posted by Luke Kenneth Casson Leighton <lk...@samba-tng.org>.
On Sun, Jul 15, 2001 at 04:43:37PM -0700, Justin Erenkrantz wrote:
> On Mon, Jul 16, 2001 at 12:46:31AM +0200, Luke Kenneth Casson Leighton wrote:
> > > In order to provide a win against the current pool code in a threaded
> > > MPM, we *need* to have thread-specific SMS that have no locks or
> > > association to anything other than a simple unlocked (from APR's
> > > perspective) malloc/free (aka std) SMS. -- justin
> >
> > okay.
> >
> > well... uhmmm... this is going to sound odd. i'm not even sure
> > if it will help, because i am a bit out of my depth in understanding
> > the problem.
> >
> > how about a 'pass-through' sms for threads?
>
> I'm not sure I'm following you. The thread SMS never needs locks.
i am not necessarily referring to locks.
if there is something _ever_ so slightly different about the
circumstances under which sms instances need to be used, then
what i am suggesting is that instead of doing this:
apr_sms_general_t
apr_sms_thread_capable_general_t
[which was what you were recommending ;)]
apr_sms_trivial_t
apr_sms_thread_capable_trivial_t
instead having a:
apr_sms_thread_wrapping_redirector_t
and use *that* to do the thread-capable-very-specific-things
that i suspect you will need to do for all-and-any SMSes,
pretty much, and that might not necessarily be locking,
which simplifies the coding needed.
> Sander was under the assumption that the parent of a thread can clean up
> a thread. That isn't supported with POSIX pthreads (cleanly).
> Apparently, Win32 has some support for this. This lack basically kills
> off the parent/child relationship between SMSes across threads.
ah, yes, but it _is_ supported by DCEthreads - see
http://sourceforge.net/projects/freedce which provides,
horror-of-horrors, thread cancellation *emulated* on
top of POSIX threads.
DCEthreads is an implementation of POSIX (draft 4) threads
which of course drastically changed from the final
release.
now, if you examine Windows NT threads, i _bet_ they
support thread cancellation properly, because they
do support thread cancellation via dce/rpc (and dcom,
which is basically an object-orientated interface on
top of dce/rpc).
why? because, well, that's a long story.
also, take a lok at www.opengroup.org/dce, download the
88mb of the dce 1.22 source code, unpack it to its 430mb
uncompressed size [no, i'm not kidding: yes, that's right,
four hundred and thirty megabytes - 3.5 million lines of code,
i had to use xargs to do wc on them], and you will find that there is
DCEthreads supported for AIX, HP/UX, Solaris, DG/UX, SGI's Unix
(forgotten what it is! IRIX *duh* :)
... but what i am basically saying is, on NT and Unix,
thread cancellation _is_ possible, it's just that i
think you might not be too happy about the coding-route
you might have to take to _do_ it :) :)
> pthread_cancel() gets us into all sorts of problems we don't want to
> deal with. It has been discussed years ago between Ryan, Dean and
> Manoj - see around 08-1999 - "First in a long line of APR related
> patches". And, it isn't guaranteed to work, either (mutex acquires
> are *not* interruptable). In that same thread, Ryan also brings up a
> precursor to SMS. Rather interesting thread, really. (See
> www.apachelabs.org if you don't have a favorite new-httpd archive.)
>
> Unless you have a clean way of killing off a thread from a parent,
> thread-based SMS must be directly parented from a basic non-locking
> SMS (like std SMS).
>
> I'm open to any creative suggestions that work. I'm just not aware
> of any that work. -- justin
:)
Re: Pools in threads
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Mon, Jul 16, 2001 at 12:46:31AM +0200, Luke Kenneth Casson Leighton wrote:
> > In order to provide a win against the current pool code in a threaded
> > MPM, we *need* to have thread-specific SMS that have no locks or
> > association to anything other than a simple unlocked (from APR's
> > perspective) malloc/free (aka std) SMS. -- justin
>
> okay.
>
> well... uhmmm... this is going to sound odd. i'm not even sure
> if it will help, because i am a bit out of my depth in understanding
> the problem.
>
> how about a 'pass-through' sms for threads?
I'm not sure I'm following you. The thread SMS never needs locks.
Sander was under the assumption that the parent of a thread can clean up
a thread. That isn't supported with POSIX pthreads (cleanly).
Apparently, Win32 has some support for this. This lack basically kills
off the parent/child relationship between SMSes across threads.
pthread_cancel() gets us into all sorts of problems we don't want to
deal with. It has been discussed years ago between Ryan, Dean and
Manoj - see around 08-1999 - "First in a long line of APR related
patches". And, it isn't guaranteed to work, either (mutex acquires
are *not* interruptable). In that same thread, Ryan also brings up a
precursor to SMS. Rather interesting thread, really. (See
www.apachelabs.org if you don't have a favorite new-httpd archive.)
Unless you have a clean way of killing off a thread from a parent,
thread-based SMS must be directly parented from a basic non-locking
SMS (like std SMS).
I'm open to any creative suggestions that work. I'm just not aware
of any that work. -- justin
Re: Pools in threads
Posted by Luke Kenneth Casson Leighton <lk...@samba-tng.org>.
> In order to provide a win against the current pool code in a threaded
> MPM, we *need* to have thread-specific SMS that have no locks or
> association to anything other than a simple unlocked (from APR's
> perspective) malloc/free (aka std) SMS. -- justin
okay.
well... uhmmm... this is going to sound odd. i'm not even sure
if it will help, because i am a bit out of my depth in understanding
the problem.
how about a 'pass-through' sms for threads?
all it does is to create an apr_sms_thread_passthrough() is:
memcpy(&thr_pthru, &someothersms_api);
then OVERRIDE the create function, calling the *someothersms*->create.
you then do your 'thread-specific' stuff that you need to do,
e.g. locking _whatever_, i really don't know, as a wrapper around
the someothersms.
in other words, via this technique, you can turn a non-thread-capable
SMS into a thread-safe one.
... does that sound a bit mad, but useful, or am i way off the mark.
luke
Re: Pools in threads
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Sun, Jul 15, 2001 at 07:52:46PM +0200, Sander Striker wrote:
> Can't we pass in a data structure to the thread function which has
> all the required information? The data the user wants to pass in will
> need to be in the data structure aswell ofcourse.
-0.9. I don't like the idea of marshaling the data to the thread's
worker function.
I'm seeing the proposed marshaling like so:
struct {
void *data;
apr_pool_t *pool;
} apr_thread_starter_t;
void * worker_function(void *v)
{
apr_thread_starter *ts = v;
whatever_the_thread_really_wants_t *d = ts->data;
...
}
apr_thread_create(blah, blah, blah)
{
apr_thread_starter *ts = apr_palloc(...);
ts->data = the-data-that-user-really-wants-to-pass-in;
ts->sms = apr_pool_create(blah, blah, blah);
thread_create(go call worker_function with ts as the sole argument)
}
Yuck.
Since I'm seeing the thread SMS be completely separate from the parent
SMS, I'm just not sure why we even want to pass the sms along.
Unless you can make a case that the thread's SMS needs to know about the
parent SMS. If so, then we've got major problems in our SMS design.
In order to provide a win against the current pool code in a threaded
MPM, we *need* to have thread-specific SMS that have no locks or
association to anything other than a simple unlocked (from APR's
perspective) malloc/free (aka std) SMS. -- justin
Re: Design of pools and threads
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Mon, Jul 16, 2001 at 10:58:17AM -0700, Brian Pane wrote:
> * The parent pool should control the cleanup of its children
> to ensure timely resource release (important for memory, but
> even more important for file descriptors). In the case of
> an httpd with a high request volume, this cleanup needs to
> happen for a request and all its subrequests right after
> the response is sent; otherwise we
Correct except for the case of a parent/child relationship across
threads as the parent can't cleanup the children in this case.
FWIW, I'm not talking about the SMS used for a request. It could be
cleaned up (i.e. reset) after the request is completed (exactly like
it is now - no code changes). It'd free the resources - just like now.
We're simply divorcing its ancestor (the per-thread SMS) from the
per-process SMS. The children of this per-thread SMS still have the
same manner of operation as before. -- justin
Re: Design of pools and threads
Posted by Brian Pane <bp...@pacbell.net>.
Justin Erenkrantz wrote:
[...]
>- Why do we want an independent per-thread SMS? Because it now removes
> the requirement for locking. A thread may not be reentrant upon
> itself - executing in two places at once. It's a thread with one flow
> of control. Memory allocation without the need for locking should
> be a good thing. In order to scale well with threads, I believe httpd
> requires independent (thread-local free list) per-thread SMS.
>- Does merely having a per-thread SMS (and its associated thread-local
> free list) necessitate a break from its parent SMS (i.e. independent)?
> Yes, I believe so. If we were to assume that the per-thread SMS had
> a parent/child relationship to the per-process SMS, then we must now
> have locking. By design, a SMS must allocate memory from its parent.
> Since the parent SMS would now be reentrant (as the parent, it has
> spawned multiple threads that are executing in parallel all asking it
> for memory), the parent SMS must acquire a lock before allocating
> memory. This lock is the very thing we are trying to avoid in the
> first place. Also, when a parent/child relationship exists between
> two SMS, the parent may clean up the child's SMS when it is destroyed.
>- However, for reasons described above, it is impossible for the parent
> to clean up its children threads forcibly. The only thing the parent
> can do is wait for the thread to exit on its own behalf (via
> apr_thread_join). The parent may only make hints (such as
> workers_may_exit in threaded MPM) to the thread that it should exit.
>- So, would having an independent per-thread SMS break any assumptions
> in APR or httpd? I don't think so. Why? I believe that the thread
> is a logical breaking-off point for memory allocation. If we were to
> assume a thread were to create a SMS when it begins, all subsequent
> processing in that thread will use this SMS. When this independent
> per-thread SMS is destroyed (as the thread is exiting - remember we
> have assumed that the thread must exit voluntarily), the memory is
> reclaimed. All descriptors or sockets opened during that threads'
> life is now returned to the OS.
>
I really think we need a hybrid model that splits the two roles
of a parent in the SMS design. One of my messages in the archives
talks about this in more detail, but the basic idea is:
* The parent pool should control the cleanup of its children
to ensure timely resource release (important for memory, but
even more important for file descriptors). In the case of
an httpd with a high request volume, this cleanup needs to
happen for a request and all its subrequests right after
the response is sent; otherwise we
* But the parent doesn't need to be where the children go
to get blocks when they need more memory; instead, they
should call directly to the most efficient source of memory
that's suitable for the application (e.g., a per-thread SMS).
--Brian
Design of pools and threads
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Sun, Jul 15, 2001 at 07:20:28PM -0700, rbb@covalent.net wrote:
> Pools are by their very nature hierarchical. That is why the relationship
> is there, and why it needs to remain there. You can't just rely on the
> thread to cleanup itself. Pools (and SMS's) are used for a LOT more than
> just memory allocation in the server. They make sure that descriptors are
> closed correctly. In some places, they are used to make sure that
> anything the child opened that isn't returned to the OS automatically is
> returned when the child goes away.
>
> If you divorce the thread pool from the pool for the child, you WILL break
> many assumptions in the server.
Here is what I understand the situation to be. Please correct me if
any of my statements are incorrect.
- A parent may not terminate any child threads forcibly (via a system
call or other mechanism). The best mechanism we can hope for (at this
time - again no real progress has been made on alternatives) is to
indicate via some way that the thread can honor the request to
terminate gracefully.
- For various safety/robustness issues, we have decided to not implement
a pthread_cancel()-like mechanism. There are three reasons for this -
we may arbitrarily cancel the thread even if it is in the middle of an
important transaction and certain operations (acquiring mutexes) are
non-cancellable. (It is true that we could set cancellation points
in the APR-using programs to prevent important transactions from being
cancelled.) Furthermore, when issuing a pthread_cancel-like call,
certain OSes may leak the resources of a thread. This is the third
consideration to implementing this option. With all of these
factors in consideration, Dean, Manoj, and you (and others) decided
not to implement a pthread_cancel-like mechanism in APR. I do think
that this was the correct decision.
- Since APR has no way of forcing a thread to exit, this agreement lives
entirely outside of the bounds of APR. What I mean by this is that
the APR library can not provide this thread-exit mechanism
transparently to APR-using programs via cleanups. Only the thread
itself may choose to shut down. If the thread so wishes, it may
consult any mechanisms it desires (such as shared memory, sockets,
pipes, etc.). But, we have no clean way for a parent to kill a child
thread that does not wish to exit.
- As an example of what I mean, please look at the threaded MPM. Each
worker thread is in control of when they may exit, and the mechanism
for determining when to exit lies outside of APR (via the
process-local workers_may_exit variable and the POD).
- You have suggested that a thread pool has a relationship to its parent
(from the process before thread was created). I believe this
assumption is invalid IF we are to attempt an independent per-thread
SMS. And, I believe that a per-process memory model does not work
with a threaded httpd architecture.
- Why do we want an independent per-thread SMS? Because it now removes
the requirement for locking. A thread may not be reentrant upon
itself - executing in two places at once. It's a thread with one flow
of control. Memory allocation without the need for locking should
be a good thing. In order to scale well with threads, I believe httpd
requires independent (thread-local free list) per-thread SMS.
- Does merely having a per-thread SMS (and its associated thread-local
free list) necessitate a break from its parent SMS (i.e. independent)?
Yes, I believe so. If we were to assume that the per-thread SMS had
a parent/child relationship to the per-process SMS, then we must now
have locking. By design, a SMS must allocate memory from its parent.
Since the parent SMS would now be reentrant (as the parent, it has
spawned multiple threads that are executing in parallel all asking it
for memory), the parent SMS must acquire a lock before allocating
memory. This lock is the very thing we are trying to avoid in the
first place. Also, when a parent/child relationship exists between
two SMS, the parent may clean up the child's SMS when it is destroyed.
- However, for reasons described above, it is impossible for the parent
to clean up its children threads forcibly. The only thing the parent
can do is wait for the thread to exit on its own behalf (via
apr_thread_join). The parent may only make hints (such as
workers_may_exit in threaded MPM) to the thread that it should exit.
- So, would having an independent per-thread SMS break any assumptions
in APR or httpd? I don't think so. Why? I believe that the thread
is a logical breaking-off point for memory allocation. If we were to
assume a thread were to create a SMS when it begins, all subsequent
processing in that thread will use this SMS. When this independent
per-thread SMS is destroyed (as the thread is exiting - remember we
have assumed that the thread must exit voluntarily), the memory is
reclaimed. All descriptors or sockets opened during that threads'
life is now returned to the OS.
- Furthermore, we know that the process may not exit before the children
threads may exit. For the entire lifetime of the thread, any
process-local variables in use by the threads must remain valid.
This is inherent in the threading model of all the operating systems
we encounter. A malicious parent indeed, could reset its SMS
*before* allowing all of its children to exit. But, let's not worry
about that (or any error conditions - we're screwed anyway).
- Again, let's consider the case of the threaded MPM. The parent of
the threads (i.e. child process) will only destroy the process-level
SMS when it calls clean_child_exit(). Any per-process open
descriptors are now closed as the process terminates. This call
occurs after the call to apr_thread_join() on all started threads.
No more child threads are present when this code is called. Again,
this code is correct.
I hope I now have made my position and rationale clearer. Again, I may
have misinterpreted code or design or made blatantly wrong statements.
I heartily welcome any feedback or corrections. I am human -
therefore, I am usually wrong.
Respectfully yours,
Justin Erenkrantz
jerenkrantz@ebuilt.com
Re: Pools in threads WAS Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Luke Kenneth Casson Leighton <lk...@samba-tng.org>.
> There is *zero* benefit to having any relationship between the
> thread's memory pool and the parent's memory pool. You can't cleanup
> the thread from the parent anyway, so trust the thread to cleanup
> itself (as well as its pool and any memory allocations). I fail to
> see the problem here. Sever the imagined relationship. The code
> becomes simpler, too. -- justin
coooool. problem-solving by conventions. follow these rules,
everything's okay.
Re: Pools in threads WAS Re: Terminating threads in a process, WAS:
RE: [PATCH] Problems with MPM threaded
Posted by rb...@covalent.net.
On Sun, 15 Jul 2001, Justin Erenkrantz wrote:
> On Sun, Jul 15, 2001 at 02:33:54PM -0700, rbb@covalent.net wrote:
> > Guys, before you make comments like this, you should really read the code.
> > First of all, the thread_exit call needs to take in the thread variable.
> > Without it, platforms like OS/2 can't return values from the thread exit
> > routine.
>
> We could stipulate in APR that on OS/2 we won't return the result
> code. Big deal. It's not the end of the world. And, it's only a hack
> that works in conjunction with OS/2's apr_thread_join. OS/2 threads
> obviously don't support return codes. APR is only as good as the
> underlying thread library.
No, because that is utterly bogus. The whole point of APR, is to allow a
program to be written that will work on ANY platform. There are very few
features that can't be implemented cross platform.
> If you are willing to lose the return code on OS/2, you now have OS/2
> not needing the thread structure either. APR is a game of tradeoffs -
> this is one we should make in order to get a better API at the cost of
> something that this native OS doesn't provide anyway.
No. APR is not a game of tradeoffs. It has never been a game of
tradeoffs. We only make decisions like this when it is absolutely
impossible to implement a feature on a given system. As we have already
shown, this can be implemented.
> We should be seeing red flags everywhere because our own code isn't
> calling the apr_thread_exit() function. Why? The API is, uh,
> lacking.
No, that is not why we don't call apr_thread_exit. We don't call
apr_thread_exit, because we don't need it. There are two ways for threads
to terminate. 1) Call apr_thread_exit. 2) return from the function
that was started when the thread was created. We happen to use option 2,
because it fits our model better. Do not misinterpret a choice for a
flaw. Even before the APR threading code was written, there was a hybrid
thread/process version of Apache. Versions of this can be found in the
apache-apr repository. That original hybrid Apache did not EVER call
pthread_exit(). This was a conscious choice made by myself, Bill
Stoddard, and Manoj Kasichainula.
> > Secondly, as long as the thread has access to the apr_thread_t, it also
> > has access to the pool within that value.
>
> Well, I disagree that the thread should have access to its underlying
> apr_thread_t. It's a very awkward thing to do. Doable? Certainly.
> Clean? No. You now need a pointer into the thread array (or
> linked-list if that is how you do it - almost all places do it now with
> arrays) passed to the worker function. Gag.
So it has been implemented poorly. That doesn't mean anything. A poor
implementation is very often where things begin. It is relatively simple
to fix the implementation.
> And, once a thread is spawned, it is now on its own for memory. So,
> it should create a SMS independent of any other SMS. That's the goal
> and driving force behind SMS - we need a per-thread SMS.
SMS has nothing to do with threads. I made a mistake when I wrote APR. I
tied it to pools. But pools don't work for all apps. They happen to work
really well for the web server, but most apps don't have memory profiles
like a web server. The idea of an SMS-like memory allocator was discussed
years ago, and the idea was to make APR useful outside of the web server.
> There is *zero* benefit to having any relationship between the
> thread's memory pool and the parent's memory pool. You can't cleanup
> the thread from the parent anyway, so trust the thread to cleanup
> itself (as well as its pool and any memory allocations). I fail to
> see the problem here. Sever the imagined relationship. The code
> becomes simpler, too. -- justin
Pools are by their very nature hierarchical. That is why the relationship
is there, and why it needs to remain there. You can't just rely on the
thread to cleanup itself. Pools (and SMS's) are used for a LOT more than
just memory allocation in the server. They make sure that descriptors are
closed correctly. In some places, they are used to make sure that
anything the child opened that isn't returned to the OS automatically is
returned when the child goes away.
If you divorce the thread pool from the pool for the child, you WILL break
many assumptions in the server.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: Pools in threads WAS Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Sun, Jul 15, 2001 at 02:33:54PM -0700, rbb@covalent.net wrote:
> Guys, before you make comments like this, you should really read the code.
> First of all, the thread_exit call needs to take in the thread variable.
> Without it, platforms like OS/2 can't return values from the thread exit
> routine.
We could stipulate in APR that on OS/2 we won't return the result
code. Big deal. It's not the end of the world. And, it's only a hack
that works in conjunction with OS/2's apr_thread_join. OS/2 threads
obviously don't support return codes. APR is only as good as the
underlying thread library.
BeOS, Unix (pthread), and Win32 don't need the thread structure to
destroy the thread or return the result code. That seems to be the
consensus on how threads should operate.
If you are willing to lose the return code on OS/2, you now have OS/2
not needing the thread structure either. APR is a game of tradeoffs -
this is one we should make in order to get a better API at the cost of
something that this native OS doesn't provide anyway.
We should be seeing red flags everywhere because our own code isn't
calling the apr_thread_exit() function. Why? The API is, uh,
lacking.
> Secondly, as long as the thread has access to the apr_thread_t, it also
> has access to the pool within that value.
Well, I disagree that the thread should have access to its underlying
apr_thread_t. It's a very awkward thing to do. Doable? Certainly.
Clean? No. You now need a pointer into the thread array (or
linked-list if that is how you do it - almost all places do it now with
arrays) passed to the worker function. Gag.
And, once a thread is spawned, it is now on its own for memory. So,
it should create a SMS independent of any other SMS. That's the goal
and driving force behind SMS - we need a per-thread SMS.
There is *zero* benefit to having any relationship between the
thread's memory pool and the parent's memory pool. You can't cleanup
the thread from the parent anyway, so trust the thread to cleanup
itself (as well as its pool and any memory allocations). I fail to
see the problem here. Sever the imagined relationship. The code
becomes simpler, too. -- justin
RE: Pools in threads
Posted by Sander Striker <st...@apache.org>.
>>> Fair enough. It's just that in order to opt-out of the
>>> child-pool creating
>>> process in apr_thread_create, we're going to have to add a parameter
>>
>> Why are we so desperate in opting out the child-pool creation?
>> I don't really have problems with a child pool for each thread.
>> Actually, it will make the dynamic locking a lot easier to implement
>> if it stays.
>
> The problem is that apr_thread_create is completely bogus right now.
>
> The child thread has no access to that pool in apr_thread_t->pool.
> If you want a pool in the thread, you should be creating inside
> of the thread's function not in apr_thread_create.
I can live with that. It's the registration of the thread with
the parent sms [when/if pools==sms] that has my interest :)
> And, apr_thread_exit call should *not* be taking in an apr_thread_t
> because the child doesn't have access to that either.
Can't we pass in a data structure to the thread function which has
all the required information? The data the user wants to pass in will
need to be in the data structure aswell ofcourse.
> We're enforcing the use of global variables to work around this and
> that's plain wrong. -- justin
Yes, I agree.
Sander
Re: Pools in threads WAS Re: Terminating threads in a process, WAS:
RE: [PATCH] Problems with MPM threaded
Posted by rb...@covalent.net.
On Sun, 15 Jul 2001, Justin Erenkrantz wrote:
> On Sun, Jul 15, 2001 at 07:16:35PM +0200, Sander Striker wrote:
> > > Fair enough. It's just that in order to opt-out of the child-pool creating
> > > process in apr_thread_create, we're going to have to add a parameter
> >
> > Why are we so desperate in opting out the child-pool creation?
> > I don't really have problems with a child pool for each thread. Actually,
> > it will make the dynamic locking a lot easier to implement if it stays.
>
> The problem is that apr_thread_create is completely bogus right now.
>
> The child thread has no access to that pool in apr_thread_t->pool.
> If you want a pool in the thread, you should be creating inside
> of the thread's function not in apr_thread_create.
>
> And, apr_thread_exit call should *not* be taking in an apr_thread_t
> because the child doesn't have access to that either.
>
> We're enforcing the use of global variables to work around this and
> that's plain wrong. -- justin
Guys, before you make comments like this, you should really read the code.
First of all, the thread_exit call needs to take in the thread variable.
Without it, platforms like OS/2 can't return values from the thread exit
routine.
Secondly, as long as the thread has access to the apr_thread_t, it also
has access to the pool within that value.
What you all need to realize, is that some of the annoying pieces of APR
exist because we tried to do it the way that makes sense to all of you,
and we had to change it to make it work on all platforms. Please, look
through what ALL platforms do before you decide to go changing API's. You
are going to break things that work today.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Pools in threads WAS Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Sun, Jul 15, 2001 at 07:16:35PM +0200, Sander Striker wrote:
> > Fair enough. It's just that in order to opt-out of the child-pool creating
> > process in apr_thread_create, we're going to have to add a parameter
>
> Why are we so desperate in opting out the child-pool creation?
> I don't really have problems with a child pool for each thread. Actually,
> it will make the dynamic locking a lot easier to implement if it stays.
The problem is that apr_thread_create is completely bogus right now.
The child thread has no access to that pool in apr_thread_t->pool.
If you want a pool in the thread, you should be creating inside
of the thread's function not in apr_thread_create.
And, apr_thread_exit call should *not* be taking in an apr_thread_t
because the child doesn't have access to that either.
We're enforcing the use of global variables to work around this and
that's plain wrong. -- justin
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Luke Kenneth Casson Leighton <lk...@samba-tng.org>.
On Tue, Jul 17, 2001 at 10:12:35AM -0700, Aaron Bannert wrote:
> - Optionally creating a child-pool for each thread. This provides us two
> things in my mind:
> 1) Allowing the application to choose an SMS for this thread-pool (which
> could very well be a special SMS created just for this purpose).
for example, you may wish to create a special 'security' thread.
it uses an apr_sms_mlock_t instance. this special thread handles
all of your security, keeping private keys and plain-text user
passwords in-memory.
luke
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Luke Kenneth Casson Leighton <lk...@samba-tng.org>.
On Tue, Jul 17, 2001 at 10:12:35AM -0700, Aaron Bannert wrote:
> - Optionally creating a child-pool for each thread. This provides us two
> things in my mind:
> 1) Allowing the application to choose an SMS for this thread-pool (which
> could very well be a special SMS created just for this purpose).
for example, you may wish to create a special 'security' thread.
it uses an apr_sms_mlock_t instance. this special thread handles
all of your security, keeping private keys and plain-text user
passwords in-memory.
luke
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
> We have hit an impass in my mind. Dean and I are saying that having each
> thread have it's own pool is a requirement. Not just for httpd, but for
> anything using pools. Dean, if I am mis-interpreting you, then I am
> sorry, and please correct me.
>
> Aaron, you disagree. you want each app to make that decision on their
> own. I see no way to implement that in a thread-safe manner.
(Just as a side note, I don't remember seeing anything refuting the technical
aspects of such an implementation. I would be interested in one, of course,
if you think it is infeasible.)
> So, I would suggest that we approach this problem in another way. My ONLY
> goal for APR is to make a library that is useful for cross-platform
> development and eases the pain of cross-platform development for
> developers. That was the goal when I created APR, and my original test
> case was the Apache web server.
>
> I believe that the problem is that the threaded code is creating the
> pool, and not advertising it to the thread itself. This is an easy thing
> to fix. I do not agree that every APR app that uses threads should have
> to create their own thread pools. That is wasted effort by every APR app.
>
> I would like to see a conclusion to this thread quickly. So, could people
> please post their desires quickly, and what they want to see from this.
In my mind, there are really only two major points that need addressing:
- Optionally creating a child-pool for each thread. This provides us two
things in my mind:
1) Allowing the application to choose an SMS for this thread-pool (which
could very well be a special SMS created just for this purpose).
2) Allowing an application to refrain from creating a child-pool.
- Like you [Ryan] said, it is a problem that the thread creation routines
right now are not advertising the pool to the thread itself. This to
me can be corrected with a better API. The two things I've brought up
here are:
1) void * worker_fn(void *opaque_data) should have an additional
pool param.
2) apr_thread_exit() is unacceptable. It currently requires a
apr_thread_t structure. I would be happy if either the worker_fn was
extended to pass in the apr_thread_t OR if we could architect a way
for apr_thread_exit() to not need the apr_thread_t struct.
I have a feeling the second point will be more palatable, and as such
I'll volunteer to cook up a patch illustrating these changes.
As for the first point, which is the reason this thread has been going on
so long, I'm willing to compromise on the idea of forcefully creating
a childpool always, if it necessary, merely because I see it only as
overhead for some applications, but not a functional defect.
-aaron
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
> > I totally agree, but only as a solution in httpd.
>
> no, everywhere.
>
> > I also believe that we should provide this [application-specific requirement]
> > outside of the basic thread support in APR.
> >
> > Please allow me to use pseudocode:
> >
> > void * worker_function(void * opaque_application_data) {
> >
> > apr_pool_t *thread_pool;
> >
> > create_child_pool(&thread_pool, global_root_pool);
>
> now you've got mutexes in global_root_pool. see my performance comment
> above.
I'm not advocating this scenario as a real solution, but merely trying
to illustrate how the child-pool creation routines don't really have to
be in apr_thread_create(). The application using APR threads can do this
before calling apr_thread_create() or it can do it in it's worker_function.
Either way, it is better to leave it totally up to the application, no?
-aaron
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
> We have hit an impass in my mind. Dean and I are saying that having each
> thread have it's own pool is a requirement. Not just for httpd, but for
> anything using pools. Dean, if I am mis-interpreting you, then I am
> sorry, and please correct me.
>
> Aaron, you disagree. you want each app to make that decision on their
> own. I see no way to implement that in a thread-safe manner.
(Just as a side note, I don't remember seeing anything refuting the technical
aspects of such an implementation. I would be interested in one, of course,
if you think it is infeasible.)
> So, I would suggest that we approach this problem in another way. My ONLY
> goal for APR is to make a library that is useful for cross-platform
> development and eases the pain of cross-platform development for
> developers. That was the goal when I created APR, and my original test
> case was the Apache web server.
>
> I believe that the problem is that the threaded code is creating the
> pool, and not advertising it to the thread itself. This is an easy thing
> to fix. I do not agree that every APR app that uses threads should have
> to create their own thread pools. That is wasted effort by every APR app.
>
> I would like to see a conclusion to this thread quickly. So, could people
> please post their desires quickly, and what they want to see from this.
In my mind, there are really only two major points that need addressing:
- Optionally creating a child-pool for each thread. This provides us two
things in my mind:
1) Allowing the application to choose an SMS for this thread-pool (which
could very well be a special SMS created just for this purpose).
2) Allowing an application to refrain from creating a child-pool.
- Like you [Ryan] said, it is a problem that the thread creation routines
right now are not advertising the pool to the thread itself. This to
me can be corrected with a better API. The two things I've brought up
here are:
1) void * worker_fn(void *opaque_data) should have an additional
pool param.
2) apr_thread_exit() is unacceptable. It currently requires a
apr_thread_t structure. I would be happy if either the worker_fn was
extended to pass in the apr_thread_t OR if we could architect a way
for apr_thread_exit() to not need the apr_thread_t struct.
I have a feeling the second point will be more palatable, and as such
I'll volunteer to cook up a patch illustrating these changes.
As for the first point, which is the reason this thread has been going on
so long, I'm willing to compromise on the idea of forcefully creating
a childpool always, if it necessary, merely because I see it only as
overhead for some applications, but not a functional defect.
-aaron
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
> > I totally agree, but only as a solution in httpd.
>
> no, everywhere.
>
> > I also believe that we should provide this [application-specific requirement]
> > outside of the basic thread support in APR.
> >
> > Please allow me to use pseudocode:
> >
> > void * worker_function(void * opaque_application_data) {
> >
> > apr_pool_t *thread_pool;
> >
> > create_child_pool(&thread_pool, global_root_pool);
>
> now you've got mutexes in global_root_pool. see my performance comment
> above.
I'm not advocating this scenario as a real solution, but merely trying
to illustrate how the child-pool creation routines don't really have to
be in apr_thread_create(). The application using APR threads can do this
before calling apr_thread_create() or it can do it in it's worker_function.
Either way, it is better to leave it totally up to the application, no?
-aaron
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Tue, Jul 17, 2001 at 04:49:04PM -0700, dean gaudet wrote:
> > Please allow me to use pseudocode:
> >
> > void * worker_function(void * opaque_application_data) {
> >
> > apr_pool_t *thread_pool;
> >
> > create_child_pool(&thread_pool, global_root_pool);
>
> now you've got mutexes in global_root_pool. see my performance comment
> above.
IMHO, the global_root_pool should be the "standard" SMS - which is
just a SMS that calls malloc/free - no locking there. It actually
has no free-list/allocation-list. This strategy may or may not
work - depending if you'd ever want to cleanup the global_root_pool
(in Aaron's example). Which I don't think we'd want to do anyway.
-- justin
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Tue, Jul 17, 2001 at 10:59:03AM -0700, Brian Pane wrote:
> the unsynchronized approach can crash due to a race condition.) Thus my
> argument is that this scenario (forking from a threaded MPM and then trying
> to clean up the other threads' resources in the child process) isn't a
> valid use case for the design of the pools.
+1. -- justin
Re: Terminating threads in a process,
WAS: RE: [PATCH] Problems with MPM threaded
Posted by Brian Pane <bp...@pacbell.net>.
Aaron Bannert wrote:
>On Tue, Jul 17, 2001 at 10:17:01AM -0700, Brian Pane wrote:
>
>>Aaron Bannert wrote:
>>
>>>>I'm not sure that the alternative is workable, either.
>>>>
>>>>At the time of the fork, when the child process gets a snapshot of
>>>>the parent's memory, it's possible that some thread other than the one
>>>>invoking fork could be halfway through registering a new resource (e.g.,
>>>>file descriptor) in its pool. There's no guarantee that it's safe to
>>>>attempt a cleanup of that other thread's pool in the child process;
>>>>if the fork has caught the data structures in an intermediate state,
>>>>attempting to destroy that pool might yield a segv.
>>>>
>>>Correct me if I'm wrong, but that would be a property of a buggy MPM.
>>>If the MPM chooses to mix non-synchronized fork()s and thread invocation,
>>>than that's what it gets.
>>>
>>Synchronizing the forks with pool resource registration isn't a scalable
>>design; it requires locking a process-wide lock every time you want to
>>register a resource in a per-thread pool. It would be a huge mistake
>>to slow down the routine operation of the httpd in order to optimize for
>>fork.
>>
>
>I wasn't suggesting that, I was suggesting that it's a bad idea to start
>fork()ing off processes from an MPM that's also doing threads and you
>want that fork()ed of process to later do cleanups.
>
I agree.
>My point is it's solely an MPM architecture decision. Are you saying that
>this is a typical scenario?
>
I think it's an atypical scenario, and one that can't be implemented very
well anyway. (The synchronized approach has a performance problem, and
the unsynchronized approach can crash due to a race condition.) Thus my
argument is that this scenario (forking from a threaded MPM and then trying
to clean up the other threads' resources in the child process) isn't a
valid use case for the design of the pools.
--Brian
Re: Terminating threads in a process,
WAS: RE: [PATCH] Problems with MPM threaded
Posted by Brian Pane <bp...@pacbell.net>.
Aaron Bannert wrote:
>On Tue, Jul 17, 2001 at 10:17:01AM -0700, Brian Pane wrote:
>
>>Aaron Bannert wrote:
>>
>>>>I'm not sure that the alternative is workable, either.
>>>>
>>>>At the time of the fork, when the child process gets a snapshot of
>>>>the parent's memory, it's possible that some thread other than the one
>>>>invoking fork could be halfway through registering a new resource (e.g.,
>>>>file descriptor) in its pool. There's no guarantee that it's safe to
>>>>attempt a cleanup of that other thread's pool in the child process;
>>>>if the fork has caught the data structures in an intermediate state,
>>>>attempting to destroy that pool might yield a segv.
>>>>
>>>Correct me if I'm wrong, but that would be a property of a buggy MPM.
>>>If the MPM chooses to mix non-synchronized fork()s and thread invocation,
>>>than that's what it gets.
>>>
>>Synchronizing the forks with pool resource registration isn't a scalable
>>design; it requires locking a process-wide lock every time you want to
>>register a resource in a per-thread pool. It would be a huge mistake
>>to slow down the routine operation of the httpd in order to optimize for
>>fork.
>>
>
>I wasn't suggesting that, I was suggesting that it's a bad idea to start
>fork()ing off processes from an MPM that's also doing threads and you
>want that fork()ed of process to later do cleanups.
>
I agree.
>My point is it's solely an MPM architecture decision. Are you saying that
>this is a typical scenario?
>
I think it's an atypical scenario, and one that can't be implemented very
well anyway. (The synchronized approach has a performance problem, and
the unsynchronized approach can crash due to a race condition.) Thus my
argument is that this scenario (forking from a threaded MPM and then trying
to clean up the other threads' resources in the child process) isn't a
valid use case for the design of the pools.
--Brian
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Tue, Jul 17, 2001 at 10:17:01AM -0700, Brian Pane wrote:
> Aaron Bannert wrote:
>
> >>I'm not sure that the alternative is workable, either.
> >>
> >>At the time of the fork, when the child process gets a snapshot of
> >>the parent's memory, it's possible that some thread other than the one
> >>invoking fork could be halfway through registering a new resource (e.g.,
> >>file descriptor) in its pool. There's no guarantee that it's safe to
> >>attempt a cleanup of that other thread's pool in the child process;
> >>if the fork has caught the data structures in an intermediate state,
> >>attempting to destroy that pool might yield a segv.
> >>
> >
> >Correct me if I'm wrong, but that would be a property of a buggy MPM.
> >If the MPM chooses to mix non-synchronized fork()s and thread invocation,
> >than that's what it gets.
> >
> Synchronizing the forks with pool resource registration isn't a scalable
> design; it requires locking a process-wide lock every time you want to
> register a resource in a per-thread pool. It would be a huge mistake
> to slow down the routine operation of the httpd in order to optimize for
> fork.
I wasn't suggesting that, I was suggesting that it's a bad idea to start
fork()ing off processes from an MPM that's also doing threads and you
want that fork()ed of process to later do cleanups.
My point is it's solely an MPM architecture decision. Are you saying that
this is a typical scenario?
-aaron
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Tue, Jul 17, 2001 at 10:17:01AM -0700, Brian Pane wrote:
> Aaron Bannert wrote:
>
> >>I'm not sure that the alternative is workable, either.
> >>
> >>At the time of the fork, when the child process gets a snapshot of
> >>the parent's memory, it's possible that some thread other than the one
> >>invoking fork could be halfway through registering a new resource (e.g.,
> >>file descriptor) in its pool. There's no guarantee that it's safe to
> >>attempt a cleanup of that other thread's pool in the child process;
> >>if the fork has caught the data structures in an intermediate state,
> >>attempting to destroy that pool might yield a segv.
> >>
> >
> >Correct me if I'm wrong, but that would be a property of a buggy MPM.
> >If the MPM chooses to mix non-synchronized fork()s and thread invocation,
> >than that's what it gets.
> >
> Synchronizing the forks with pool resource registration isn't a scalable
> design; it requires locking a process-wide lock every time you want to
> register a resource in a per-thread pool. It would be a huge mistake
> to slow down the routine operation of the httpd in order to optimize for
> fork.
I wasn't suggesting that, I was suggesting that it's a bad idea to start
fork()ing off processes from an MPM that's also doing threads and you
want that fork()ed of process to later do cleanups.
My point is it's solely an MPM architecture decision. Are you saying that
this is a typical scenario?
-aaron
Re: Terminating threads in a process,
WAS: RE: [PATCH] Problems with MPM threaded
Posted by Brian Pane <bp...@pacbell.net>.
Aaron Bannert wrote:
>>I'm not sure that the alternative is workable, either.
>>
>>At the time of the fork, when the child process gets a snapshot of
>>the parent's memory, it's possible that some thread other than the one
>>invoking fork could be halfway through registering a new resource (e.g.,
>>file descriptor) in its pool. There's no guarantee that it's safe to
>>attempt a cleanup of that other thread's pool in the child process;
>>if the fork has caught the data structures in an intermediate state,
>>attempting to destroy that pool might yield a segv.
>>
>
>Correct me if I'm wrong, but that would be a property of a buggy MPM.
>If the MPM chooses to mix non-synchronized fork()s and thread invocation,
>than that's what it gets.
>
Synchronizing the forks with pool resource registration isn't a scalable
design; it requires locking a process-wide lock every time you want to
register a resource in a per-thread pool. It would be a huge mistake
to slow down the routine operation of the httpd in order to optimize for
fork.
--Brian
Re: Terminating threads in a process,
WAS: RE: [PATCH] Problems with MPM threaded
Posted by Brian Pane <bp...@pacbell.net>.
Aaron Bannert wrote:
>>I'm not sure that the alternative is workable, either.
>>
>>At the time of the fork, when the child process gets a snapshot of
>>the parent's memory, it's possible that some thread other than the one
>>invoking fork could be halfway through registering a new resource (e.g.,
>>file descriptor) in its pool. There's no guarantee that it's safe to
>>attempt a cleanup of that other thread's pool in the child process;
>>if the fork has caught the data structures in an intermediate state,
>>attempting to destroy that pool might yield a segv.
>>
>
>Correct me if I'm wrong, but that would be a property of a buggy MPM.
>If the MPM chooses to mix non-synchronized fork()s and thread invocation,
>than that's what it gets.
>
Synchronizing the forks with pool resource registration isn't a scalable
design; it requires locking a process-wide lock every time you want to
register a resource in a per-thread pool. It would be a huge mistake
to slow down the routine operation of the httpd in order to optimize for
fork.
--Brian
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
> I'm not sure that the alternative is workable, either.
>
> At the time of the fork, when the child process gets a snapshot of
> the parent's memory, it's possible that some thread other than the one
> invoking fork could be halfway through registering a new resource (e.g.,
> file descriptor) in its pool. There's no guarantee that it's safe to
> attempt a cleanup of that other thread's pool in the child process;
> if the fork has caught the data structures in an intermediate state,
> attempting to destroy that pool might yield a segv.
Correct me if I'm wrong, but that would be a property of a buggy MPM.
If the MPM chooses to mix non-synchronized fork()s and thread invocation,
than that's what it gets.
-aaron
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
> I'm not sure that the alternative is workable, either.
>
> At the time of the fork, when the child process gets a snapshot of
> the parent's memory, it's possible that some thread other than the one
> invoking fork could be halfway through registering a new resource (e.g.,
> file descriptor) in its pool. There's no guarantee that it's safe to
> attempt a cleanup of that other thread's pool in the child process;
> if the fork has caught the data structures in an intermediate state,
> attempting to destroy that pool might yield a segv.
Correct me if I'm wrong, but that would be a property of a buggy MPM.
If the MPM chooses to mix non-synchronized fork()s and thread invocation,
than that's what it gets.
-aaron
Re: Terminating threads in a process,
WAS: RE: [PATCH] Problems with MPM threaded
Posted by Brian Pane <bp...@pacbell.net>.
William A. Rowe, Jr. wrote:
>From: <rb...@covalent.net>
>Sent: Tuesday, July 17, 2001 11:13 AM
>
>>I believe that the problem is that the threaded code is creating the
>>pool, and not advertising it to the thread itself. This is an easy thing
>>to fix. I do not agree that every APR app that uses threads should have
>>to create their own thread pools. That is wasted effort by every APR app.
>>
>>I would like to see a conclusion to this thread quickly. So, could people
>>please post their desires quickly, and what they want to see from this.
>>
>
>thread-local pools would be a nice improvement, perhaps as an argument to
>apr_thread_create. They can be flagged such that they avoid locking their own
>own allocations (of course their 'parent' allocator, malloc, or the parent
>free-list management, may still need to lock based on platform and so forth).
>
>But _unless_ they remain rooted to the top level pool, in apr_process_create
>they become orphaned, and their cleanups are never executed. That isn't workable.
>
I'm not sure that the alternative is workable, either.
At the time of the fork, when the child process gets a snapshot of
the parent's memory, it's possible that some thread other than the one
invoking fork could be halfway through registering a new resource (e.g.,
file descriptor) in its pool. There's no guarantee that it's safe to
attempt a cleanup of that other thread's pool in the child process;
if the fork has caught the data structures in an intermediate state,
attempting to destroy that pool might yield a segv.
--Brian
Re: Terminating threads in a process,
WAS: RE: [PATCH] Problems with MPM threaded
Posted by Brian Pane <bp...@pacbell.net>.
William A. Rowe, Jr. wrote:
>From: <rb...@covalent.net>
>Sent: Tuesday, July 17, 2001 11:13 AM
>
>>I believe that the problem is that the threaded code is creating the
>>pool, and not advertising it to the thread itself. This is an easy thing
>>to fix. I do not agree that every APR app that uses threads should have
>>to create their own thread pools. That is wasted effort by every APR app.
>>
>>I would like to see a conclusion to this thread quickly. So, could people
>>please post their desires quickly, and what they want to see from this.
>>
>
>thread-local pools would be a nice improvement, perhaps as an argument to
>apr_thread_create. They can be flagged such that they avoid locking their own
>own allocations (of course their 'parent' allocator, malloc, or the parent
>free-list management, may still need to lock based on platform and so forth).
>
>But _unless_ they remain rooted to the top level pool, in apr_process_create
>they become orphaned, and their cleanups are never executed. That isn't workable.
>
I'm not sure that the alternative is workable, either.
At the time of the fork, when the child process gets a snapshot of
the parent's memory, it's possible that some thread other than the one
invoking fork could be halfway through registering a new resource (e.g.,
file descriptor) in its pool. There's no guarantee that it's safe to
attempt a cleanup of that other thread's pool in the child process;
if the fork has caught the data structures in an intermediate state,
attempting to destroy that pool might yield a segv.
--Brian
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Tue, Jul 17, 2001 at 12:13:08PM -0500, William A. Rowe, Jr. wrote:
> Because, otherwise, all resources (files, sockets, locks, etc) are not closed when
> another thread forks.
>
> There is a difference between a clean thread exit and cleaning up after a thread.
> We can't do a clean thread exit. We can clean up after one, if the resources are
> apr_entities with properly registered cleanups.
But, if the thread exits cleanly by calling apr_thread_exit() or has its
own private SMS (I don't care which, but I think storing the SMS in the
apr_thread_t object is a bit unclean), it'll return all of its
resources with properly registered cleanups when the per-thread SMS is
destroyed as the thread exits cleanly. Again, I'm just not seeing what
the problem is here. -- justin
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
From: "Justin Erenkrantz" <je...@ebuilt.com>
Sent: Tuesday, July 17, 2001 12:00 PM
> On Tue, Jul 17, 2001 at 11:26:55AM -0500, William A. Rowe, Jr. wrote:
> > But _unless_ they remain rooted to the top level pool, in apr_process_create
> > they become orphaned, and their cleanups are never executed. That isn't workable.
>
> I disagree with this. I gave my big long monologue on this on Sunday
> night. In summary, since you can't cleanup the thread itself, why are
> we cleaning up its memory? -- justin
Because, otherwise, all resources (files, sockets, locks, etc) are not closed when
another thread forks.
There is a difference between a clean thread exit and cleaning up after a thread.
We can't do a clean thread exit. We can clean up after one, if the resources are
apr_entities with properly registered cleanups.
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Tue, Jul 17, 2001 at 11:26:55AM -0500, William A. Rowe, Jr. wrote:
> But _unless_ they remain rooted to the top level pool, in apr_process_create
> they become orphaned, and their cleanups are never executed. That isn't workable.
I disagree with this. I gave my big long monologue on this on Sunday
night. In summary, since you can't cleanup the thread itself, why are
we cleaning up its memory? -- justin
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
From: <rb...@covalent.net>
Sent: Tuesday, July 17, 2001 11:13 AM
>
> I believe that the problem is that the threaded code is creating the
> pool, and not advertising it to the thread itself. This is an easy thing
> to fix. I do not agree that every APR app that uses threads should have
> to create their own thread pools. That is wasted effort by every APR app.
>
> I would like to see a conclusion to this thread quickly. So, could people
> please post their desires quickly, and what they want to see from this.
thread-local pools would be a nice improvement, perhaps as an argument to
apr_thread_create. They can be flagged such that they avoid locking their own
own allocations (of course their 'parent' allocator, malloc, or the parent
free-list management, may still need to lock based on platform and so forth).
But _unless_ they remain rooted to the top level pool, in apr_process_create
they become orphaned, and their cleanups are never executed. That isn't workable.
It sounds like (forgive me if I'm stating the obvious) that a pool needs two
kinds of parent sms pools. One is the 'heirarchy' for purposes of setup and
teardown. The other is the 'allocator', where the blocks will be allocated from.
Want to use the conventional 1.3 model? The heirarchial parent is the parent pool,
but the allocatation parent is the single top level pool.
Want to flip to a malloc, that trusts the clib to 'do the right thing'(sm) with
cpu/thread specific free lists? Then we build a very simple malloc/free sms that
can be passed as the allocation parent when the pool is created.
If I've abused the sms concept here, then correct me, but I don't see any issues
remaining beyond semantics and syntax.
Bill
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
From: <rb...@covalent.net>
Sent: Tuesday, July 17, 2001 11:13 AM
>
> I believe that the problem is that the threaded code is creating the
> pool, and not advertising it to the thread itself. This is an easy thing
> to fix. I do not agree that every APR app that uses threads should have
> to create their own thread pools. That is wasted effort by every APR app.
>
> I would like to see a conclusion to this thread quickly. So, could people
> please post their desires quickly, and what they want to see from this.
thread-local pools would be a nice improvement, perhaps as an argument to
apr_thread_create. They can be flagged such that they avoid locking their own
own allocations (of course their 'parent' allocator, malloc, or the parent
free-list management, may still need to lock based on platform and so forth).
But _unless_ they remain rooted to the top level pool, in apr_process_create
they become orphaned, and their cleanups are never executed. That isn't workable.
It sounds like (forgive me if I'm stating the obvious) that a pool needs two
kinds of parent sms pools. One is the 'heirarchy' for purposes of setup and
teardown. The other is the 'allocator', where the blocks will be allocated from.
Want to use the conventional 1.3 model? The heirarchial parent is the parent pool,
but the allocatation parent is the single top level pool.
Want to flip to a malloc, that trusts the clib to 'do the right thing'(sm) with
cpu/thread specific free lists? Then we build a very simple malloc/free sms that
can be passed as the allocation parent when the pool is created.
If I've abused the sms concept here, then correct me, but I don't see any issues
remaining beyond semantics and syntax.
Bill
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with
MPM threaded
Posted by rb...@covalent.net.
On Tue, 17 Jul 2001, Aaron Bannert wrote:
> On Tue, Jul 17, 2001 at 01:29:47AM -0700, dean gaudet wrote:
> > On Sun, 15 Jul 2001, Sander Striker wrote:
> >
> > > Why are we so desperate in opting out the child-pool creation?
> > > I don't really have problems with a child pool for each thread. Actually,
> > > it will make the dynamic locking a lot easier to implement if it stays.
> >
> > all threads MUST have their own private pool root.
> >
> > otherwise you're just throwing scalability away in locks. (which is
> > proved by the claim i saw that ian got better performance by defining
> > ALLOC_USE_MALLOC on an 8-way.)
>
> I totally agree, but only as a solution in httpd.
>
>
> I also believe that we should provide this [application-specific requirement]
> outside of the basic thread support in APR.
>
> Please allow me to use pseudocode:
>
> void * worker_function(void * opaque_application_data) {
>
> apr_pool_t *thread_pool;
>
> create_child_pool(&thread_pool, global_root_pool);
>
> do_thread_stuff();
>
> cleanup_child_pool(thread_pool);
>
> if (apr_thread_exit_makes_sense_to_call_from_inside_worker_function) {
> int rv = APR_SUCCESS;
> apr_thread_exit(&rv);
> }
>
> return NULL;
> }
>
> What I mean by this gibberish is that the pool-handling code can exist
> completely outside of our thread_creation-type functions in APR. Is there
> any reason for APR threads to do more than this (that is, just simple thread
> creation)?
We have hit an impass in my mind. Dean and I are saying that having each
thread have it's own pool is a requirement. Not just for httpd, but for
anything using pools. Dean, if I am mis-interpreting you, then I am
sorry, and please correct me.
Aaron, you disagree. you want each app to make that decision on their
own. I see no way to implement that in a thread-safe manner.
So, I would suggest that we approach this problem in another way. My ONLY
goal for APR is to make a library that is useful for cross-platform
development and eases the pain of cross-platform development for
developers. That was the goal when I created APR, and my original test
case was the Apache web server.
I believe that the problem is that the threaded code is creating the
pool, and not advertising it to the thread itself. This is an easy thing
to fix. I do not agree that every APR app that uses threads should have
to create their own thread pools. That is wasted effort by every APR app.
I would like to see a conclusion to this thread quickly. So, could people
please post their desires quickly, and what they want to see from this.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with
MPM threaded
Posted by dean gaudet <dg...@arctic.org>.
On Tue, 17 Jul 2001, Aaron Bannert wrote:
> On Tue, Jul 17, 2001 at 01:29:47AM -0700, dean gaudet wrote:
> > On Sun, 15 Jul 2001, Sander Striker wrote:
> >
> > > Why are we so desperate in opting out the child-pool creation?
> > > I don't really have problems with a child pool for each thread. Actually,
> > > it will make the dynamic locking a lot easier to implement if it stays.
> >
> > all threads MUST have their own private pool root.
> >
> > otherwise you're just throwing scalability away in locks. (which is
> > proved by the claim i saw that ian got better performance by defining
> > ALLOC_USE_MALLOC on an 8-way.)
>
> I totally agree, but only as a solution in httpd.
no, everywhere.
> I also believe that we should provide this [application-specific requirement]
> outside of the basic thread support in APR.
>
> Please allow me to use pseudocode:
>
> void * worker_function(void * opaque_application_data) {
>
> apr_pool_t *thread_pool;
>
> create_child_pool(&thread_pool, global_root_pool);
now you've got mutexes in global_root_pool. see my performance comment
above.
-dean
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with
MPM threaded
Posted by rb...@covalent.net.
On Tue, 17 Jul 2001, Aaron Bannert wrote:
> On Tue, Jul 17, 2001 at 01:29:47AM -0700, dean gaudet wrote:
> > On Sun, 15 Jul 2001, Sander Striker wrote:
> >
> > > Why are we so desperate in opting out the child-pool creation?
> > > I don't really have problems with a child pool for each thread. Actually,
> > > it will make the dynamic locking a lot easier to implement if it stays.
> >
> > all threads MUST have their own private pool root.
> >
> > otherwise you're just throwing scalability away in locks. (which is
> > proved by the claim i saw that ian got better performance by defining
> > ALLOC_USE_MALLOC on an 8-way.)
>
> I totally agree, but only as a solution in httpd.
>
>
> I also believe that we should provide this [application-specific requirement]
> outside of the basic thread support in APR.
>
> Please allow me to use pseudocode:
>
> void * worker_function(void * opaque_application_data) {
>
> apr_pool_t *thread_pool;
>
> create_child_pool(&thread_pool, global_root_pool);
>
> do_thread_stuff();
>
> cleanup_child_pool(thread_pool);
>
> if (apr_thread_exit_makes_sense_to_call_from_inside_worker_function) {
> int rv = APR_SUCCESS;
> apr_thread_exit(&rv);
> }
>
> return NULL;
> }
>
> What I mean by this gibberish is that the pool-handling code can exist
> completely outside of our thread_creation-type functions in APR. Is there
> any reason for APR threads to do more than this (that is, just simple thread
> creation)?
We have hit an impass in my mind. Dean and I are saying that having each
thread have it's own pool is a requirement. Not just for httpd, but for
anything using pools. Dean, if I am mis-interpreting you, then I am
sorry, and please correct me.
Aaron, you disagree. you want each app to make that decision on their
own. I see no way to implement that in a thread-safe manner.
So, I would suggest that we approach this problem in another way. My ONLY
goal for APR is to make a library that is useful for cross-platform
development and eases the pain of cross-platform development for
developers. That was the goal when I created APR, and my original test
case was the Apache web server.
I believe that the problem is that the threaded code is creating the
pool, and not advertising it to the thread itself. This is an easy thing
to fix. I do not agree that every APR app that uses threads should have
to create their own thread pools. That is wasted effort by every APR app.
I would like to see a conclusion to this thread quickly. So, could people
please post their desires quickly, and what they want to see from this.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with
MPM threaded
Posted by dean gaudet <dg...@arctic.org>.
On Tue, 17 Jul 2001, Aaron Bannert wrote:
> On Tue, Jul 17, 2001 at 01:29:47AM -0700, dean gaudet wrote:
> > On Sun, 15 Jul 2001, Sander Striker wrote:
> >
> > > Why are we so desperate in opting out the child-pool creation?
> > > I don't really have problems with a child pool for each thread. Actually,
> > > it will make the dynamic locking a lot easier to implement if it stays.
> >
> > all threads MUST have their own private pool root.
> >
> > otherwise you're just throwing scalability away in locks. (which is
> > proved by the claim i saw that ian got better performance by defining
> > ALLOC_USE_MALLOC on an 8-way.)
>
> I totally agree, but only as a solution in httpd.
no, everywhere.
> I also believe that we should provide this [application-specific requirement]
> outside of the basic thread support in APR.
>
> Please allow me to use pseudocode:
>
> void * worker_function(void * opaque_application_data) {
>
> apr_pool_t *thread_pool;
>
> create_child_pool(&thread_pool, global_root_pool);
now you've got mutexes in global_root_pool. see my performance comment
above.
-dean
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Tue, Jul 17, 2001 at 01:29:47AM -0700, dean gaudet wrote:
> On Sun, 15 Jul 2001, Sander Striker wrote:
>
> > Why are we so desperate in opting out the child-pool creation?
> > I don't really have problems with a child pool for each thread. Actually,
> > it will make the dynamic locking a lot easier to implement if it stays.
>
> all threads MUST have their own private pool root.
>
> otherwise you're just throwing scalability away in locks. (which is
> proved by the claim i saw that ian got better performance by defining
> ALLOC_USE_MALLOC on an 8-way.)
I totally agree, but only as a solution in httpd.
I also believe that we should provide this [application-specific requirement]
outside of the basic thread support in APR.
Please allow me to use pseudocode:
void * worker_function(void * opaque_application_data) {
apr_pool_t *thread_pool;
create_child_pool(&thread_pool, global_root_pool);
do_thread_stuff();
cleanup_child_pool(thread_pool);
if (apr_thread_exit_makes_sense_to_call_from_inside_worker_function) {
int rv = APR_SUCCESS;
apr_thread_exit(&rv);
}
return NULL;
}
What I mean by this gibberish is that the pool-handling code can exist
completely outside of our thread_creation-type functions in APR. Is there
any reason for APR threads to do more than this (that is, just simple thread
creation)?
-aaron
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Tue, Jul 17, 2001 at 01:29:47AM -0700, dean gaudet wrote:
> On Sun, 15 Jul 2001, Sander Striker wrote:
>
> > Why are we so desperate in opting out the child-pool creation?
> > I don't really have problems with a child pool for each thread. Actually,
> > it will make the dynamic locking a lot easier to implement if it stays.
>
> all threads MUST have their own private pool root.
>
> otherwise you're just throwing scalability away in locks. (which is
> proved by the claim i saw that ian got better performance by defining
> ALLOC_USE_MALLOC on an 8-way.)
I totally agree, but only as a solution in httpd.
I also believe that we should provide this [application-specific requirement]
outside of the basic thread support in APR.
Please allow me to use pseudocode:
void * worker_function(void * opaque_application_data) {
apr_pool_t *thread_pool;
create_child_pool(&thread_pool, global_root_pool);
do_thread_stuff();
cleanup_child_pool(thread_pool);
if (apr_thread_exit_makes_sense_to_call_from_inside_worker_function) {
int rv = APR_SUCCESS;
apr_thread_exit(&rv);
}
return NULL;
}
What I mean by this gibberish is that the pool-handling code can exist
completely outside of our thread_creation-type functions in APR. Is there
any reason for APR threads to do more than this (that is, just simple thread
creation)?
-aaron
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with
MPM threaded
Posted by rb...@covalent.net.
On Sun, 15 Jul 2001, Sander Striker wrote:
> > Fair enough. It's just that in order to opt-out of the child-pool creating
> > process in apr_thread_create, we're going to have to add a parameter
>
> Why are we so desperate in opting out the child-pool creation?
> I don't really have problems with a child pool for each thread. Actually,
> it will make the dynamic locking a lot easier to implement if it stays.
The reality is that it should stay. The problem we have right now, is
that httpd creates a thread-pool, and then APR creates one. So, if httpd
didn't create one, and just used the one APR created automatically, this
problem would go away completely.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: request_rec question
Posted by rb...@covalent.net.
On Tue, 17 Jul 2001, Ian Holsman wrote:
> rbb@covalent.net wrote:
>
> >On Tue, 17 Jul 2001, Brian Pane wrote:
> >
> >>Is there anything that actually uses the
> >>headers_out or err_headers_out fields in
> >>the request_rec that's created for a subrequest?
> >>Anything written to these fields appears to
> >>be discarded upon completion of the subrequest.
> >>We could save a couple of apr_table_make calls
> >>per subrequest if the httpd didn't have to
> >>create these two tables.
> >>
> >
> >You can't save those calls, because that would require every module in the
> >world to check to see if the table had been made before it tried to add
> >something to it. Remember that a sub-request goes through the same basic
> >logic as a real request.
> >
>
> so should we fix the code so that the headers_out/err_headers_out of a
> subrequest gets merged into
> the parents request?
Nope. While a sub-request goes through the same logic, the headers from a
sub-request do not always apply equally to the main-line request. I can
easily an instance where an output header on a sub-request, say Location,
doesn't apply to the main request.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: request_rec question
Posted by Ian Holsman <ia...@cnet.com>.
rbb@covalent.net wrote:
>On Tue, 17 Jul 2001, Brian Pane wrote:
>
>>Is there anything that actually uses the
>>headers_out or err_headers_out fields in
>>the request_rec that's created for a subrequest?
>>Anything written to these fields appears to
>>be discarded upon completion of the subrequest.
>>We could save a couple of apr_table_make calls
>>per subrequest if the httpd didn't have to
>>create these two tables.
>>
>
>You can't save those calls, because that would require every module in the
>world to check to see if the table had been made before it tried to add
>something to it. Remember that a sub-request goes through the same basic
>logic as a real request.
>
so should we fix the code so that the headers_out/err_headers_out of a
subrequest gets merged into
the parents request?
>
>
>Ryan
>
>_____________________________________________________________________________
>Ryan Bloom rbb@apache.org
>Covalent Technologies rbb@covalent.net
>-----------------------------------------------------------------------------
>
Re: request_rec question
Posted by rb...@covalent.net.
On Tue, 17 Jul 2001, Brian Pane wrote:
> Is there anything that actually uses the
> headers_out or err_headers_out fields in
> the request_rec that's created for a subrequest?
> Anything written to these fields appears to
> be discarded upon completion of the subrequest.
> We could save a couple of apr_table_make calls
> per subrequest if the httpd didn't have to
> create these two tables.
You can't save those calls, because that would require every module in the
world to check to see if the table had been made before it tried to add
something to it. Remember that a sub-request goes through the same basic
logic as a real request.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: request_rec question
Posted by Brian Pane <bp...@pacbell.net>.
dean gaudet wrote:
>see modules/mappers/mod_negotiation.c search for "fast redirect".
>
>no comment on how clean this is :)
>
>it's something that should be moved to a core routine.
>
As a side-effect of the table overlays that this code does, it
code can 'promote' anything that another module has stored in
the subrequest's headers_out into the parent request's headers_out.
Is that a bug or a feature?
--Brian
Re: request_rec question
Posted by dean gaudet <dg...@arctic.org>.
see modules/mappers/mod_negotiation.c search for "fast redirect".
no comment on how clean this is :)
it's something that should be moved to a core routine.
-dean
On Tue, 17 Jul 2001, Brian Pane wrote:
> Is there anything that actually uses the
> headers_out or err_headers_out fields in
> the request_rec that's created for a subrequest?
> Anything written to these fields appears to
> be discarded upon completion of the subrequest.
> We could save a couple of apr_table_make calls
> per subrequest if the httpd didn't have to
> create these two tables.
>
> --Brian
>
>
>
>
request_rec question
Posted by Brian Pane <bp...@pacbell.net>.
Is there anything that actually uses the
headers_out or err_headers_out fields in
the request_rec that's created for a subrequest?
Anything written to these fields appears to
be discarded upon completion of the subrequest.
We could save a couple of apr_table_make calls
per subrequest if the httpd didn't have to
create these two tables.
--Brian
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with
MPM threaded
Posted by dean gaudet <dg...@arctic.org>.
On Sun, 15 Jul 2001, Sander Striker wrote:
> Why are we so desperate in opting out the child-pool creation?
> I don't really have problems with a child pool for each thread. Actually,
> it will make the dynamic locking a lot easier to implement if it stays.
all threads MUST have their own private pool root.
otherwise you're just throwing scalability away in locks. (which is
proved by the claim i saw that ian got better performance by defining
ALLOC_USE_MALLOC on an 8-way.)
-dean
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with
MPM threaded
Posted by dean gaudet <dg...@arctic.org>.
On Sun, 15 Jul 2001, Sander Striker wrote:
> Why are we so desperate in opting out the child-pool creation?
> I don't really have problems with a child pool for each thread. Actually,
> it will make the dynamic locking a lot easier to implement if it stays.
all threads MUST have their own private pool root.
otherwise you're just throwing scalability away in locks. (which is
proved by the claim i saw that ian got better performance by defining
ALLOC_USE_MALLOC on an 8-way.)
-dean
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with
MPM threaded
Posted by rb...@covalent.net.
On Sun, 15 Jul 2001, Sander Striker wrote:
> > Fair enough. It's just that in order to opt-out of the child-pool creating
> > process in apr_thread_create, we're going to have to add a parameter
>
> Why are we so desperate in opting out the child-pool creation?
> I don't really have problems with a child pool for each thread. Actually,
> it will make the dynamic locking a lot easier to implement if it stays.
The reality is that it should stay. The problem we have right now, is
that httpd creates a thread-pool, and then APR creates one. So, if httpd
didn't create one, and just used the one APR created automatically, this
problem would go away completely.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Sander Striker <st...@apache.org>.
> Fair enough. It's just that in order to opt-out of the child-pool creating
> process in apr_thread_create, we're going to have to add a parameter
Why are we so desperate in opting out the child-pool creation?
I don't really have problems with a child pool for each thread. Actually,
it will make the dynamic locking a lot easier to implement if it stays.
> to the call and that means changing programs that currently using it,
> like httpd.
>
> Would it be nicer to just add new parallel apr_smsthread_create
> [name is not
> important] functions that do this and allow for the opting-out also? I can
> provide a patch for this.
Hmmm, the thought of a double interface doesn't bring out happy feelings
with me I'm afraid.
Sander
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Sander Striker <st...@apache.org>.
> Fair enough. It's just that in order to opt-out of the child-pool creating
> process in apr_thread_create, we're going to have to add a parameter
Why are we so desperate in opting out the child-pool creation?
I don't really have problems with a child pool for each thread. Actually,
it will make the dynamic locking a lot easier to implement if it stays.
> to the call and that means changing programs that currently using it,
> like httpd.
>
> Would it be nicer to just add new parallel apr_smsthread_create
> [name is not
> important] functions that do this and allow for the opting-out also? I can
> provide a patch for this.
Hmmm, the thought of a double interface doesn't bring out happy feelings
with me I'm afraid.
Sander
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Sun, Jul 15, 2001 at 06:49:51PM +0200, Luke Kenneth Casson Leighton wrote:
> On Sat, Jul 14, 2001 at 12:40:06PM -0700, Aaron Bannert wrote:
>
> > APR threads, when created, would now take an additional parameter that
> > is the mechanism (an sms implementation) by which it should create child
> > pools. As it is now, the "pool" that is passed in to apr_thread_create()
> > serves as the "parent" pool for the new thread-specific sms. If this
> > parameter were null, the apr_thread_create() function would not create
> > a sub-pool or register cleanup routines (which satisfies my requirements).
>
> if at all possible, the behaviour should match as closely as possible
> the existing situation, when this parameter is NULL, even if the
> current situation has bugs / is not very nice.
>
> this is mainly so that people do not object to having the behaviour
> of existing code disrupted.
Fair enough. It's just that in order to opt-out of the child-pool creating
process in apr_thread_create, we're going to have to add a parameter
to the call and that means changing programs that currently using it,
like httpd.
Would it be nicer to just add new parallel apr_smsthread_create [name is not
important] functions that do this and allow for the opting-out also? I can
provide a patch for this.
-aaron
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Sun, Jul 15, 2001 at 06:49:51PM +0200, Luke Kenneth Casson Leighton wrote:
> On Sat, Jul 14, 2001 at 12:40:06PM -0700, Aaron Bannert wrote:
>
> > APR threads, when created, would now take an additional parameter that
> > is the mechanism (an sms implementation) by which it should create child
> > pools. As it is now, the "pool" that is passed in to apr_thread_create()
> > serves as the "parent" pool for the new thread-specific sms. If this
> > parameter were null, the apr_thread_create() function would not create
> > a sub-pool or register cleanup routines (which satisfies my requirements).
>
> if at all possible, the behaviour should match as closely as possible
> the existing situation, when this parameter is NULL, even if the
> current situation has bugs / is not very nice.
>
> this is mainly so that people do not object to having the behaviour
> of existing code disrupted.
Fair enough. It's just that in order to opt-out of the child-pool creating
process in apr_thread_create, we're going to have to add a parameter
to the call and that means changing programs that currently using it,
like httpd.
Would it be nicer to just add new parallel apr_smsthread_create [name is not
important] functions that do this and allow for the opting-out also? I can
provide a patch for this.
-aaron
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Luke Kenneth Casson Leighton <lk...@samba-tng.org>.
On Sat, Jul 14, 2001 at 12:40:06PM -0700, Aaron Bannert wrote:
> APR threads, when created, would now take an additional parameter that
> is the mechanism (an sms implementation) by which it should create child
> pools. As it is now, the "pool" that is passed in to apr_thread_create()
> serves as the "parent" pool for the new thread-specific sms. If this
> parameter were null, the apr_thread_create() function would not create
> a sub-pool or register cleanup routines (which satisfies my requirements).
if at all possible, the behaviour should match as closely as possible
the existing situation, when this parameter is NULL, even if the
current situation has bugs / is not very nice.
this is mainly so that people do not object to having the behaviour
of existing code disrupted.
luke
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Luke Kenneth Casson Leighton <lk...@samba-tng.org>.
On Sat, Jul 14, 2001 at 12:40:06PM -0700, Aaron Bannert wrote:
> APR threads, when created, would now take an additional parameter that
> is the mechanism (an sms implementation) by which it should create child
> pools. As it is now, the "pool" that is passed in to apr_thread_create()
> serves as the "parent" pool for the new thread-specific sms. If this
> parameter were null, the apr_thread_create() function would not create
> a sub-pool or register cleanup routines (which satisfies my requirements).
if at all possible, the behaviour should match as closely as possible
the existing situation, when this parameter is NULL, even if the
current situation has bugs / is not very nice.
this is mainly so that people do not object to having the behaviour
of existing code disrupted.
luke
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM
threaded
Posted by dean gaudet <dg...@arctic.org>.
On Sat, 14 Jul 2001, Aaron Bannert wrote:
> On Sat, Jul 14, 2001 at 12:10:30PM -0700, rbb@covalent.net wrote:
> > On Sat, 14 Jul 2001, Sander Striker wrote:
> > > The way I see it, each process has a single pool instance as the parent
> > > for all the threads. Resetting or destroying that pool should effectively
> > > kill all threads. What am I missing?
> >
> > Nothing.
>
> Good, I think we're all in agreement here.
no, this thread is long and i haven't read it, but i'm not at all in
agreement. in fact i'll veto outright any change requiring async
notification.
i furthermore veto any change requiring more mutexes in pools: if you
guys weren't paying attention the last two weeks when this performance
problem was pointed out then i suggest going back and re-reading many of
the threads i contributed to.
maybe i'm just missing the point, i'll go try to read this thread now.
-dean
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM
threaded
Posted by dean gaudet <dg...@arctic.org>.
On Sat, 14 Jul 2001, Aaron Bannert wrote:
> On Sat, Jul 14, 2001 at 12:10:30PM -0700, rbb@covalent.net wrote:
> > On Sat, 14 Jul 2001, Sander Striker wrote:
> > > The way I see it, each process has a single pool instance as the parent
> > > for all the threads. Resetting or destroying that pool should effectively
> > > kill all threads. What am I missing?
> >
> > Nothing.
>
> Good, I think we're all in agreement here.
no, this thread is long and i haven't read it, but i'm not at all in
agreement. in fact i'll veto outright any change requiring async
notification.
i furthermore veto any change requiring more mutexes in pools: if you
guys weren't paying attention the last two weeks when this performance
problem was pointed out then i suggest going back and re-reading many of
the threads i contributed to.
maybe i'm just missing the point, i'll go try to read this thread now.
-dean
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Sat, Jul 14, 2001 at 12:10:30PM -0700, rbb@covalent.net wrote:
> On Sat, 14 Jul 2001, Sander Striker wrote:
> > The way I see it, each process has a single pool instance as the parent
> > for all the threads. Resetting or destroying that pool should effectively
> > kill all threads. What am I missing?
>
> Nothing.
Good, I think we're all in agreement here.
Now let's move on to a discussion of how it is currently implemented in
httpd and APR. As an application developer, I'd like to see APR's threads
more generally applicable (ie. allowing various sms implementations
for each thread's pool, but not necessarily forcing all threads to have
a child pool).
As a [wannabe] contributor to Apache, I'd like to see httpd work. So I
have a proposal:
APR threads, when created, would now take an additional parameter that
is the mechanism (an sms implementation) by which it should create child
pools. As it is now, the "pool" that is passed in to apr_thread_create()
serves as the "parent" pool for the new thread-specific sms. If this
parameter were null, the apr_thread_create() function would not create
a sub-pool or register cleanup routines (which satisfies my requirements).
Alternatively, the subpool can be already passed into apr_thread_create(),
so it doesn't need to know anything about sms, and it leaves it up
to the parent to take care of cleaning up the thread when the pool
falls out of scope (which is what I was originally proposing).
-aaron
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Sat, Jul 14, 2001 at 12:10:30PM -0700, rbb@covalent.net wrote:
> On Sat, 14 Jul 2001, Sander Striker wrote:
> > The way I see it, each process has a single pool instance as the parent
> > for all the threads. Resetting or destroying that pool should effectively
> > kill all threads. What am I missing?
>
> Nothing.
Good, I think we're all in agreement here.
Now let's move on to a discussion of how it is currently implemented in
httpd and APR. As an application developer, I'd like to see APR's threads
more generally applicable (ie. allowing various sms implementations
for each thread's pool, but not necessarily forcing all threads to have
a child pool).
As a [wannabe] contributor to Apache, I'd like to see httpd work. So I
have a proposal:
APR threads, when created, would now take an additional parameter that
is the mechanism (an sms implementation) by which it should create child
pools. As it is now, the "pool" that is passed in to apr_thread_create()
serves as the "parent" pool for the new thread-specific sms. If this
parameter were null, the apr_thread_create() function would not create
a sub-pool or register cleanup routines (which satisfies my requirements).
Alternatively, the subpool can be already passed into apr_thread_create(),
so it doesn't need to know anything about sms, and it leaves it up
to the parent to take care of cleaning up the thread when the pool
falls out of scope (which is what I was originally proposing).
-aaron
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM
threaded
Posted by rb...@covalent.net.
On Sat, 14 Jul 2001, Sander Striker wrote:
> >> By having the possibility of having other children processes, you now
> >> need a mechanism to kill all threads in the same process efficiently.
> >> You'd need to kick them out of the accept mutex, but I'm not seeing
> >> how that's going to happen quickly. -- justin
> >
> > You need to be able to kill all threads in a process efficiently
> > regardless. We will always need to be able to re-spawn processes that
> > die, and we will always have the concept of MaxRequestsPerChild. If a
> > process is going to die and be replaced, then we need to kill all the
> > threads in that process as quickly as possible. Only allowing one
> > process at a time doesn't remove that requirement.
>
> The way I see it, each process has a single pool instance as the parent
> for all the threads. Resetting or destroying that pool should effectively
> kill all threads. What am I missing?
Nothing.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Tue, Jul 17, 2001 at 06:28:37PM -0500, William A. Rowe, Jr. wrote:
> Yes, yes, yes. Can we please split the concept of a heirarchial parent (the
> 'creator' thread's or process pool, in this case) from the allocation parent
> (the actual give me memory for my pool from ... here!) Then we have an
> "OS Knows Best" malloc/free mpm for threading, just as you suggest.
>
> This solves your thread-specific requirements and our scoping issues, along
> with fixing the 'walk the chain of pools for a block' problem, both at once.
Okay, I will try and commit something into the SMS code that does
this. If I understand you, there will be one parent which will handle
allocation and another that will handle cleanups. This should handle
your concerns about a thread that didn't exit properly and my concern
about having a unnecessary locking in the allocation code. Deal?
I'll make it default to having both parents be identical, and the
apr_thread_create call can set it to be null (or a std. SMS).
The only problem is the SMS code is chock-full of subtle race
conditions right now. We *really* need more eyes on this code.
-- justin
Re: Seperating cleanups from memory allocations, WAS: RE: Terminating threads in a process
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Tue, Jul 17, 2001 at 08:33:09PM -0700, dean gaudet wrote:
[snip]
> p.s. maybe this example will help highlight why apr_lock_t is far too
> generic and there should be a more lightweight thread only mutex that
> doesn't require lots of extra memory allocation?
Oh don't get me started on that again... ;)
I'd LOVE to get some really lightweight (intraprocess) mutexes into APR...
Something akin to true POSIX mutexes would be fantastic.
-aaron
Re: Seperating cleanups from memory allocations, WAS: RE: Terminating threads in a process
Posted by Aaron Bannert <aa...@ebuilt.com>.
On Tue, Jul 17, 2001 at 08:33:09PM -0700, dean gaudet wrote:
[snip]
> p.s. maybe this example will help highlight why apr_lock_t is far too
> generic and there should be a more lightweight thread only mutex that
> doesn't require lots of extra memory allocation?
Oh don't get me started on that again... ;)
I'd LOVE to get some really lightweight (intraprocess) mutexes into APR...
Something akin to true POSIX mutexes would be fantastic.
-aaron
Re: Seperating cleanups from memory allocations, WAS: RE: Terminating
threads in a process
Posted by dean gaudet <de...@arctic.org>.
On Wed, 18 Jul 2001, Sander Striker wrote:
> Wouldn't it be wiser to implement cleanup management as a
> seperate entity, instead of piggy backing it on allocators?
i'm not sure... 'cause the only thing i think which is missing is
something like this:
typedef struct {
int die_soon_please;
int ref_count;
apr_lock_t *lock;
} thread_cleanup_t;
die_soon_please is initialised to 0
ref_count to 2
lock is initialised and released
to create a thread that you want cleanup for you create one of these
thread_cleanup_ts (malloc'd). then you register a cleanup in
the "parent thread"'s pool, such as the following:
apr_status_t thread_cleanup(void *_tc)
{
thread_cleanup_t *tc = _tc;
apr_lock_acquire(tc->lock);
if (--tc->ref_count) {
/* thread is still around, tell it to die asap */
tc->die_soon_please = 1;
apr_lock_release(tc->lock);
}
else {
/* thread has already exited, it took care of
* cleaning up its pool on the way out
*/
apr_lock_destroy(tc->lock);
free(tc);
}
return 0;
}
and then when the "child" thread wants to exit, it should do:
apr_pool_destroy(my_root_pool);
apr_lock_acquire(tc->lock);
if (--tc->ref_count == 0) {
/* parent asked us to go away, so we get to free up the
* structure
*/
free(tc->lock);
}
else {
apr_lock_release(tc->lock);
}
and at any time the child cares to ask if the parent told it to die
it can just test tc->die_soon_please.
this basically just moves the mutexes from the common path to the less
common path of parent/child relationship.
-dean
p.s. maybe this example will help highlight why apr_lock_t is far too
generic and there should be a more lightweight thread only mutex that
doesn't require lots of extra memory allocation?
Re: Seperating cleanups from memory allocations, WAS: RE: Terminating
threads in a process
Posted by dean gaudet <de...@arctic.org>.
On Wed, 18 Jul 2001, Sander Striker wrote:
> Wouldn't it be wiser to implement cleanup management as a
> seperate entity, instead of piggy backing it on allocators?
i'm not sure... 'cause the only thing i think which is missing is
something like this:
typedef struct {
int die_soon_please;
int ref_count;
apr_lock_t *lock;
} thread_cleanup_t;
die_soon_please is initialised to 0
ref_count to 2
lock is initialised and released
to create a thread that you want cleanup for you create one of these
thread_cleanup_ts (malloc'd). then you register a cleanup in
the "parent thread"'s pool, such as the following:
apr_status_t thread_cleanup(void *_tc)
{
thread_cleanup_t *tc = _tc;
apr_lock_acquire(tc->lock);
if (--tc->ref_count) {
/* thread is still around, tell it to die asap */
tc->die_soon_please = 1;
apr_lock_release(tc->lock);
}
else {
/* thread has already exited, it took care of
* cleaning up its pool on the way out
*/
apr_lock_destroy(tc->lock);
free(tc);
}
return 0;
}
and then when the "child" thread wants to exit, it should do:
apr_pool_destroy(my_root_pool);
apr_lock_acquire(tc->lock);
if (--tc->ref_count == 0) {
/* parent asked us to go away, so we get to free up the
* structure
*/
free(tc->lock);
}
else {
apr_lock_release(tc->lock);
}
and at any time the child cares to ask if the parent told it to die
it can just test tc->die_soon_please.
this basically just moves the mutexes from the common path to the less
common path of parent/child relationship.
-dean
p.s. maybe this example will help highlight why apr_lock_t is far too
generic and there should be a more lightweight thread only mutex that
doesn't require lots of extra memory allocation?
Seperating cleanups from memory allocations, WAS: RE: Terminating threads in a process
Posted by Sander Striker <st...@apache.org>.
[...]
>> Are you saying you want the thread function to have access to both a
>> "scope" pool as well as an "allocator" pool, in case they are different?
>
> I've officially graduated to the rbb (insert a decent name here) club :-)
>
> Thank you, yes, scope pool defines teardowns such as cleanups,
> while the allocator
> pool addresses our performance concerns :)
>
> An obvious test is that 'allocator' is unique (e.g. thread
> private), a parent
> (at any depth) of the 'scope' pool, or the 'scope' pool itself.
Well, looking at this it seems that the cleanups need to be
hierarchial (almost) all the time. The allocators seem to be
needing to break the hierarchy for performance reasons.
Wouldn't it be wiser to implement cleanup management as a
seperate entity, instead of piggy backing it on allocators?
Sander
Seperating cleanups from memory allocations, WAS: RE: Terminating threads in a process
Posted by Sander Striker <st...@apache.org>.
[...]
>> Are you saying you want the thread function to have access to both a
>> "scope" pool as well as an "allocator" pool, in case they are different?
>
> I've officially graduated to the rbb (insert a decent name here) club :-)
>
> Thank you, yes, scope pool defines teardowns such as cleanups,
> while the allocator
> pool addresses our performance concerns :)
>
> An obvious test is that 'allocator' is unique (e.g. thread
> private), a parent
> (at any depth) of the 'scope' pool, or the 'scope' pool itself.
Well, looking at this it seems that the cleanups need to be
hierarchial (almost) all the time. The allocators seem to be
needing to break the hierarchy for performance reasons.
Wouldn't it be wiser to implement cleanup management as a
seperate entity, instead of piggy backing it on allocators?
Sander
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
From: "Aaron Bannert" <aa...@ebuilt.com>
Sent: Tuesday, July 17, 2001 6:41 PM
> [snip]
> > > that would be registered in the "parent" thread's pool -- and would only
> > > be invoked by the "parent" thread.
> > >
> > > pools let you do this, you don't need the mutexes for it, you just have to
> > > be explicit about parallelism. (combine that with a root pool per thread
> > > and then we can remove alloc_mutex and free lists and push the real gnarly
> > > problems into the libc malloc where it's probably best solved.)
> >
> > Yes, yes, yes. Can we please split the concept of a heirarchial parent (the
> > 'creator' thread's or process pool, in this case) from the allocation parent
> > (the actual give me memory for my pool from ... here!) Then we have an
> > "OS Knows Best" malloc/free mpm for threading, just as you suggest.
> >
> > This solves your thread-specific requirements and our scoping issues, along
> > with fixing the 'walk the chain of pools for a block' problem, both at once.
>
> It's probably just me, but I'm having trouble parsing this (I think I'm
> getting a cold :( ).
>
> Are you saying you want the thread function to have access to both a
> "scope" pool as well as an "allocator" pool, in case they are different?
I've officially graduated to the rbb (insert a decent name here) club :-)
Thank you, yes, scope pool defines teardowns such as cleanups, while the allocator
pool addresses our performance concerns :)
An obvious test is that 'allocator' is unique (e.g. thread private), a parent
(at any depth) of the 'scope' pool, or the 'scope' pool itself.
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
From: "Aaron Bannert" <aa...@ebuilt.com>
Sent: Tuesday, July 17, 2001 6:41 PM
> [snip]
> > > that would be registered in the "parent" thread's pool -- and would only
> > > be invoked by the "parent" thread.
> > >
> > > pools let you do this, you don't need the mutexes for it, you just have to
> > > be explicit about parallelism. (combine that with a root pool per thread
> > > and then we can remove alloc_mutex and free lists and push the real gnarly
> > > problems into the libc malloc where it's probably best solved.)
> >
> > Yes, yes, yes. Can we please split the concept of a heirarchial parent (the
> > 'creator' thread's or process pool, in this case) from the allocation parent
> > (the actual give me memory for my pool from ... here!) Then we have an
> > "OS Knows Best" malloc/free mpm for threading, just as you suggest.
> >
> > This solves your thread-specific requirements and our scoping issues, along
> > with fixing the 'walk the chain of pools for a block' problem, both at once.
>
> It's probably just me, but I'm having trouble parsing this (I think I'm
> getting a cold :( ).
>
> Are you saying you want the thread function to have access to both a
> "scope" pool as well as an "allocator" pool, in case they are different?
I've officially graduated to the rbb (insert a decent name here) club :-)
Thank you, yes, scope pool defines teardowns such as cleanups, while the allocator
pool addresses our performance concerns :)
An obvious test is that 'allocator' is unique (e.g. thread private), a parent
(at any depth) of the 'scope' pool, or the 'scope' pool itself.
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
[snip]
> > that would be registered in the "parent" thread's pool -- and would only
> > be invoked by the "parent" thread.
> >
> > pools let you do this, you don't need the mutexes for it, you just have to
> > be explicit about parallelism. (combine that with a root pool per thread
> > and then we can remove alloc_mutex and free lists and push the real gnarly
> > problems into the libc malloc where it's probably best solved.)
>
> Yes, yes, yes. Can we please split the concept of a heirarchial parent (the
> 'creator' thread's or process pool, in this case) from the allocation parent
> (the actual give me memory for my pool from ... here!) Then we have an
> "OS Knows Best" malloc/free mpm for threading, just as you suggest.
>
> This solves your thread-specific requirements and our scoping issues, along
> with fixing the 'walk the chain of pools for a block' problem, both at once.
It's probably just me, but I'm having trouble parsing this (I think I'm
getting a cold :( ).
Are you saying you want the thread function to have access to both a
"scope" pool as well as an "allocator" pool, in case they are different?
-aaron
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
[snip]
> > that would be registered in the "parent" thread's pool -- and would only
> > be invoked by the "parent" thread.
> >
> > pools let you do this, you don't need the mutexes for it, you just have to
> > be explicit about parallelism. (combine that with a root pool per thread
> > and then we can remove alloc_mutex and free lists and push the real gnarly
> > problems into the libc malloc where it's probably best solved.)
>
> Yes, yes, yes. Can we please split the concept of a heirarchial parent (the
> 'creator' thread's or process pool, in this case) from the allocation parent
> (the actual give me memory for my pool from ... here!) Then we have an
> "OS Knows Best" malloc/free mpm for threading, just as you suggest.
>
> This solves your thread-specific requirements and our scoping issues, along
> with fixing the 'walk the chain of pools for a block' problem, both at once.
It's probably just me, but I'm having trouble parsing this (I think I'm
getting a cold :( ).
Are you saying you want the thread function to have access to both a
"scope" pool as well as an "allocator" pool, in case they are different?
-aaron
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
From: "dean gaudet" <de...@arctic.org>
Sent: Tuesday, July 17, 2001 6:15 PM
> if you assume that you want some form of notification, but you want to
> leave it unspecified because you're not sure what each apr thread will be
> used for, then you can make a somewhat generic "kill off other threads"
> cleanup.
>
> so for example, when an httpd thread is created it would register
> http_thread_cleanup which would use whatever magic global int
> die_now_please = 1, and release some condition variable, or throw
> something into a queue / and so on.
>
> that would be registered in the "parent" thread's pool -- and would only
> be invoked by the "parent" thread.
>
> pools let you do this, you don't need the mutexes for it, you just have to
> be explicit about parallelism. (combine that with a root pool per thread
> and then we can remove alloc_mutex and free lists and push the real gnarly
> problems into the libc malloc where it's probably best solved.)
Yes, yes, yes. Can we please split the concept of a heirarchial parent (the
'creator' thread's or process pool, in this case) from the allocation parent
(the actual give me memory for my pool from ... here!) Then we have an
"OS Knows Best" malloc/free mpm for threading, just as you suggest.
This solves your thread-specific requirements and our scoping issues, along
with fixing the 'walk the chain of pools for a block' problem, both at once.
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Luke Kenneth Casson Leighton <lk...@samba-tng.org>.
On Tue, Jul 17, 2001 at 10:30:33PM -0400, Bill Stoddard wrote:
> the ultimate goal of abstracting out the essentials for an event driven API and
> implementing them on Windows using completion ports, on FreeBSD with kqueue/kevent,
ooh, goodie! i get to see some actual real-live code using
NT (aka VMS) IoCompletionPorts! *excited*
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Bill Stoddard <bi...@wstoddard.com>.
> :)
>
> all is not lost.
>
> if you assume that you want some form of notification, but you want to
> leave it unspecified because you're not sure what each apr thread will be
> used for, then you can make a somewhat generic "kill off other threads"
> cleanup.
>
> so for example, when an httpd thread is created it would register
> http_thread_cleanup which would use whatever magic global int
> die_now_please = 1, and release some condition variable, or throw
> something into a queue / and so on.
>
> that would be registered in the "parent" thread's pool -- and would only
> be invoked by the "parent" thread.
>
> pools let you do this, you don't need the mutexes for it, you just have to
> be explicit about parallelism. (combine that with a root pool per thread
> and then we can remove alloc_mutex and free lists and push the real gnarly
> problems into the libc malloc where it's probably best solved.)
>
> the main problem i see is that there's no easy way to break a thread out
> of accept(). but folks may want to look at something such as SIGIO, or
> having a single acceptor thread per process,
yes. I think a single acceptor thread per process is reasonable. And it sets the code up
nicely for
> or ... using kevent
> (freebsd), rt signals (linux), /dev/poll (solaris), completion ports (nt)
> which i believe all have other methods of stuffing events into the queues.
this.
>
> if i were to write a webserver today i'd probably start with a model such
> as kevent or rt signals and make the rest of the old legacy world emulate
> that, because those are the way of the future (and the past actually, vms
> had this model :). i don't care about performance on legacy operating
> systems.
>
I completely agree. I have the start of an async/event driven MPM working on Windows with
the ultimate goal of abstracting out the essentials for an event driven API and
implementing them on Windows using completion ports, on FreeBSD with kqueue/kevent,
/dev/poll on Solaris, etc. I know the AIX team is looking into an event API too (I
recommended kqueue/kevent :-)
> -dean
>
Bill
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by "William A. Rowe, Jr." <wr...@rowe-clan.net>.
From: "dean gaudet" <de...@arctic.org>
Sent: Tuesday, July 17, 2001 6:15 PM
> if you assume that you want some form of notification, but you want to
> leave it unspecified because you're not sure what each apr thread will be
> used for, then you can make a somewhat generic "kill off other threads"
> cleanup.
>
> so for example, when an httpd thread is created it would register
> http_thread_cleanup which would use whatever magic global int
> die_now_please = 1, and release some condition variable, or throw
> something into a queue / and so on.
>
> that would be registered in the "parent" thread's pool -- and would only
> be invoked by the "parent" thread.
>
> pools let you do this, you don't need the mutexes for it, you just have to
> be explicit about parallelism. (combine that with a root pool per thread
> and then we can remove alloc_mutex and free lists and push the real gnarly
> problems into the libc malloc where it's probably best solved.)
Yes, yes, yes. Can we please split the concept of a heirarchial parent (the
'creator' thread's or process pool, in this case) from the allocation parent
(the actual give me memory for my pool from ... here!) Then we have an
"OS Knows Best" malloc/free mpm for threading, just as you suggest.
This solves your thread-specific requirements and our scoping issues, along
with fixing the 'walk the chain of pools for a block' problem, both at once.
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with
MPM threaded
Posted by dean gaudet <de...@arctic.org>.
:)
all is not lost.
if you assume that you want some form of notification, but you want to
leave it unspecified because you're not sure what each apr thread will be
used for, then you can make a somewhat generic "kill off other threads"
cleanup.
so for example, when an httpd thread is created it would register
http_thread_cleanup which would use whatever magic global int
die_now_please = 1, and release some condition variable, or throw
something into a queue / and so on.
that would be registered in the "parent" thread's pool -- and would only
be invoked by the "parent" thread.
pools let you do this, you don't need the mutexes for it, you just have to
be explicit about parallelism. (combine that with a root pool per thread
and then we can remove alloc_mutex and free lists and push the real gnarly
problems into the libc malloc where it's probably best solved.)
the main problem i see is that there's no easy way to break a thread out
of accept(). but folks may want to look at something such as SIGIO, or
having a single acceptor thread per process, or ... using kevent
(freebsd), rt signals (linux), /dev/poll (solaris), completion ports (nt)
which i believe all have other methods of stuffing events into the queues.
for legacy systems (which probably don't have native threads to begin
with) you can just use SIGTERM like we did in 1.3 and block it everywhere
except during accept().
if i were to write a webserver today i'd probably start with a model such
as kevent or rt signals and make the rest of the old legacy world emulate
that, because those are the way of the future (and the past actually, vms
had this model :). i don't care about performance on legacy operating
systems.
-dean
On Tue, 17 Jul 2001, Aaron Bannert wrote:
> Uh...you knew that already, didn't you... duh...
>
> jeez now i'm the smartass ;)
>
> -aaron
>
>
> On Tue, Jul 17, 2001 at 08:43:18AM -0700, Aaron Bannert wrote:
> > Normally this would be done (in POSIX) with pthread_cancel(), passing it
> > the pthread_t from the other thread.
> >
> > Unfortunately, this is not a part of APR because many of the current OS
> > implementations of this mechanism will leak resources (aparently in the
> > kernel), and that is bad.
> >
> > -aaron
> >
> >
> > On Tue, Jul 17, 2001 at 01:32:52AM -0700, dean gaudet wrote:
> > > On Sat, 14 Jul 2001, Sander Striker wrote:
> > >
> > > > The way I see it, each process has a single pool instance as the parent
> > > > for all the threads. Resetting or destroying that pool should effectively
> > > > kill all threads. What am I missing?
> > >
> > > how does a thread kill another thread?
> > >
> > > -dean
>
>
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with
MPM threaded
Posted by dean gaudet <de...@arctic.org>.
:)
all is not lost.
if you assume that you want some form of notification, but you want to
leave it unspecified because you're not sure what each apr thread will be
used for, then you can make a somewhat generic "kill off other threads"
cleanup.
so for example, when an httpd thread is created it would register
http_thread_cleanup which would use whatever magic global int
die_now_please = 1, and release some condition variable, or throw
something into a queue / and so on.
that would be registered in the "parent" thread's pool -- and would only
be invoked by the "parent" thread.
pools let you do this, you don't need the mutexes for it, you just have to
be explicit about parallelism. (combine that with a root pool per thread
and then we can remove alloc_mutex and free lists and push the real gnarly
problems into the libc malloc where it's probably best solved.)
the main problem i see is that there's no easy way to break a thread out
of accept(). but folks may want to look at something such as SIGIO, or
having a single acceptor thread per process, or ... using kevent
(freebsd), rt signals (linux), /dev/poll (solaris), completion ports (nt)
which i believe all have other methods of stuffing events into the queues.
for legacy systems (which probably don't have native threads to begin
with) you can just use SIGTERM like we did in 1.3 and block it everywhere
except during accept().
if i were to write a webserver today i'd probably start with a model such
as kevent or rt signals and make the rest of the old legacy world emulate
that, because those are the way of the future (and the past actually, vms
had this model :). i don't care about performance on legacy operating
systems.
-dean
On Tue, 17 Jul 2001, Aaron Bannert wrote:
> Uh...you knew that already, didn't you... duh...
>
> jeez now i'm the smartass ;)
>
> -aaron
>
>
> On Tue, Jul 17, 2001 at 08:43:18AM -0700, Aaron Bannert wrote:
> > Normally this would be done (in POSIX) with pthread_cancel(), passing it
> > the pthread_t from the other thread.
> >
> > Unfortunately, this is not a part of APR because many of the current OS
> > implementations of this mechanism will leak resources (aparently in the
> > kernel), and that is bad.
> >
> > -aaron
> >
> >
> > On Tue, Jul 17, 2001 at 01:32:52AM -0700, dean gaudet wrote:
> > > On Sat, 14 Jul 2001, Sander Striker wrote:
> > >
> > > > The way I see it, each process has a single pool instance as the parent
> > > > for all the threads. Resetting or destroying that pool should effectively
> > > > kill all threads. What am I missing?
> > >
> > > how does a thread kill another thread?
> > >
> > > -dean
>
>
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
Uh...you knew that already, didn't you... duh...
jeez now i'm the smartass ;)
-aaron
On Tue, Jul 17, 2001 at 08:43:18AM -0700, Aaron Bannert wrote:
> Normally this would be done (in POSIX) with pthread_cancel(), passing it
> the pthread_t from the other thread.
>
> Unfortunately, this is not a part of APR because many of the current OS
> implementations of this mechanism will leak resources (aparently in the
> kernel), and that is bad.
>
> -aaron
>
>
> On Tue, Jul 17, 2001 at 01:32:52AM -0700, dean gaudet wrote:
> > On Sat, 14 Jul 2001, Sander Striker wrote:
> >
> > > The way I see it, each process has a single pool instance as the parent
> > > for all the threads. Resetting or destroying that pool should effectively
> > > kill all threads. What am I missing?
> >
> > how does a thread kill another thread?
> >
> > -dean
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
Uh...you knew that already, didn't you... duh...
jeez now i'm the smartass ;)
-aaron
On Tue, Jul 17, 2001 at 08:43:18AM -0700, Aaron Bannert wrote:
> Normally this would be done (in POSIX) with pthread_cancel(), passing it
> the pthread_t from the other thread.
>
> Unfortunately, this is not a part of APR because many of the current OS
> implementations of this mechanism will leak resources (aparently in the
> kernel), and that is bad.
>
> -aaron
>
>
> On Tue, Jul 17, 2001 at 01:32:52AM -0700, dean gaudet wrote:
> > On Sat, 14 Jul 2001, Sander Striker wrote:
> >
> > > The way I see it, each process has a single pool instance as the parent
> > > for all the threads. Resetting or destroying that pool should effectively
> > > kill all threads. What am I missing?
> >
> > how does a thread kill another thread?
> >
> > -dean
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
Normally this would be done (in POSIX) with pthread_cancel(), passing it
the pthread_t from the other thread.
Unfortunately, this is not a part of APR because many of the current OS
implementations of this mechanism will leak resources (aparently in the
kernel), and that is bad.
-aaron
On Tue, Jul 17, 2001 at 01:32:52AM -0700, dean gaudet wrote:
> On Sat, 14 Jul 2001, Sander Striker wrote:
>
> > The way I see it, each process has a single pool instance as the parent
> > for all the threads. Resetting or destroying that pool should effectively
> > kill all threads. What am I missing?
>
> how does a thread kill another thread?
>
> -dean
Re: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Aaron Bannert <aa...@ebuilt.com>.
Normally this would be done (in POSIX) with pthread_cancel(), passing it
the pthread_t from the other thread.
Unfortunately, this is not a part of APR because many of the current OS
implementations of this mechanism will leak resources (aparently in the
kernel), and that is bad.
-aaron
On Tue, Jul 17, 2001 at 01:32:52AM -0700, dean gaudet wrote:
> On Sat, 14 Jul 2001, Sander Striker wrote:
>
> > The way I see it, each process has a single pool instance as the parent
> > for all the threads. Resetting or destroying that pool should effectively
> > kill all threads. What am I missing?
>
> how does a thread kill another thread?
>
> -dean
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with
MPM threaded
Posted by dean gaudet <de...@arctic.org>.
On Sat, 14 Jul 2001, Sander Striker wrote:
> The way I see it, each process has a single pool instance as the parent
> for all the threads. Resetting or destroying that pool should effectively
> kill all threads. What am I missing?
how does a thread kill another thread?
-dean
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with
MPM threaded
Posted by dean gaudet <de...@arctic.org>.
On Sat, 14 Jul 2001, Sander Striker wrote:
> The way I see it, each process has a single pool instance as the parent
> for all the threads. Resetting or destroying that pool should effectively
> kill all threads. What am I missing?
how does a thread kill another thread?
-dean
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Sander Striker <st...@apache.org>.
Right, changed the subject line again, the typo was hurting my
eyes and I wouldn't want a thread with that subj. line.
Sorry for reposting.
Justin:
>> By having the possibility of having other children processes, you now
>> need a mechanism to kill all threads in the same process efficiently.
>> You'd need to kick them out of the accept mutex, but I'm not seeing
>> how that's going to happen quickly. -- justin
Ryan:
> You need to be able to kill all threads in a process efficiently
> regardless. We will always need to be able to re-spawn processes that
> die, and we will always have the concept of MaxRequestsPerChild. If a
> process is going to die and be replaced, then we need to kill all the
> threads in that process as quickly as possible. Only allowing one
> process at a time doesn't remove that requirement.
The way I see it, each process has a single pool instance as the parent
for all the threads. Resetting or destroying that pool should effectively
kill all threads. What am I missing?
Sander
RE: Terminating threads in a process, WAS: RE: [PATCH] Problems with MPM threaded
Posted by Sander Striker <st...@apache.org>.
Right, changed the subject line again, the typo was hurting my
eyes and I wouldn't want a thread with that subj. line.
Sorry for reposting.
Justin:
>> By having the possibility of having other children processes, you now
>> need a mechanism to kill all threads in the same process efficiently.
>> You'd need to kick them out of the accept mutex, but I'm not seeing
>> how that's going to happen quickly. -- justin
Ryan:
> You need to be able to kill all threads in a process efficiently
> regardless. We will always need to be able to re-spawn processes that
> die, and we will always have the concept of MaxRequestsPerChild. If a
> process is going to die and be replaced, then we need to kill all the
> threads in that process as quickly as possible. Only allowing one
> process at a time doesn't remove that requirement.
The way I see it, each process has a single pool instance as the parent
for all the threads. Resetting or destroying that pool should effectively
kill all threads. What am I missing?
Sander
Re: Terminiting threads in a process RE: [PATCH] Problems with MPM
threaded
Posted by rb...@covalent.net.
On Sat, 14 Jul 2001, Sander Striker wrote:
> >> By having the possibility of having other children processes, you now
> >> need a mechanism to kill all threads in the same process efficiently.
> >> You'd need to kick them out of the accept mutex, but I'm not seeing
> >> how that's going to happen quickly. -- justin
> >
> > You need to be able to kill all threads in a process efficiently
> > regardless. We will always need to be able to re-spawn processes that
> > die, and we will always have the concept of MaxRequestsPerChild. If a
> > process is going to die and be replaced, then we need to kill all the
> > threads in that process as quickly as possible. Only allowing one
> > process at a time doesn't remove that requirement.
>
> The way I see it, each process has a single pool instance as the parent
> for all the threads. Resetting or destroying that pool should effectively
> kill all threads. What am I missing?
Nothing.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Sander Striker <st...@apache.org>.
>> By having the possibility of having other children processes, you now
>> need a mechanism to kill all threads in the same process efficiently.
>> You'd need to kick them out of the accept mutex, but I'm not seeing
>> how that's going to happen quickly. -- justin
>
> You need to be able to kill all threads in a process efficiently
> regardless. We will always need to be able to re-spawn processes that
> die, and we will always have the concept of MaxRequestsPerChild. If a
> process is going to die and be replaced, then we need to kill all the
> threads in that process as quickly as possible. Only allowing one
> process at a time doesn't remove that requirement.
The way I see it, each process has a single pool instance as the parent
for all the threads. Resetting or destroying that pool should effectively
kill all threads. What am I missing?
Sander
Terminiting threads in a process RE: [PATCH] Problems with MPM threaded
Posted by Sander Striker <st...@apache.org>.
>> By having the possibility of having other children processes, you now
>> need a mechanism to kill all threads in the same process efficiently.
>> You'd need to kick them out of the accept mutex, but I'm not seeing
>> how that's going to happen quickly. -- justin
>
> You need to be able to kill all threads in a process efficiently
> regardless. We will always need to be able to re-spawn processes that
> die, and we will always have the concept of MaxRequestsPerChild. If a
> process is going to die and be replaced, then we need to kill all the
> threads in that process as quickly as possible. Only allowing one
> process at a time doesn't remove that requirement.
The way I see it, each process has a single pool instance as the parent
for all the threads. Resetting or destroying that pool should effectively
kill all threads. What am I missing?
Sander
Re: [PATCH] Problems with MPM threaded
Posted by rb...@covalent.net.
On Sat, 14 Jul 2001, Justin Erenkrantz wrote:
> On Sat, Jul 14, 2001 at 09:19:17AM -0700, rbb@covalent.net wrote:
> > No! The threaded MPM does NOT and was not intended to implement a
> > thread-only server. It was designed to implement a hybrid thread/process
> > server. Because we expect both threads and processes, the rest of this
> > message is based on a misconception. Having multiple processes each with
> > multiple threads provides for FAR more robustness than just a single
> > process with multiple threads. If you want that, then you want the
> > perchild MPM, so that the number of threads can grow and shrink correctly.
>
> What are we gaining by having multiple children processes? I certainly
> don't think the robustness gained by having multiple children processes
> is all that great. And, it serves to make the threaded MPM code almost
Apache allows 3rd-party modules, which makes it much more likely that the
server will seg fault. If you only have one process, then your server
won't be serving any requests during the time it takes to start a second
server. Add to that the time it takes for the parent process to notice
that the child has died (up to 1 second), and you have missed serving a
lot of requests.
Now, add scalability concerns. The threaded MPM creates a set number of
threads per child process. This means that at ALL times, you have to have
enough threads to handle the maximum number of concurrent connections.
That is bogus. I realize that threads are less expensive than processes,
but come on. Why should have my enough threads in my server to handle my
peak load at all times?
The Perchild MPM was designed to allow you to grow the number of threads
in a given process. So, you can bring the number of threads down to 10 or
20, and grow as needed.
> unmaintainble. Perchild looks a little closer, but I still think it is
> tainted by having multiple children processes. Oh, and it needs the
> POD fix as well.
Perchild can have as few children as you want. Set it up to run 1 child,
and it will just have one child.
> By having the possibility of having other children processes, you now
> need a mechanism to kill all threads in the same process efficiently.
> You'd need to kick them out of the accept mutex, but I'm not seeing
> how that's going to happen quickly. -- justin
You need to be able to kill all threads in a process efficiently
regardless. We will always need to be able to re-spawn processes that
die, and we will always have the concept of MaxRequestsPerChild. If a
process is going to die and be replaced, then we need to kill all the
threads in that process as quickly as possible. Only allowing one
process at a time doesn't remove that requirement.
Feel free to create an MPM that only allows one process at a time, that is
why the MPM mechanism exists, but Threaded and Perchild were both designed
to allow multiple child processes for a reason.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Sat, Jul 14, 2001 at 09:19:17AM -0700, rbb@covalent.net wrote:
> No! The threaded MPM does NOT and was not intended to implement a
> thread-only server. It was designed to implement a hybrid thread/process
> server. Because we expect both threads and processes, the rest of this
> message is based on a misconception. Having multiple processes each with
> multiple threads provides for FAR more robustness than just a single
> process with multiple threads. If you want that, then you want the
> perchild MPM, so that the number of threads can grow and shrink correctly.
What are we gaining by having multiple children processes? I certainly
don't think the robustness gained by having multiple children processes
is all that great. And, it serves to make the threaded MPM code almost
unmaintainble. Perchild looks a little closer, but I still think it is
tainted by having multiple children processes. Oh, and it needs the
POD fix as well.
By having the possibility of having other children processes, you now
need a mechanism to kill all threads in the same process efficiently.
You'd need to kick them out of the accept mutex, but I'm not seeing
how that's going to happen quickly. -- justin
Why I think we need a SPMT model was Re: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Wed, Jul 18, 2001 at 02:53:33PM -0400, Benjamin W. Ritcey wrote:
> As an end-user of apache, I'd like to chime in: please, please, please leave
> the N processes with M threads model available -- "the hypothetical buggy
> 3rd party modules should be fixed rather than hacked around" is fine in
> theory, but in practice I'd rather have an Apache that can soldier on
> regardless of the crappy modules it's forced to run. Tuxedo has used the N
> x M model for a long time with good results for similar reasons.
Let me just say that the entire point of the MPM system is to let the
admin choose which model they wish to run under. I think there is a
place for the multi-process/multi-thread (mpmt) model.
I do think that a single-process/multiple-thread (spmt) model will be
able to show us a lot of the scalability problems inherent in our
code and in the overall system. I agree with Dean that we should try
and expose problems where they exist rather than covering them up
by blindly using a mpmt model.
My personal take on it is that the spmt model is easier to code. The
mpmt code currently in Apache 2.0 is quite complicated. This
complexity in the code prevents people from analyzing the code and
getting it to work properly. The mpmt MPM in Apache 2.0 needs a lot
more work/analysis before it is of the quality/robustness of the
prefork MPM. Please remember that we have a lot of experience doing a
prefork-style model for many years. The threading MPM is still
relatively new to us and our code.
I think it might be a good exercise on the development side to prove
that a spmt is solid and then move on to a mpmt MPM. Doing a mpmt
after a solid spmt model is proven is a lot easier (IMHO) than
jumping straight to the mpmt model without knowing that the spmt model
even works. The mpmt MPM implicitly relies on the fact that a spmt
works well.
And, yes, I do plan on writing a spmt MPM. But, first we need SMS
bug-free and the APR thread API needs some tweaks. A patch has been
submitted for the latter and we need more exposure with the SMS code.
It's time to stop squabbling about design and lead with the code. =)
-- justin
Re[2]: [PATCH] Problems with MPM threaded
Posted by Jeff Stuart <js...@neo.rr.com>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
AMEN! My goal as a sys admin/web admin is to keep serving pages
during ALL
problems. :) IE the server is up and STAYS up.
Wednesday, July 18, 2001, 2:53:33 PM, Benjamin W. Ritcey wrote:
BWR> As an end-user of apache, I'd like to chime in: please, please,
please leave
BWR> the N processes with M threads model available -- "the
hypothetical buggy
BWR> 3rd party modules should be fixed rather than hacked around" is
fine in
BWR> theory, but in practice I'd rather have an Apache that can
soldier on
BWR> regardless of the crappy modules it's forced to run. Tuxedo has
used the N
BWR> x M model for a long time with good results for similar reasons.
BWR> My $0.02
BWR> Thanks,
BWR> -b
- --
Jeff Stuart
jstuart@neo.rr.com
--8
-----BEGIN PGP SIGNATURE-----
Version: PGP 6.5i
iQA/AwUBO1YHXo2gaWDOuGieEQLhggCfU4mo99fsaEqlGyG/w8UfiG4jIfsAoPJz
MLNVick8G5YlVlHE75aTBSW/
=6kNT
-----END PGP SIGNATURE-----
RE: [PATCH] Problems with MPM threaded
Posted by "Benjamin W. Ritcey" <be...@ritcey.com>.
As an end-user of apache, I'd like to chime in: please, please, please leave
the N processes with M threads model available -- "the hypothetical buggy
3rd party modules should be fixed rather than hacked around" is fine in
theory, but in practice I'd rather have an Apache that can soldier on
regardless of the crappy modules it's forced to run. Tuxedo has used the N
x M model for a long time with good results for similar reasons.
My $0.02
Thanks,
-b
-----Original Message-----
From: dean gaudet [mailto:dgaudet-list-new-httpd@arctic.org]
Sent: Tuesday, July 17, 2001 4:13 AM
To: new-httpd@apache.org
Subject: Re: [PATCH] Problems with MPM threaded
On Sat, 14 Jul 2001 rbb@covalent.net wrote:
> Having multiple processes each with multiple threads provides for FAR
> more robustness than just a single process with multiple threads.
ya know, i'm not really convinced of the desirability of this explanation
anymore. maybe the hypothetical buggy 3rd party modules should be fixed
rather than hacked around. and they won't be fixed as long as apache
continues to hide the problem.
however i can provide two more justifications which are pretty weak:
- userland thread libraries need multiple processes in order to get disk
i/o parallelism (and multi-cpu parallelism)
- suppose you configure apache with exactly the number of children as
you have processors. and suppose that somehow magically the CPU
requirements of each process' threads is about even. then a good
(kernel) scheduler will likely migrate each process to its own CPU and
time slice just the threads within that process. this will tend to
decrease the overall memory system traffic (because L2 caches won't be
competing for the same process' memory), and increase the TLB hit rate.
it may even be possible to avoid interprocessor interrupts every
time the memory map is changed (to make sure that remote TLBs are
up to date).
that's a lot of conditionals to get right.
-dean
Re: [PATCH] Problems with MPM threaded
Posted by dean gaudet <dg...@arctic.org>.
On Sat, 14 Jul 2001 rbb@covalent.net wrote:
> Having multiple processes each with multiple threads provides for FAR
> more robustness than just a single process with multiple threads.
ya know, i'm not really convinced of the desirability of this explanation
anymore. maybe the hypothetical buggy 3rd party modules should be fixed
rather than hacked around. and they won't be fixed as long as apache
continues to hide the problem.
however i can provide two more justifications which are pretty weak:
- userland thread libraries need multiple processes in order to get disk
i/o parallelism (and multi-cpu parallelism)
- suppose you configure apache with exactly the number of children as
you have processors. and suppose that somehow magically the CPU
requirements of each process' threads is about even. then a good
(kernel) scheduler will likely migrate each process to its own CPU and
time slice just the threads within that process. this will tend to
decrease the overall memory system traffic (because L2 caches won't be
competing for the same process' memory), and increase the TLB hit rate.
it may even be possible to avoid interprocessor interrupts every
time the memory map is changed (to make sure that remote TLBs are
up to date).
that's a lot of conditionals to get right.
-dean
Re: [PATCH] Problems with MPM threaded
Posted by rb...@covalent.net.
> > SOLUTION 2: One could use a slightly more _involved_ approach, where the
> > dying thread could send a signal to its sibling threads, each of which will
> > then handle that signal with a graceful exit.
>
> How? A thread doesn't have a process id associated with it. And, I'm
> curious to whether the signal would somehow kick us out of the acquire
> function. Doubtful. I may be wrong (probably am). Please correct me.
> We need to be able to tell the threads they need to exit, but you can't
> force them to just exit (which is why pthread_cancel wouldn't work).
> They need to exit on their own - otherwise, they won't cleanup.
>
> > SOLUTION 3: We could turn our heads the other way by not worrying about this
> > situation at all, since these threads would eventually die (when more
> > requests are handled by the webserver). However, by ignoring the problem,
> > the purpose of perform_idle_server_maintenance() would be lost!!
>
> And, based on my interpretation and analysis of the threaded MPM and its
> intentions, we shouldn't even need to worry about this case. Let me
> explain:
>
> The problem you are stating is that the POD is received by one thread in
> one child process, it marks workers_may_exit and then quits. The accept
> mutex is then shifted over to another child process which doesn't have
> workers_may_exit set and goes on its merry way. The only time we check
> for workers_may_exit is after the mutex has been acquired. Therefore,
> it can take a while to have all of the threads in that POD-receiving
> process to see workers_may_exit.
>
> Threaded MPM is designed (rather should be - the implementation
> doesn't suggest this) to only have ONE child process that does the
No! The threaded MPM does NOT and was not intended to implement a
thread-only server. It was designed to implement a hybrid thread/process
server. Because we expect both threads and processes, the rest of this
message is based on a misconception. Having multiple processes each with
multiple threads provides for FAR more robustness than just a single
process with multiple threads. If you want that, then you want the
perchild MPM, so that the number of threads can grow and shrink correctly.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: [PATCH] Problems with MPM threaded
Posted by Justin Erenkrantz <je...@ebuilt.com>.
On Sat, Jul 14, 2001 at 12:42:29AM -0400, GUMMALAM,MOHAN (HP-Cupertino,ex2) wrote:
> I propose the following patch [PATCH A]: It will partially fix the unwanted
> child deaths problem (symptoms mentioned in the mails included below). It
> fixes the problem by making sure that perform_idle_server_maintenance() does
> not count the threads of the process that recd the POD, in the calculation
> of idle_thread_count. To do that I have used
> ap_scoreboard_image->parent[process_slot]->process_status field. I am
> temporarily using an already defined value, SB_IDLE_DIE. If the general
> idea is acceptable, I can work on solidifying the details. PATCH A is
> attached below.
Have you taken a look at the patch I posted that merges the POD code in
threaded with the version in mpm_common.c? Threaded shouldn't be doing
POD checks in threaded.c. It's redundant and it's done incorrectly
anyway in the threaded MPM.
Admittedly, this doesn't fix the issue of having a child who received a
POD kill its sibling threads. More on that in a sec.
> However this patch exposes another problem in the code - by this new fix,
> although the untargetted childs do not get a POD, the targetted child
> process does not die immediately either. Here is why that happens: In the
> worker_thread() routine in threaded.c, at the instant when worker 1.0
> (represented in the <process_slot>.<thread_slot> format, i.e, 1 is the
> process_slot and 0 is the thread_slot) gets the POD, the remaining threads
> of process 1 are all waiting at the apr_lock_acquire(accept_mutex). If the
> web-server is really idle, the chances are slim that all the remaining
> worker threads for process 1 would acquire the lock in a very short time.
> As an effect, the remaining worker threads of process 1 do not die
> immediately. To resolve this:
>
> SOLUTION 1: I plan to temporarily implement a new
> apr_lock_acquire_timeout() function, which would cause the threads waiting
> on the mutex to give-up after sometime. The piece of code would look
> something like this:
>
> while ((rv = SAFE_ACCEPT(apr_lock_acquire_timeout(accept_mutex, timevalue)))
> != APR_SUCCESS) {
> if (check_if_timer_popped)
> if (workers_may_exit)
> break;
> else { /* apr_lock_acquire failed */
> ap_log_error(....);
> workers_may_exit = 1;
> }
> }
>
> I know that this would cause some performance impact, but my guess is that
> it would not be a lot, especially if we keep the timevalue reasonably high.
> However, in order to get the functionality of
> perform_idle_server_maintenance() working right, we _will_ have to implement
> the above solution (or maybe something similar)!
-1 (my veto only matters in APR-land). This is ugly. How are you going
to implement this? The problem is that we need to wake the system up from
holding the mutex. But, that is almost impossible to implement portably
AFAIK. Please enlighten me if you know of a way to do this. You can't
do a "trylock" (assuming you can do so with all variants) and then sleep
for the timeout value. That doesn't work either. (All of the threads are
sleeping after trying a held lock when someone comes along? Oops.)
The only alternative that I can think of is pthread_cancel(), BUT (at
least Solaris) says (man cancellation):
A mutex is explicitly not a cancellation point and should
be held for only the minimal essential time.
A pthread_cond_wait() and fcntl() are viable options according to the
manpage. But, I'd much prefer us to use pthread_mutex_t if at all
possible for cross-process locking. (Yeah, yeah, we require Solaris 8+
to do pthread_mutex_t for a cross-process lock now for robust locking.
That's what you get for having child processes - see below...)
> SOLUTION 2: One could use a slightly more _involved_ approach, where the
> dying thread could send a signal to its sibling threads, each of which will
> then handle that signal with a graceful exit.
How? A thread doesn't have a process id associated with it. And, I'm
curious to whether the signal would somehow kick us out of the acquire
function. Doubtful. I may be wrong (probably am). Please correct me.
We need to be able to tell the threads they need to exit, but you can't
force them to just exit (which is why pthread_cancel wouldn't work).
They need to exit on their own - otherwise, they won't cleanup.
> SOLUTION 3: We could turn our heads the other way by not worrying about this
> situation at all, since these threads would eventually die (when more
> requests are handled by the webserver). However, by ignoring the problem,
> the purpose of perform_idle_server_maintenance() would be lost!!
And, based on my interpretation and analysis of the threaded MPM and its
intentions, we shouldn't even need to worry about this case. Let me
explain:
The problem you are stating is that the POD is received by one thread in
one child process, it marks workers_may_exit and then quits. The accept
mutex is then shifted over to another child process which doesn't have
workers_may_exit set and goes on its merry way. The only time we check
for workers_may_exit is after the mutex has been acquired. Therefore,
it can take a while to have all of the threads in that POD-receiving
process to see workers_may_exit.
Threaded MPM is designed (rather should be - the implementation
doesn't suggest this) to only have ONE child process that does the
actual work. Think about it. A thread is by its very nature (unless
we have some thread-poor OSes - but they shouldn't be using threads in
the first place) designed to be isolated and cheap (cheaper than a child
process on many OSes, but not others). There is no gain (IMHO) to
having it fork to create other servers. The only reason is to protect
us from segfaults or other bad things. But, as my recent foray into
robust locks show, you can have one child process doing threading,
segfault and we'll recover fairly quickly. (We'll detect the missing
child in the parent process when we do idle_server_maint and recreate
it.) Ideally, you'll see the core of the segfault and submit us a bug
report and we'll fix it. =) (If only I could reproduce the segfaults
I was receiving last night!)
Upon further review, having the StartServers directive seems bogus in a
pure threaded MPM. We're going crazy by doing the forking to create
multiple child processes in a *threaded* MPM. If we're a threaded MPM,
then we need to only concern ourselves with threads not child processes.
I believe that we shouldn't be doing forking in a threaded MPM.
So, what does this mean? Well, if we only have one child process
(admittedly the implementation allows for more, but I think that needs
to go away), then when one worker receives the POD line, all of the
other threads are going to go away very quickly. Why? Because the
accept_mutex is now going to be released. Each of the threads will
acquire the mutex in turn and then check the value of workers_may_exit.
It'll be one, so they will ignore the accept and then exit voluntarily.
Since there should only be one child process, the mutex is going to go
to another thread in the same process - and it'll have workers_may_exit
set to 1. So forth and so on.
So, what does the POD now gain for us? The ability to shut down the
child process full of worker threads cleanly. And, I think we can do
that other ways besides POD, but I'd have to play with the code more.
I'm not blaming the POD for this - there are much bigger concerns.
The POD makes sense for prefork, but not really for a single-process,
multi-threaded MPM.
I think the threaded MPM suffers a bit from its prefork origins. I'd
like to take a stab at reworking it to make more sense in a pure
threaded environment. I realize what I'm volunteering myself for.
Since part of my OS focus is on Solaris (FreeBSD doesn't support threads
and Linux treats threads almost identically to processes), I realize
what a proper threading MPM will do to Solaris and its performance
(it'll do almost nothing to the other platforms). That's what I care
about. =) Not to mention the code is *very* confusing (decipherable,
but not easy to understand - Roy rolls his eyes when I discuss the
current threaded MPM).
And, since we have MPM, I can do this separately without affecting the
current threaded MPM code, but I wouldn't call what we have now a pure
threaded MPM.
The big thing to kill is the StartServer directive and remove the
concept of processes in the threaded MPM. I'm not totally sold on
the stability of the current threaded MPM code. I think there are
lots of room for improvement and simplificiation (it took me two
whiteboards to diagram the current threaded MPM and how it functions!).
If it doesn't fit on a napkin, it's too complicated...
Thoughts? How far off base am I? -- justin
Re: [PATCH] Problems with MPM threaded
Posted by rb...@covalent.net.
-1. The problems detailed in the messages are a misconfigured server, not
a bug in the code. The threaded MPM should count ALL idle threads,
whether they are a part of the process that was just told to die or not.
Any idle threads that are a part of the process that was just told to die
will be dead the next time idle_server_maintenance is run, so everything
works out well in the end.
You must count the non-idle threads of the server that was told to die,
because if you don't you take the real chance of going over MaxClients.
Also, the process_status field is going away soon. It isn't used
anywhere, and it isn't needed.
Ryan
On Sat, 14 Jul 2001, GUMMALAM,MOHAN (HP-Cupertino,ex2) wrote:
> I propose the following patch [PATCH A]: It will partially fix the unwanted
> child deaths problem (symptoms mentioned in the mails included below). It
> fixes the problem by making sure that perform_idle_server_maintenance() does
> not count the threads of the process that recd the POD, in the calculation
> of idle_thread_count. To do that I have used
> ap_scoreboard_image->parent[process_slot]->process_status field. I am
> temporarily using an already defined value, SB_IDLE_DIE. If the general
> idea is acceptable, I can work on solidifying the details. PATCH A is
> attached below.
>
> However this patch exposes another problem in the code - by this new fix,
> although the untargetted childs do not get a POD, the targetted child
> process does not die immediately either. Here is why that happens: In the
> worker_thread() routine in threaded.c, at the instant when worker 1.0
> (represented in the <process_slot>.<thread_slot> format, i.e, 1 is the
> process_slot and 0 is the thread_slot) gets the POD, the remaining threads
> of process 1 are all waiting at the apr_lock_acquire(accept_mutex). If the
> web-server is really idle, the chances are slim that all the remaining
> worker threads for process 1 would acquire the lock in a very short time.
> As an effect, the remaining worker threads of process 1 do not die
> immediately. To resolve this:
>
> SOLUTION 1: I plan to temporarily implement a new
> apr_lock_acquire_timeout() function, which would cause the threads waiting
> on the mutex to give-up after sometime. The piece of code would look
> something like this:
>
> while ((rv = SAFE_ACCEPT(apr_lock_acquire_timeout(accept_mutex, timevalue)))
> != APR_SUCCESS) {
> if (check_if_timer_popped)
> if (workers_may_exit)
> break;
> else { /* apr_lock_acquire failed */
> ap_log_error(....);
> workers_may_exit = 1;
> }
> }
>
> I know that this would cause some performance impact, but my guess is that
> it would not be a lot, especially if we keep the timevalue reasonably high.
> However, in order to get the functionality of
> perform_idle_server_maintenance() working right, we _will_ have to implement
> the above solution (or maybe something similar)!
>
> SOLUTION 2: One could use a slightly more _involved_ approach, where the
> dying thread could send a signal to its sibling threads, each of which will
> then handle that signal with a graceful exit.
>
> SOLUTION 3: We could turn our heads the other way by not worrying about this
> situation at all, since these threads would eventually die (when more
> requests are handled by the webserver). However, by ignoring the problem,
> the purpose of perform_idle_server_maintenance() would be lost!!
>
> Please respond with your thought, and if there are no objections, I will go
> ahead and post a patch for SOLUTION 1 sometime soon.
>
> Thanks,
> Mohan
>
> ************************* Start PATCH A ********************************
> --- server/mpm/threaded/threaded.c.orig Tue Jul 3 06:58:10 2001
> +++ server/mpm/threaded/threaded.c Fri Jul 13 18:28:44 2001
> @@ -495,7 +495,7 @@
> }
>
> /* Sets workers_may_exit if we received a character on the pipe_of_death */
> -static void check_pipe_of_death(void)
> +static void check_pipe_of_death(int process_slot)
> {
> apr_lock_acquire(pipe_of_death_mutex);
> if (!workers_may_exit) {
> @@ -511,6 +511,7 @@
> else {
> /* It won the lottery (or something else is very
> * wrong). Embrace death with open arms. */
> + ap_scoreboard_image->parent[process_slot].process_status =
> SB_IDLE_DIE;
> workers_may_exit = 1;
> }
> }
> @@ -584,7 +585,7 @@
> if (event & APR_POLLIN) {
> /* A process got a signal on the shutdown pipe. Check if
> we're
> * the lucky process to die. */
> - check_pipe_of_death();
> + check_pipe_of_death(process_slot);
> continue;
> }
>
> @@ -972,6 +973,9 @@
> int status = SERVER_DEAD;
> int any_dying_threads = 0;
> int any_dead_threads = 0;
>
> +
> + if (ap_scoreboard_image->parent[i].process_status == SB_IDLE_DIE)
> + continue;
>
> if (i >= ap_max_daemons_limit && free_length == idle_spawn_rate)
> break;
> ************************* End PATCH A ********************************
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------