You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Manoj Kasichainula <ma...@raleigh.ibm.com> on 1999/03/18 22:29:19 UTC

Is a separate sigwait thread a good idea?

Right now, the child processes in apache-pthreads use a seperate
thread running sigwait() to watch for signals to gracefully or
ungracefully die. One thing we're trying to figure out is if there is
a way to get rid of this thread. One possibility is switching to
making one of the acceptor threads be the signal handling thread.

The problem with this change is that it requires setting up an actual
signal handler, e.g. with sigaction. Pthreads calls can't be made from
signal handlers, so we can't do things like watching for thread death,
cleaning up pools (requires grabbing thread mutexes), or almost
anything else useful from within a signal handler.

So, to switch to this solution, we need to essentially set a flag
within the signal handler and hope that another thread reaches a point
at which it will check the flag.

But, what happens if we're trying to kill a hung server? For example,
we could have a child process with all the worker threads hung on a
dead NFS server or something.  We will have race conditions in the
acceptor threads that cause us not to be able to reach a flag check
point, or if we switch to a select-accept model, there may be no
acceptor threads at all. In this case, the parent process will note
that the child is still alive and SIGKILL it.

So, we can either set aside one thread for waiting on signals in each
child, or we could increase the possiblility somewhat that cleanups
won't be executed when things are going very badly. Now, these
cleanups may be important (closing long-lived database connections,
killing Kerberos tickets, or whatever). What are your preferences
here?

-- 
Manoj Kasichainula - manojk@raleigh.ibm.com
IBM Apache Development Team

Re: Is a separate sigwait thread a good idea?

Posted by Bill Stoddard <st...@raleigh.ibm.com>.
Vincent Janelle wrote:
> 
> Is there any way I can get a copy of apache-pthreads?  About my only
> requirement is mod_perl =)
> 
> On Thu, 18 Mar 1999, Manoj Kasichainula wrote:
> > Right now, the child processes in apache-pthreads use a seperate
> > thread running sigwait() to watch for signals to gracefully or
> > ungracefully die. One thing we're trying to figure out is if there is
> > a way to get rid of this thread. One possibility is switching to
> > making one of the acceptor threads be the signal handling thread.
> >
> --

Vincent,
You can pick up the code from http://dev.apache.org/from-cvs/apache-apr/

Comments and patches welcome :-)

Bill

-- 
Bill Stoddard
stoddard@raleigh.ibm.com

Re: Is a separate sigwait thread a good idea?

Posted by Vincent Janelle <vj...@home.com>.
Is there any way I can get a copy of apache-pthreads?  About my only
requirement is mod_perl =)

On Thu, 18 Mar 1999, Manoj Kasichainula wrote:
> Right now, the child processes in apache-pthreads use a seperate
> thread running sigwait() to watch for signals to gracefully or
> ungracefully die. One thing we're trying to figure out is if there is
> a way to get rid of this thread. One possibility is switching to
> making one of the acceptor threads be the signal handling thread.
>
--
------------
According to Microsoft KnowledgeBase article Q216641, Microsoft Windows
95/98 are limited to an uptime of *exactly* 49.7 days.
http://support.microsoft.com/support/kb/articles/q216/6/41.aspy
--http://random.gimp.org --mailto:random@gimp.org --UIN 23939474
--Try "talk random@random.themes.org" sometime =)

Is a separate sigwait thread a good idea?

Posted by bh...@pobox.com.
Manoj Kasichainula writes:
> Right now, the child processes in apache-pthreads use a seperate
> thread running sigwait() to watch for signals to gracefully or
> ungracefully die. One thing we're trying to figure out is if there is
> a way to get rid of this thread. One possibility is switching to
> making one of the acceptor threads be the signal handling thread.

My sense of this stuff is a single sigwait thread the "right" way.

> But, what happens if we're trying to kill a hung server? For example,
> we could have a child process with all the worker threads hung on a
> dead NFS server or something.  We will have race conditions in the
> acceptor threads that cause us not to be able to reach a flag check
> point, or if we switch to a select-accept model, there may be no
> acceptor threads at all. In this case, the parent process will note
> that the child is still alive and SIGKILL it.

I was of the impression that the consensus was that no I/O would
be allowed to take place without a time out.

If you must destroy a thread you have to destory the process
until such time as the threads start managing a "you can destroy
me if you must" lock, and that would then allow the destroy +
pool cleanups to happen on timeout.

In my venue we refer to all these problems as the "push back
problem", or "clogging".  I.e. what to do when the sewage starts
backup up in the pipes.  NFS lockup is a classic case, but we get
a lot of it from clients on slow connections who attempt to view
extremely dynamic content.  We have built lots of delightful (or 
bogus if you like) schemes over the years for managing this.  For
example we'd like to avoid have a server's outbound memory queues
all inflate up like 400 pilsbury dough boy everytime the server's
ISP takes a nap or some NFS server goes out for coffee.

In the best cases the I/O channels provide meta info about their
status so the generating end can temper it's output and the
recieving end can manage it's expectations.  Such sophistication
is extremely hard to implement since the raw data is in 
in practice mostly just noise.  So don't try unless you have to.

At this layer, i.e. an httpd server, the problem comes down to
a) how much buffer are you going to authorize, and b) how patient
should the blocking be.  Generally a budget for maximum total
buffer is a very good idea and a single number setting how
patient you are is sufficent.

If the client end hangs for that period then drop the request.
If the server back end hangs for that period drop the request.
If you pasionate about making it more complex then this
the configuration hierarchy is pretty complex :-).

You have to design for the case where some NFS server clogs
up and you are forced to start destroying things.  Of course
you want to gracefully drop requests if you can, if not then
you have to start doing more cruel things.

 - ben

Re: Is a separate sigwait thread a good idea?

Posted by Dean Gaudet <dg...@arctic.org>.
I don't see why a special signal thread is a problem... it shouldn't chew
up much of anything.  (If it does we can blame the library authors and
shame them into fixing things ;)  or something like that!)

Dean

On Thu, 18 Mar 1999, Manoj Kasichainula wrote:

> Right now, the child processes in apache-pthreads use a seperate
> thread running sigwait() to watch for signals to gracefully or
> ungracefully die. One thing we're trying to figure out is if there is
> a way to get rid of this thread. One possibility is switching to
> making one of the acceptor threads be the signal handling thread.
> 
> The problem with this change is that it requires setting up an actual
> signal handler, e.g. with sigaction. Pthreads calls can't be made from
> signal handlers, so we can't do things like watching for thread death,
> cleaning up pools (requires grabbing thread mutexes), or almost
> anything else useful from within a signal handler.
> 
> So, to switch to this solution, we need to essentially set a flag
> within the signal handler and hope that another thread reaches a point
> at which it will check the flag.
> 
> But, what happens if we're trying to kill a hung server? For example,
> we could have a child process with all the worker threads hung on a
> dead NFS server or something.  We will have race conditions in the
> acceptor threads that cause us not to be able to reach a flag check
> point, or if we switch to a select-accept model, there may be no
> acceptor threads at all. In this case, the parent process will note
> that the child is still alive and SIGKILL it.
> 
> So, we can either set aside one thread for waiting on signals in each
> child, or we could increase the possiblility somewhat that cleanups
> won't be executed when things are going very badly. Now, these
> cleanups may be important (closing long-lived database connections,
> killing Kerberos tickets, or whatever). What are your preferences
> here?
> 
> -- 
> Manoj Kasichainula - manojk@raleigh.ibm.com
> IBM Apache Development Team
>