You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Jeff Trawick <tr...@attglobal.net> on 2001/06/19 19:53:34 UTC
plz vote on tagging current CVS as APACHE_2_0_19
subject says it all...
--
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by "William A. Rowe, Jr." <ad...@rowe-clan.net>.
From: "Jeff Trawick" <tr...@attglobal.net>
Sent: Tuesday, June 19, 2001 12:53 PM
> subject says it all...
+1 on an alpha. I'll veto this as a beta candidate, though, for the
reasons thoroughly discussed today.
If you believe it will help folks to have a 'fresher' alpha to beat upon,
then tag.
Notice you never need to call a vote to tag, just do so. We might not even
agree to call it an alpha (in which case, we untag, and retag when the issue
is resolved, AFAIC.) But to get it to alpha, beta, needs an actual tag and
tarball for folks to vote upon.
If you were just 'testing the waters' though, feel free to ask first :-)
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Bill Stoddard <bi...@wstoddard.com>.
+1
----- Original Message -----
From: "Jeff Trawick" <tr...@attglobal.net>
To: <ne...@apache.org>
Sent: Tuesday, June 19, 2001 1:53 PM
Subject: plz vote on tagging current CVS as APACHE_2_0_19
> subject says it all...
>
> --
> Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
> http://www.geocities.com/SiliconValley/Park/9289/
> Born in Roswell... married an alien...
>
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by rb...@covalent.net.
On Thu, 21 Jun 2001, Paul J. Reder wrote:
> rbb@covalent.net wrote:
> > It will
> > require locking anytime somebody wants to walk the scoreboard, which means
> > that potentially the scoreboard will be run once a request. Think
> > mod_status and/or mod_snmp.
>
> This is just plain not true. The pointer to the worker is placed in the
> conn_rec for direct access. Neither the worker, nor the scoreboard need
> to be locked for the worker to be updated during a request. The request
> processing knows that the worker pointer is valid for the duration of
> the request. The request processing does not care about any other
> workers or processes during update. The status code doesn't care if the
> worker data is updated while it is doing a walk.
>
> The locking is only required during the time that a worker or process is
> being added or removed from the tree or mod_status is running. And the
> locking is only needed to keep those two events out of each others hair.
> Mod_status does not happen often, nor does adding/removing workers and
> processes.
This is exactly what I said. The scoreboard needs to be locked, whenever
it is walked, namely whenever mod_status or (potentially) mod_snmp is
called. Mod_status may only be called once in a while, but what about
mod_snmp? Does that walk the scoreboard every request? Remember, you
will also still need to logic to make sure that you don't create too many
threads. If a user sets it up for 250 MaxClients, they mean 250
MaxClients. All in all, I am convinced from reading the design that a
shared linked list is the wrong solution.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by "Paul J. Reder" <re...@raleigh.ibm.com>.
rbb@covalent.net wrote:
> It will
> require locking anytime somebody wants to walk the scoreboard, which means
> that potentially the scoreboard will be run once a request. Think
> mod_status and/or mod_snmp.
This is just plain not true. The pointer to the worker is placed in the
conn_rec for direct access. Neither the worker, nor the scoreboard need
to be locked for the worker to be updated during a request. The request
processing knows that the worker pointer is valid for the duration of
the request. The request processing does not care about any other
workers or processes during update. The status code doesn't care if the
worker data is updated while it is doing a walk.
The locking is only required during the time that a worker or process is
being added or removed from the tree or mod_status is running. And the
locking is only needed to keep those two events out of each others hair.
Mod_status does not happen often, nor does adding/removing workers and
processes.
I am looking forward to seeing the 100 lines of code it takes to make
processes share worker space in the scoreboard.
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by "Paul J. Reder" <re...@raleigh.ibm.com>.
Bill Stoddard wrote:
> A few other benefits to Pauls design:
> 1. Eliminates the requirement for compiled in HARD_SERVER_LIMIT or HARD_THREAD_LIMIT.
> 2. You don't need to allocate child score if you don't care about mod_status (but it can
> be added during a restart)
> 3. If you choose to not enable mod_status, you will likely see a nice performance boost on
> multi CPU machines because we are not invalidating a CPUs cache each time we touch a
> worker score entry (which is on every request).
> 4. Does not require any changes to the MPM. Each MPM can start threads according to its'
> ThreadsPerChild setting w/o needing to pay attention to the scoreboard (I believe your
> design required child processes to wait for worker score slots to become available before
> it can start threads. This is imposing too much unnecessary work on the MPM.).
5. It fixes the problem with the threaded mpm where no work can be done while workers are
quiescing due to MRPC, leaving processes with a small number of workers finishing their
last responses and no workers available to process new requests.
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Ian Holsman <ia...@cnet.com>.
On 26 Jun 2001 21:26:03 -0400, Bill Stoddard wrote:
>
>
> I would definitely like to decouple the portion of the scoreboard used to track status
> from the portion used for child process management.
>
this is a nice lead in to the patch I submitted a while ago which
implemented the mgmt_get_lines hook in the scoreboard.
this at least put the code into the same file.
has anyone had a chance to have a look at the patch? should I resubmit
it?
http://www.apachelabs.org/apache-mbox/200106.mbox/%3c200106182251.PAA07011@cn-sfo1-g7.cnet.com%3e
(even a "it's a pile of shit, you should do XXX" would be great)
..Ian
> Bill
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Greg Marr <gr...@alum.wpi.edu>.
At 08:49 AM 06/27/2001, Bill Stoddard wrote:
>Consider a mod_status request being accepted by a thread in a
>multithreaded process then getting swapped out for an indeterminate
>period of time.
Okay, having this one get old data is fine, since the request was
submitted before the restart.
>no matter what, you need to keep the old processes from attempting
>to access the new storage.
That's true.
--
Greg Marr
gregm@alum.wpi.edu
"We thought you were dead."
"I was, but I'm better now." - Sheridan, "The Summoning"
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Bill Stoddard <bi...@wstoddard.com>.
> On Tue, 26 Jun 2001 21:26:03 -0400
> "Bill Stoddard" <bi...@wstoddard.com> wrote:
> > > Why would any of these processes still be serving new requests?
> >
> > A timing window. When a mod_status request is the last request
> > accepted before a process dies.
>
> I thought the children were woken up during the restart by the POD, and
> they exited immediately, without serving new requests.
>
Consider a mod_status request being accepted by a thread in a multithreaded process then
getting swapped out for an indeterminate period of time. During this period of time, all
the other threads in the process exit; the parent process allocates new scoreboard
storage, starts new processes, etc. Then your mod_status thread runs and walks the
scoreboard. One of two things will happen...either the mod_status thread will trip over
the newly allocated storage (if the new storage is simply added to the existing
scoreboard) or it will miss reporting status on the new threads.
Missing status of the new threads immediately after a restart where the thread/process
limits were increased is not a big deal to me. But no matter what, you need to keep the
old processes from attempting to access the new storage.
Bill
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by "Paul J. Reder" <re...@raleigh.ibm.com>.
Greg Marr wrote:
>
> At 11:18 AM 06/27/2001, rbb@covalent.net wrote:
> >On Tue, 26 Jun 2001, Greg Marr wrote:
> >
> > > On Tue, 26 Jun 2001 21:26:03 -0400
> > > "Bill Stoddard" <bi...@wstoddard.com> wrote:
> > > > > Why would any of these processes still be serving new requests?
> > > >
> > > > A timing window. When a mod_status request is the last request
> > > > accepted before a process dies.
> > >
> > > I thought the children were woken up during the restart by the
> > POD, and
> > > they exited immediately, without serving new requests.>
> >
> >They are, but a mod_status request could come in a millisecond
> >before the pod is triggered, and you will have this problem.
>
> Ah, okay. So there is a really tiny window. I'd say that you'd need
> to allow AT LEAST 5-10 seconds from the time of the restart until
> things have stabilized enough to get a reasonable status request. :)
So the long and short of this is that no one thinks it is a problem that
during a graceful restart there is a period of fluctuation which cannot
be avoided. While the system is in flux, results may be innaccurate during
certain short windows.
Any complaints?
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Greg Marr <gr...@alum.wpi.edu>.
At 11:18 AM 06/27/2001, rbb@covalent.net wrote:
>On Tue, 26 Jun 2001, Greg Marr wrote:
>
> > On Tue, 26 Jun 2001 21:26:03 -0400
> > "Bill Stoddard" <bi...@wstoddard.com> wrote:
> > > > Why would any of these processes still be serving new requests?
> > >
> > > A timing window. When a mod_status request is the last request
> > > accepted before a process dies.
> >
> > I thought the children were woken up during the restart by the
> POD, and
> > they exited immediately, without serving new requests.>
>
>They are, but a mod_status request could come in a millisecond
>before the pod is triggered, and you will have this problem.
Ah, okay. So there is a really tiny window. I'd say that you'd need
to allow AT LEAST 5-10 seconds from the time of the restart until
things have stabilized enough to get a reasonable status request. :)
--
Greg Marr
gregm@alum.wpi.edu
"We thought you were dead."
"I was, but I'm better now." - Sheridan, "The Summoning"
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by rb...@covalent.net.
On Tue, 26 Jun 2001, Greg Marr wrote:
> On Tue, 26 Jun 2001 21:26:03 -0400
> "Bill Stoddard" <bi...@wstoddard.com> wrote:
> > > Why would any of these processes still be serving new requests?
> >
> > A timing window. When a mod_status request is the last request
> > accepted before a process dies.
>
> I thought the children were woken up during the restart by the POD, and
> they exited immediately, without serving new requests.>
They are, but a mod_status request could come in a millisecond before the
pod is triggered, and you will have this problem.
Ryan
_____________________________________________________________________________
Ryan Bloom rbb@apache.org
Covalent Technologies rbb@covalent.net
-----------------------------------------------------------------------------
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Greg Marr <gr...@alum.wpi.edu>.
On Tue, 26 Jun 2001 21:26:03 -0400
"Bill Stoddard" <bi...@wstoddard.com> wrote:
> > Why would any of these processes still be serving new requests?
>
> A timing window. When a mod_status request is the last request
> accepted before a process dies.
I thought the children were woken up during the restart by the POD, and
they exited immediately, without serving new requests.
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Bill Stoddard <bi...@wstoddard.com>.
> At 05:42 PM 06/26/2001, Bill Stoddard wrote:
> >>1. Eliminates the requirement for compiled in HARD_SERVER_LIMIT or
> >>HARD_THREAD_LIMIT.
> >
> >I retract this statement. You -can- get rid of HARD_*_LIMIT iff you
> >don't mind mod_status missing some thread status from time to time
> >(which I doubt is acceptable). Consider the case where the admin
> >increases either thread or server limit then does a restart. The
> >scoreboard MUST grow. And the growth will -not- be visible to the
> >old (before the restart) processes. If a mod_status request comes
> >in and is handled by one of the old processes, it cannot report
> >status of any of the threads that are writing status to the
> >extended (new) portion of the scoreboard. I just don't see any way
> >around this.
>
> Why would any of these processes still be serving new
> requests? Wouldn't they have been killed off during the
> restart? They'd only still be around if they were serving old
> requests, those that came in before the restart. The mod_status
> request would have to go to one of the new processes.
>
> Am I missing something here?
A timing window. When a mod_status request is the last request accepted before a process
dies. Yea, this is an edge case and maybe we shouldn't care. That is my opinion, but
generally tradeoffs like this don't wash with the group :-)
>
> If this is the case, then this sounds like a good argument for
> decoupling mod_status from the scoreboard format, and using the hooks
> that were discussed here recently. (I think they were implemented,
> but I'm not sure.)
I would definitely like to decouple the portion of the scoreboard used to track status
from the portion used for child process management.
Bill
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Greg Marr <gr...@alum.wpi.edu>.
At 05:42 PM 06/26/2001, Bill Stoddard wrote:
>>1. Eliminates the requirement for compiled in HARD_SERVER_LIMIT or
>>HARD_THREAD_LIMIT.
>
>I retract this statement. You -can- get rid of HARD_*_LIMIT iff you
>don't mind mod_status missing some thread status from time to time
>(which I doubt is acceptable). Consider the case where the admin
>increases either thread or server limit then does a restart. The
>scoreboard MUST grow. And the growth will -not- be visible to the
>old (before the restart) processes. If a mod_status request comes
>in and is handled by one of the old processes, it cannot report
>status of any of the threads that are writing status to the
>extended (new) portion of the scoreboard. I just don't see any way
>around this.
Why would any of these processes still be serving new
requests? Wouldn't they have been killed off during the
restart? They'd only still be around if they were serving old
requests, those that came in before the restart. The mod_status
request would have to go to one of the new processes.
Am I missing something here?
If this is the case, then this sounds like a good argument for
decoupling mod_status from the scoreboard format, and using the hooks
that were discussed here recently. (I think they were implemented,
but I'm not sure.)
--
Greg Marr
gregm@alum.wpi.edu
"We thought you were dead."
"I was, but I'm better now." - Sheridan, "The Summoning"
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Jeff Trawick <tr...@attglobal.net>.
"Bill Stoddard" <bi...@wstoddard.com> writes:
> > A few other benefits to Pauls design:
> > 1. Eliminates the requirement for compiled in HARD_SERVER_LIMIT or HARD_THREAD_LIMIT.
>
> I retract this statement. You -can- get rid of HARD_*_LIMIT iff you don;t mind mod_status
> missing some thread status from time to time (which I doubt is acceptable). Consider the
> case where the admin increases either thread or server limit then does a restart. The
> scoreboard MUST grow. And the growth will -not- be visible to the old (before the
> restart) processes. If a mod_status request comes in and is handled by one of the old
> processes, it cannot report status of any of the threads that are writing status to the
> extended (new) portion of the scoreboard. I just don't see any way around this.
So if the last request processed by a worker in the old generation
is mod_status then it won't show status on the new generation. So
what? There is a timing window inherent in graceful restart for what
status can be displayed anyway (or what configuration is used in
general for that matter). Whether or not the new generation is
displayed *properly* is dependent upon when the request arrives
regardless of the scoreboard design. Having the request arrive
milliseconds later or earlier can show a different set of workers;
again, this is regardless of the scoreboard design.
--
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Bill Stoddard <bi...@wstoddard.com>.
> >
> > Bill,
> >
> > I will finish my code.
>
> If you spend the time to write it, I will look at it.
And I will... :-)
<snip>
> A few other benefits to Pauls design:
> 1. Eliminates the requirement for compiled in HARD_SERVER_LIMIT or HARD_THREAD_LIMIT.
I retract this statement. You -can- get rid of HARD_*_LIMIT iff you don;t mind mod_status
missing some thread status from time to time (which I doubt is acceptable). Consider the
case where the admin increases either thread or server limit then does a restart. The
scoreboard MUST grow. And the growth will -not- be visible to the old (before the
restart) processes. If a mod_status request comes in and is handled by one of the old
processes, it cannot report status of any of the threads that are writing status to the
extended (new) portion of the scoreboard. I just don't see any way around this.
Bill
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by "Paul J. Reder" <re...@raleigh.ibm.com>.
"Roy T. Fielding" wrote:
>
> > They have configured their system for the general case. We are talking
> > about the edge case here. The user has asked for 10 processes with 25
> > threads each. What they are saying, is I want 250 threads. If we are
> > all of a sudden under heavy load, then we have to give them 250 threads,
> > 227 isn't good enough.
>
> The process/thread allocation system is supposed to be adaptive to server
> load -- its goal is to keep the server going as well as possible. There is
> no way to do that and insist on a specific number of threads on every graceful
> restart. That isn't what a graceful restart means. In fact, we should
> not be generating more than (max threads - current threads) regardless of
> whether those threads are old or new server instances.
>
> ....Roy
I was talking about algorithms for keeping workers and processes going during
normal processing, with idle processing and MRPC processing. The restart stuff
is different.
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by rb...@covalent.net.
On Fri, 22 Jun 2001, Roy T. Fielding wrote:
> > They have configured their system for the general case. We are talking
> > about the edge case here. The user has asked for 10 processes with 25
> > threads each. What they are saying, is I want 250 threads. If we are
> > all of a sudden under heavy load, then we have to give them 250 threads,
> > 227 isn't good enough.
>
> The process/thread allocation system is supposed to be adaptive to server
> load -- its goal is to keep the server going as well as possible. There is
> no way to do that and insist on a specific number of threads on every graceful
> restart. That isn't what a graceful restart means. In fact, we should
> not be generating more than (max threads - current threads) regardless of
> whether those threads are old or new server instances.
Yes, we can't create more than a total of max threads, but my point is we
shouldn't be allocating less either.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by "Roy T. Fielding" <fi...@ebuilt.com>.
> They have configured their system for the general case. We are talking
> about the edge case here. The user has asked for 10 processes with 25
> threads each. What they are saying, is I want 250 threads. If we are
> all of a sudden under heavy load, then we have to give them 250 threads,
> 227 isn't good enough.
The process/thread allocation system is supposed to be adaptive to server
load -- its goal is to keep the server going as well as possible. There is
no way to do that and insist on a specific number of threads on every graceful
restart. That isn't what a graceful restart means. In fact, we should
not be generating more than (max threads - current threads) regardless of
whether those threads are old or new server instances.
....Roy
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by "Paul J. Reder" <re...@raleigh.ibm.com>.
rbb@covalent.net wrote:
> Yes, the requirement is to keep the information about the threads that are
> still processing requests. Summarizing that data is not good enough. If
> the thread is still active, the scoreboard entry must still exist.
Sorry, wasn't clear. Of course the info is all kept around until the worker goes
away. The summation happens only after all workers and processes have departed and
the shmem chunk is to be freed, if the data is needed.
>
> > You should never be able to start more processes than the config specifies. The
> > user set the config for a reason. Creating more processes than are allowed in the
> > config violates the principle of least astonishment in my opinion. Let the user
> > define what their upper bound is. They know their system. Allow them to config
> > in the amount of extra transitional processes that they determine their system
> > can handle. We should not be guessing that we can start extra processes that
> > weren't configed. IMHO this is just asking for trouble.
>
> They have configured their system for the general case. We are talking
> about the edge case here. The user has asked for 10 processes with 25
> threads each. What they are saying, is I want 250 threads. If we are
> all of a sudden under heavy load, then we have to give them 250 threads,
> 227 isn't good enough.
I agree that they want 250. Give or take small fluctuations due to MRPC and
idle processing...
What I disagree with you about is that if they say MaxClients=10 that they
meant 10 not 12 or 15 or some other random number > 10. Some systems pay a
heavy price for processes.
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by rb...@covalent.net.
> rbb@covalent.net wrote:
> > Your going to lose information between restarts, or you are going to
> > require copying large chunks of shared memory on graceful restarts.
>
> If the requirement is to maintain exactly all of th individual values, then yes the
> info will either be lost or lots of copies will be required. If the requirement is
> to collect Apache statistics and collect statistics of active workers then no, info
> will not be lost and no copying is required. The values in the individual workers that
> are going away are summed up and the totalled Apache results are stored at a higher
> level. Those workers no longer exist.
Yes, the requirement is to keep the information about the threads that are
still processing requests. Summarizing that data is not good enough. If
the thread is still active, the scoreboard entry must still exist.
> You should never be able to start more processes than the config specifies. The
> user set the config for a reason. Creating more processes than are allowed in the
> config violates the principle of least astonishment in my opinion. Let the user
> define what their upper bound is. They know their system. Allow them to config
> in the amount of extra transitional processes that they determine their system
> can handle. We should not be guessing that we can start extra processes that
> weren't configed. IMHO this is just asking for trouble.
They have configured their system for the general case. We are talking
about the edge case here. The user has asked for 10 processes with 25
threads each. What they are saying, is I want 250 threads. If we are
all of a sudden under heavy load, then we have to give them 250 threads,
227 isn't good enough.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by "Paul J. Reder" <re...@raleigh.ibm.com>.
rbb@covalent.net wrote:
> Your going to lose information between restarts, or you are going to
> require copying large chunks of shared memory on graceful restarts.
If the requirement is to maintain exactly all of th individual values, then yes the
info will either be lost or lots of copies will be required. If the requirement is
to collect Apache statistics and collect statistics of active workers then no, info
will not be lost and no copying is required. The values in the individual workers that
are going away are summed up and the totalled Apache results are stored at a higher
level. Those workers no longer exist.
> You are either going to leak shared memory like a sieve, or you are going
> to need to copy the data. At what point are you planning to free the
> shared memory that was allocated suring the first starting of the server?
During the perform_idle_server_maintenance it checks if the shmem chunk is scheduled
for cleanup and has no processes left under it. The shmem segment is then freed via
perform_idle_server_maintenance garbage collection. No leak.
> > You will need to be able to start up a new process per row to take advantage
> > of these dribs and drabs that are appearing. With my design, as soon as there
> > is ThreadsPerChild number of workers on the free list, and there is a free
> > process, I can start a new process with ThreadsPerChild workers.
>
> What do you mean, and there is a free process? Does that mean that if I
> am in a position where I have MaxClients = 5, and I have 5 processes with
> one thread in a long-lived request, that I won't start a new process?
> That won't work. You need to be able to start a new process to replace an
> old one before an old process has died, otherwise, in pathological cases,
> you will have a dead server.
My feeling is that there should be another config directive called MaxTotalWorkers
so you could define MaxClients=20, ThreadsPerChild=100, MaxTotalWorkers=1000. You
could only start up 10 processes with 100 workers each, but if some of the
processes got below 100 workers (thus below the total of 1000 workers - in fact,
below MaxTotalWorkers - ThreadsPerChild) you could start another process.
You should never be able to start more processes than the config specifies. The
user set the config for a reason. Creating more processes than are allowed in the
config violates the principle of least astonishment in my opinion. Let the user
define what their upper bound is. They know their system. Allow them to config
in the amount of extra transitional processes that they determine their system
can handle. We should not be guessing that we can start extra processes that
weren't configed. IMHO this is just asking for trouble.
>
> > What am I missing here? There is no overrun of the configured values. And the
> > algorithm isn't any more complex.
>
> The algorithm is MUCH more complex. I know it is, because I implemented
> my entire patch in 155 lines. That is the size of the entire patch. I
> need to test it more, but I expect to post later today or tomorrow.
The specific algorithm I was talking about here was the one to determine if
another process with workers can be started up. Granted, the list processing
algorithm is more complex than the static table algorithm, but I think the
gains are worth it.
Now pardon me, I am going to cease debating for a bit and work on getting the
code done and tested. We'll let the code and the possibilities do the talking.
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by rb...@covalent.net.
< snipping the pieces about locking. Not because I don't care, but
because I have stated too often that I can't accept locking the scoreboard on
every request. If this design requires that lock, I have modules that
will be greatly impacted. >
> > > The only reason we have locking at all is to prevent the 4 cases listed above from
> > > colliding with each other. Even in the 4 cases above, the lock contention will be minimal
> > > and the performance degradation minimal and perhaps not even measureable.
> > >
> > > A few other benefits to Pauls design:
> > > 1. Eliminates the requirement for compiled in HARD_SERVER_LIMIT or HARD_THREAD_LIMIT.
> >
> > You still need to request a certain amount of shared memory when you
> > start the parent process, and you will require HARD_SERVER_LIMIT and
> > HARD_THREAD_LIMIT to know how much to request. Either that, or whenever
> > you add a new child, you will need to allocate more shared memory, and
> > somehow tell your child processes about it.
>
> The allocation is based on the configured values for the MaxClients and ThreadsPerChild.
> These values can changed from one start to the next (allowing system tuning to happen
> between restarts - without a recompile!)
Your going to lose information between restarts, or you are going to
require copying large chunks of shared memory on graceful restarts.
> > > 2. You don't need to allocate child score if you don't care about mod_status (but it can
> > > be added during a restart)
> >
> > You still need to request a certain amount of shared memory when you
> > start the parent process, and you will require HARD_SERVER_LIMIT and
> > HARD_THREAD_LIMIT to know how much to request. Either that, or whenever
> > you restart, you will forget all information about your child processes.
>
> See above for allocation size basis. As for keeping info from one restart to the next,
> the code currently allocates a new scoreboard on graceless restarts and all info from the
> old scoreboard is lost. On gracefull restarts the existing scoreoard is reused. In the
> future gracefull restarts would get a clean scorebaord and the old one would eventually
> be cleaned up. The results would be collected and stored at a common top level.
You are either going to leak shared memory like a sieve, or you are going
to need to copy the data. At what point are you planning to free the
shared memory that was allocated suring the first starting of the server?
> > > 4. Does not require any changes to the MPM. Each MPM can start threads according to its'
> > > ThreadsPerChild setting w/o needing to pay attention to the scoreboard (I believe your
> > > design required child processes to wait for worker score slots to become available before
> > > it can start threads. This is imposing too much unnecessary work on the MPM.).
> >
> > You are ignoring one problem with the design. I have now posted about it
> > three times, and nobody has told me how it will be fixed. You need to
> > honor MaxClients. If I set MaxClients to 250, then I expect MaxClients to
> > be 250, regardless of whether those clients are long-lived or not. If
> > every child can always start threads according to ThreadsPerChild, you
> > will be violating MaxClients on a heavily loaded server. This means that
> > you are actually requiring MORE changes to the starting logic than the
> > table implementation, because each child, will need to determine how many
> > threads to start at any given time.
>
> There is currently code to look if another process can be started (in
> process_idle_server_maintenance), there will still be code to do that with
> my design. The design still limits the number of processes and the number
> of workers per process. The difference is that a new process can be started
> when (current_total_workers < ((MaxClients-1) * ThreadsPerChild)) provided
> there is a spare process.
>
> "My design can handle this" (I heard you say that all the way from CA). This
> is true to a point. Currently, in the table design, a process cannot own
> workers from mutiple rows. A process owns all of the workers in a given row.
> Currently there is a one to one mapping of process slots to worker rows.
> Your proposal is to allow multiple processes to map to a single row, but still
> not map a process to workers in multiple rows.
>
> Apache does a wonderful job of distributing the request load to workers.
> There is a cycle that happens where many workers start running into the MRPC
> limit and start quiescing. As workers quiesce you will see 5 freed up from
> this row, and 6 from that, and another 10 from that...
So what? Worst case, if you have a lot of child processes dying and being
respawned, you will lose some performance. If your server is displaying
this behavior, then I would argue that MRPC is set too low, and you should
take a look at how it is setup.
> You will need to be able to start up a new process per row to take advantage
> of these dribs and drabs that are appearing. With my design, as soon as there
> is ThreadsPerChild number of workers on the free list, and there is a free
> process, I can start a new process with ThreadsPerChild workers.
What do you mean, and there is a free process? Does that mean that if I
am in a position where I have MaxClients = 5, and I have 5 processes with
one thread in a long-lived request, that I won't start a new process?
That won't work. You need to be able to start a new process to replace an
old one before an old process has died, otherwise, in pathological cases,
you will have a dead server.
> What am I missing here? There is no overrun of the configured values. And the
> algorithm isn't any more complex.
The algorithm is MUCH more complex. I know it is, because I implemented
my entire patch in 155 lines. That is the size of the entire patch. I
need to test it more, but I expect to post later today or tomorrow.
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by "Paul J. Reder" <re...@raleigh.ibm.com>.
rbb@covalent.net wrote:
>
> > I don't completely understand you comment, but you are quite right that anytime the
> > scoreboard needs to be walked, locking will be required. The scoreboard will -not- need to
> > be walked except in these relatively rare cases:
> >
> > 1. During a graceful restart (manipulates elements in the scoreboard)
> > 2. During MaxRequestPerChild triggered child shutdown (manipulates elements in the
> > scoreboard)
> > 3. Processing a mod_status request (walks the scoreboard)
> > 4. Processing an SNMP request (walks the scoreboard)
> >
> > Requests that do not require a scoreboard walk or adding/deleting entries from the
> > scoreboard (which is 99.999% or more of all requests; all requests except the ones above)
> > will not require locking at all. Let me restate this... mainline requests will NOT be
> > required to make any lock calls at all. Ever. Mainline requests can be served and their
> > status tracked even when the scoreboard is locked, so this design does not suffer -any-
> > performance penalties in the mainline case caused by locking.
>
> You are looking at this from a Web server point of view. Don't. Look at
> this from an ops point of view. How often is a large site going to query
> the current connections or child status (for example) using SNMP? Both of
> those require a walk of the scoreboard. If you ask for them once a
> minute, you are going to seriously hinder the performance of your server.
> The problem gets worse as the number of threads increases. How many other
> SNMP requests carry the same penalty?
Please. Locking the scoreboard once per minute for the period of a walk would
hardly impact performance. If locking the whole scoreboard is a problem, then
locking can be performed at finer granularities. You could lock the list of
processes, each list of workers, and each free list. Now you can have multiple
walks happening at the same time in different parts of the tree. The initial
version will have a simple lock of the whole scoreboard. *If* testing shows
this to be a problem, which I doubt, then we can look at finer granularity locking.
>
> BTW, there are a few more that modules that make use of the scoreboard
> information:
>
> 5. Whenever Apache::Scoreboard walks the scoreboard (this has been used
> by perl modules to download the scoreboard once every 15 seconds to
> generate graphs on another machine.)
Once every 15 seconds is an eternity.
> 6. Whenever Apache::VMonitor walks the scoreboard (Another graphicl
> monitor of configurable refresh rate.
> 7. Whenever any other management module walks the scoreboard.
People are used to tuning the access to get the info that they need without
impacting performance. It is a trade off. I'm sure there are those who would
like to have realtime statistics, but they recognize that isn't possible.
>
> You can't guarantee how often a third party module will walk the
> scoreboard, because you don't know everything about every third part
> module. Since you don't know how often it will be walked, you don't know
> how often the mutex will be locked.
You are quite correct. But again, no one is going to install dozens of
statistics generating modules. No one is going to schedule statistics reporting
so often that it impacts service.
If someone sets the interval to be once per second, they will pay a price no
matter what scoreboard scheme is used. More often than that they pay a heavier
price. This isn't rocket science.
We have a tradeoff here. They may not be able to gather stats quite as often
due to the locking. The benefit that they get is a great deal more robustness
and flexibility. If they get a system that can scale better, perform better on
multi-processor machines, run faster and longer, and is easier to configure
do you think they might be willing to poll for statistics once every 5 seconds
instead of once per second?
And this is all handwaving anyway. You disagree. Fine. The true test will be the
performance of the end product.
>
> > The only reason we have locking at all is to prevent the 4 cases listed above from
> > colliding with each other. Even in the 4 cases above, the lock contention will be minimal
> > and the performance degradation minimal and perhaps not even measureable.
> >
> > A few other benefits to Pauls design:
> > 1. Eliminates the requirement for compiled in HARD_SERVER_LIMIT or HARD_THREAD_LIMIT.
>
> You still need to request a certain amount of shared memory when you
> start the parent process, and you will require HARD_SERVER_LIMIT and
> HARD_THREAD_LIMIT to know how much to request. Either that, or whenever
> you add a new child, you will need to allocate more shared memory, and
> somehow tell your child processes about it.
The allocation is based on the configured values for the MaxClients and ThreadsPerChild.
These values can changed from one start to the next (allowing system tuning to happen
between restarts - without a recompile!)
>
> > 2. You don't need to allocate child score if you don't care about mod_status (but it can
> > be added during a restart)
>
> You still need to request a certain amount of shared memory when you
> start the parent process, and you will require HARD_SERVER_LIMIT and
> HARD_THREAD_LIMIT to know how much to request. Either that, or whenever
> you restart, you will forget all information about your child processes.
See above for allocation size basis. As for keeping info from one restart to the next,
the code currently allocates a new scoreboard on graceless restarts and all info from the
old scoreboard is lost. On gracefull restarts the existing scoreoard is reused. In the
future gracefull restarts would get a clean scorebaord and the old one would eventually
be cleaned up. The results would be collected and stored at a common top level.
>
> > 3. If you choose to not enable mod_status, you will likely see a nice performance boost on
> > multi CPU machines because we are not invalidating a CPUs cache each time we touch a
> > worker score entry (which is on every request).
>
> This can be implemented just as well with the table implementation. The
> only thing you have to do is pad the scoreboard entry size to make it
> equal to one cache line.
Padding the entries uses more memory but makes mapping into the cache easier. It *does not*
in any way limit the cache invalidation impact of updating scoreboard entries *that are never
read* when a status module is not loaded.
>
> > 4. Does not require any changes to the MPM. Each MPM can start threads according to its'
> > ThreadsPerChild setting w/o needing to pay attention to the scoreboard (I believe your
> > design required child processes to wait for worker score slots to become available before
> > it can start threads. This is imposing too much unnecessary work on the MPM.).
>
> You are ignoring one problem with the design. I have now posted about it
> three times, and nobody has told me how it will be fixed. You need to
> honor MaxClients. If I set MaxClients to 250, then I expect MaxClients to
> be 250, regardless of whether those clients are long-lived or not. If
> every child can always start threads according to ThreadsPerChild, you
> will be violating MaxClients on a heavily loaded server. This means that
> you are actually requiring MORE changes to the starting logic than the
> table implementation, because each child, will need to determine how many
> threads to start at any given time.
There is currently code to look if another process can be started (in
process_idle_server_maintenance), there will still be code to do that with
my design. The design still limits the number of processes and the number
of workers per process. The difference is that a new process can be started
when (current_total_workers < ((MaxClients-1) * ThreadsPerChild)) provided
there is a spare process.
"My design can handle this" (I heard you say that all the way from CA). This
is true to a point. Currently, in the table design, a process cannot own
workers from mutiple rows. A process owns all of the workers in a given row.
Currently there is a one to one mapping of process slots to worker rows.
Your proposal is to allow multiple processes to map to a single row, but still
not map a process to workers in multiple rows.
Apache does a wonderful job of distributing the request load to workers.
There is a cycle that happens where many workers start running into the MRPC
limit and start quiescing. As workers quiesce you will see 5 freed up from
this row, and 6 from that, and another 10 from that...
You will need to be able to start up a new process per row to take advantage
of these dribs and drabs that are appearing. With my design, as soon as there
is ThreadsPerChild number of workers on the free list, and there is a free
process, I can start a new process with ThreadsPerChild workers.
What am I missing here? There is no overrun of the configured values. And the
algorithm isn't any more complex.
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by dean gaudet <dg...@arctic.org>.
On Thu, 21 Jun 2001, Bill Stoddard wrote:
> > This can be implemented just as well with the table implementation. The
> > only thing you have to do is pad the scoreboard entry size to make it
> > equal to one cache line.
> >
> And waste more storage? I am not a CPU designer...how big are cache lines on the different CPUs?
> Wouldn;t padding the storage just cause you to load your cache with memory that is NOT USED at all?
cache line sizes vary, they're typically 32, 64, or 128 bytes these days.
32 bytes is the L1 line size on the p-iii family, the p4 family uses a
128-byte L1 line size.
so what happens when you have two processors banging on the same line is
that the memory accesses proceed at memory bus speed. so instead of
taking 1 cycle for a read/write it takes 100s of cycles.
loading the cache with unused memory is a small penalty if it means you
can reap the benefits of exclusive access to that line of memory.
if you want to see some of this stuff for yourself grab
<ftp://ftp.bitkeeper.com/lmbench/lmbench-2beta3.tgz> and look at the
memory timings on your favourite system.
-dean
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Bill Stoddard <bi...@wstoddard.com>.
>
> > I don't completely understand you comment, but you are quite right that anytime the
> > scoreboard needs to be walked, locking will be required. The scoreboard will -not- need to
> > be walked except in these relatively rare cases:
> >
> > 1. During a graceful restart (manipulates elements in the scoreboard)
> > 2. During MaxRequestPerChild triggered child shutdown (manipulates elements in the
> > scoreboard)
> > 3. Processing a mod_status request (walks the scoreboard)
> > 4. Processing an SNMP request (walks the scoreboard)
> >
> > Requests that do not require a scoreboard walk or adding/deleting entries from the
> > scoreboard (which is 99.999% or more of all requests; all requests except the ones above)
> > will not require locking at all. Let me restate this... mainline requests will NOT be
> > required to make any lock calls at all. Ever. Mainline requests can be served and their
> > status tracked even when the scoreboard is locked, so this design does not suffer -any-
> > performance penalties in the mainline case caused by locking.
>
> You are looking at this from a Web server point of view. Don't. Look at
> this from an ops point of view. How often is a large site going to query
> the current connections or child status (for example) using SNMP? Both of
> those require a walk of the scoreboard. If you ask for them once a
> minute, you are going to seriously hinder the performance of your server.
No way! Where are the cycles going? Waiting for locks? Probably not. CPU overhead of calling a
lock? If there is no contention (and their most likely will not be contention) the overhed of
calling a lock is minimal. Running the linked list? Running a linked list of even 10,000 (which
would be a -very- large site) would not be a big deal if you do it even once a minute. I have bench
markedstuff like this in the past and it is just not that big a deal as infrequently as you will
need to walk the scoreboard. The only real way to resolve this unambigously is to do benchmaarks on
implementations of the two designs. Yea, your design will be faster, but not significantly so I'll
wager.
> > The only reason we have locking at all is to prevent the 4 cases listed above from
> > colliding with each other. Even in the 4 cases above, the lock contention will be minimal
> > and the performance degradation minimal and perhaps not even measureable.
> >
> > A few other benefits to Pauls design:
> > 1. Eliminates the requirement for compiled in HARD_SERVER_LIMIT or HARD_THREAD_LIMIT.
>
> You still need to request a certain amount of shared memory when you
> start the parent process, and you will require HARD_SERVER_LIMIT and
> HARD_THREAD_LIMIT to know how much to request. Either that, or whenever
> you add a new child, you will need to allocate more shared memory, and
> somehow tell your child processes about it.
>
> > 2. You don't need to allocate child score if you don't care about mod_status (but it can
> > be added during a restart)
>
> You still need to request a certain amount of shared memory when you
> start the parent process, and you will require HARD_SERVER_LIMIT and
> HARD_THREAD_LIMIT to know how much to request. Either that, or whenever
> you restart, you will forget all information about your child processes.
>
Yadda yadda. Like I said in my first post, communicating the design via email is not easy. You can
eliminate HARD_SERVER_LIMIT and HARD_THREAD_LIMIT with Paul's design. Sorry I don;t care to explain
it via e-mail :-) Maybe Paul or Jeff will step up :-)
> > 3. If you choose to not enable mod_status, you will likely see a nice performance boost on
> > multi CPU machines because we are not invalidating a CPUs cache each time we touch a
> > worker score entry (which is on every request).
>
> This can be implemented just as well with the table implementation. The
> only thing you have to do is pad the scoreboard entry size to make it
> equal to one cache line.
>
And waste more storage? I am not a CPU designer...how big are cache lines on the different CPUs?
Wouldn;t padding the storage just cause you to load your cache with memory that is NOT USED at all?
I don't see the goodness in that. This is going way far out but with Paul's design, you could bind
a set of threads to a particular processor, then allocate your scoreboard storage for these threads
in contiguous storage. That way, all your threads on one processor is accessing storage that has a
high degree of locality. Good for the cache.
> > 4. Does not require any changes to the MPM. Each MPM can start threads according to its'
> > ThreadsPerChild setting w/o needing to pay attention to the scoreboard (I believe your
> > design required child processes to wait for worker score slots to become available before
> > it can start threads. This is imposing too much unnecessary work on the MPM.).
>
> You are ignoring one problem with the design. I have now posted about it
> three times, and nobody has told me how it will be fixed. You need to
> honor MaxClients. If I set MaxClients to 250, then I expect MaxClients to
> be 250, regardless of whether those clients are long-lived or not. If
> every child can always start threads according to ThreadsPerChild, you
> will be violating MaxClients on a heavily loaded server. This means that
> you are actually requiring MORE changes to the starting logic than the
> table implementation, because each child, will need to determine how many
> threads to start at any given time.
>
We can impose whatever disipline we like on when new child processes are allowed to start. Paul's
design doesn't directly specify or more importantly hinder implementing a disipline.
Bill
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by rb...@covalent.net.
> I don't completely understand you comment, but you are quite right that anytime the
> scoreboard needs to be walked, locking will be required. The scoreboard will -not- need to
> be walked except in these relatively rare cases:
>
> 1. During a graceful restart (manipulates elements in the scoreboard)
> 2. During MaxRequestPerChild triggered child shutdown (manipulates elements in the
> scoreboard)
> 3. Processing a mod_status request (walks the scoreboard)
> 4. Processing an SNMP request (walks the scoreboard)
>
> Requests that do not require a scoreboard walk or adding/deleting entries from the
> scoreboard (which is 99.999% or more of all requests; all requests except the ones above)
> will not require locking at all. Let me restate this... mainline requests will NOT be
> required to make any lock calls at all. Ever. Mainline requests can be served and their
> status tracked even when the scoreboard is locked, so this design does not suffer -any-
> performance penalties in the mainline case caused by locking.
You are looking at this from a Web server point of view. Don't. Look at
this from an ops point of view. How often is a large site going to query
the current connections or child status (for example) using SNMP? Both of
those require a walk of the scoreboard. If you ask for them once a
minute, you are going to seriously hinder the performance of your server.
The problem gets worse as the number of threads increases. How many other
SNMP requests carry the same penalty?
BTW, there are a few more that modules that make use of the scoreboard
information:
5. Whenever Apache::Scoreboard walks the scoreboard (this has been used
by perl modules to download the scoreboard once every 15 seconds to
generate graphs on another machine.)
6. Whenever Apache::VMonitor walks the scoreboard (Another graphicl
monitor of configurable refresh rate.
7. Whenever any other management module walks the scoreboard.
You can't guarantee how often a third party module will walk the
scoreboard, because you don't know everything about every third part
module. Since you don't know how often it will be walked, you don't know
how often the mutex will be locked.
> The only reason we have locking at all is to prevent the 4 cases listed above from
> colliding with each other. Even in the 4 cases above, the lock contention will be minimal
> and the performance degradation minimal and perhaps not even measureable.
>
> A few other benefits to Pauls design:
> 1. Eliminates the requirement for compiled in HARD_SERVER_LIMIT or HARD_THREAD_LIMIT.
You still need to request a certain amount of shared memory when you
start the parent process, and you will require HARD_SERVER_LIMIT and
HARD_THREAD_LIMIT to know how much to request. Either that, or whenever
you add a new child, you will need to allocate more shared memory, and
somehow tell your child processes about it.
> 2. You don't need to allocate child score if you don't care about mod_status (but it can
> be added during a restart)
You still need to request a certain amount of shared memory when you
start the parent process, and you will require HARD_SERVER_LIMIT and
HARD_THREAD_LIMIT to know how much to request. Either that, or whenever
you restart, you will forget all information about your child processes.
> 3. If you choose to not enable mod_status, you will likely see a nice performance boost on
> multi CPU machines because we are not invalidating a CPUs cache each time we touch a
> worker score entry (which is on every request).
This can be implemented just as well with the table implementation. The
only thing you have to do is pad the scoreboard entry size to make it
equal to one cache line.
> 4. Does not require any changes to the MPM. Each MPM can start threads according to its'
> ThreadsPerChild setting w/o needing to pay attention to the scoreboard (I believe your
> design required child processes to wait for worker score slots to become available before
> it can start threads. This is imposing too much unnecessary work on the MPM.).
You are ignoring one problem with the design. I have now posted about it
three times, and nobody has told me how it will be fixed. You need to
honor MaxClients. If I set MaxClients to 250, then I expect MaxClients to
be 250, regardless of whether those clients are long-lived or not. If
every child can always start threads according to ThreadsPerChild, you
will be violating MaxClients on a heavily loaded server. This means that
you are actually requiring MORE changes to the starting logic than the
table implementation, because each child, will need to determine how many
threads to start at any given time.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Bill Stoddard <bi...@wstoddard.com>.
>
> Bill,
>
> I will finish my code.
If you spend the time to write it, I will look at it.
> It will
> require locking anytime somebody wants to walk the scoreboard, which means
> that potentially the scoreboard will be run once a request. Think
> mod_status and/or mod_snmp.
I don't completely understand you comment, but you are quite right that anytime the
scoreboard needs to be walked, locking will be required. The scoreboard will -not- need to
be walked except in these relatively rare cases:
1. During a graceful restart (manipulates elements in the scoreboard)
2. During MaxRequestPerChild triggered child shutdown (manipulates elements in the
scoreboard)
3. Processing a mod_status request (walks the scoreboard)
4. Processing an SNMP request (walks the scoreboard)
Requests that do not require a scoreboard walk or adding/deleting entries from the
scoreboard (which is 99.999% or more of all requests; all requests except the ones above)
will not require locking at all. Let me restate this... mainline requests will NOT be
required to make any lock calls at all. Ever. Mainline requests can be served and their
status tracked even when the scoreboard is locked, so this design does not suffer -any-
performance penalties in the mainline case caused by locking.
The only reason we have locking at all is to prevent the 4 cases listed above from
colliding with each other. Even in the 4 cases above, the lock contention will be minimal
and the performance degradation minimal and perhaps not even measureable.
A few other benefits to Pauls design:
1. Eliminates the requirement for compiled in HARD_SERVER_LIMIT or HARD_THREAD_LIMIT.
2. You don't need to allocate child score if you don't care about mod_status (but it can
be added during a restart)
3. If you choose to not enable mod_status, you will likely see a nice performance boost on
multi CPU machines because we are not invalidating a CPUs cache each time we touch a
worker score entry (which is on every request).
4. Does not require any changes to the MPM. Each MPM can start threads according to its'
ThreadsPerChild setting w/o needing to pay attention to the scoreboard (I believe your
design required child processes to wait for worker score slots to become available before
it can start threads. This is imposing too much unnecessary work on the MPM.).
Bill
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Jeff Trawick <tr...@attglobal.net>.
rbb@covalent.net writes:
> I will finish my code. My design should take under 100 lines of changes,
> and requires no linked lists in shared memory. I have reviewed the design
> that Paul has detailed, and I disagree that it is correct. It will
> require locking anytime somebody wants to walk the scoreboard, which means
> that potentially the scoreboard will be run once a request. Think
> mod_status and/or mod_snmp.
It is hard to believe that there would be so many mod_status requests,
but maybe MPMs want to use scoreboard data to help manage things. I
dunno...
Whatever the reason, switching from a single global lock to a single
global read/write lock will help this scenario (constant scoreboard
reading) quite a bit.
If it turns out that processes/threads are coming/going constantly,
then something more complicated may be needed to make the scoreboard
go fast, but I think it can be hidden in a few routines which help
access the scoreboard.
--
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by rb...@covalent.net.
On Thu, 21 Jun 2001, Bill Stoddard wrote:
> > On 19 Jun 2001, Jeff Trawick wrote:
> >
> > > rbb@covalent.net writes:
> > >
> > > > The issue of how to deal with restart processes when one thread is stuck
> > > > serving a long-lived request. That was the major reason for not releasing
> > > > 2.0.18 as a beta AFAICR. I posted a design that solves the problem, but
> > > > then other issues got in the way of implementing it. I started the patch
> > > > last Friday, and if I can find a few hours to devote to it, I should have
> > > > a patch this week sometime.
> > >
> > > I guess you're talking about
> > >
> > > Message-ID: <Pi...@koj.rkbloom.net>
> >
> > Nope. I tend to design by evolution. This message discusses my first go
> > at this design. In later messages on this thread, I explained how it was
> > actually possible to have the temporary process become the real child
> > process.
> >
> > The short story is that when the child process begins to die, the parent
> > spawns a new child. That child starts to create threads, but only creates
> > as many as it can. When it can't create any more, it waits until it can
> > finish creating the rest of it's threads.
> >
> > The code is actually relatively simple, I just haven't had the time to
> > write the code up yet.
>
> Ryan,
> I suggest you hold off writing any code and see what Paul Reder is working on. Paul just
> spent some time reviewing his design with Jeff and I and all three of us are convinced his
> is the right solution. I am sure we can convince you too but communicating the nitty
> gritty details of the design via e-mail is not going to be simple. Can you fly out to
> Raleigh :-) ? Or perhaps get on a conference call (I can set up a call in number) so
> sother folks can partcipate?
Bill,
I will finish my code. My design should take under 100 lines of changes,
and requires no linked lists in shared memory. I have reviewed the design
that Paul has detailed, and I disagree that it is correct. It will
require locking anytime somebody wants to walk the scoreboard, which means
that potentially the scoreboard will be run once a request. Think
mod_status and/or mod_snmp.
I look forward to Paul's code, but I disagree that his design will work.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Bill Stoddard <bi...@wstoddard.com>.
> On 19 Jun 2001, Jeff Trawick wrote:
>
> > rbb@covalent.net writes:
> >
> > > The issue of how to deal with restart processes when one thread is stuck
> > > serving a long-lived request. That was the major reason for not releasing
> > > 2.0.18 as a beta AFAICR. I posted a design that solves the problem, but
> > > then other issues got in the way of implementing it. I started the patch
> > > last Friday, and if I can find a few hours to devote to it, I should have
> > > a patch this week sometime.
> >
> > I guess you're talking about
> >
> > Message-ID: <Pi...@koj.rkbloom.net>
>
> Nope. I tend to design by evolution. This message discusses my first go
> at this design. In later messages on this thread, I explained how it was
> actually possible to have the temporary process become the real child
> process.
>
> The short story is that when the child process begins to die, the parent
> spawns a new child. That child starts to create threads, but only creates
> as many as it can. When it can't create any more, it waits until it can
> finish creating the rest of it's threads.
>
> The code is actually relatively simple, I just haven't had the time to
> write the code up yet.
Ryan,
I suggest you hold off writing any code and see what Paul Reder is working on. Paul just
spent some time reviewing his design with Jeff and I and all three of us are convinced his
is the right solution. I am sure we can convince you too but communicating the nitty
gritty details of the design via e-mail is not going to be simple. Can you fly out to
Raleigh :-) ? Or perhaps get on a conference call (I can set up a call in number) so
sother folks can partcipate?
Bill
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by rb...@covalent.net.
On 19 Jun 2001, Jeff Trawick wrote:
> rbb@covalent.net writes:
>
> > The issue of how to deal with restart processes when one thread is stuck
> > serving a long-lived request. That was the major reason for not releasing
> > 2.0.18 as a beta AFAICR. I posted a design that solves the problem, but
> > then other issues got in the way of implementing it. I started the patch
> > last Friday, and if I can find a few hours to devote to it, I should have
> > a patch this week sometime.
>
> I guess you're talking about
>
> Message-ID: <Pi...@koj.rkbloom.net>
Nope. I tend to design by evolution. This message discusses my first go
at this design. In later messages on this thread, I explained how it was
actually possible to have the temporary process become the real child
process.
The short story is that when the child process begins to die, the parent
spawns a new child. That child starts to create threads, but only creates
as many as it can. When it can't create any more, it waits until it can
finish creating the rest of it's threads.
The code is actually relatively simple, I just haven't had the time to
write the code up yet.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Jeff Trawick <tr...@attglobal.net>.
rbb@covalent.net writes:
> The issue of how to deal with restart processes when one thread is stuck
> serving a long-lived request. That was the major reason for not releasing
> 2.0.18 as a beta AFAICR. I posted a design that solves the problem, but
> then other issues got in the way of implementing it. I started the patch
> last Friday, and if I can find a few hours to devote to it, I should have
> a patch this week sometime.
I guess you're talking about
Message-ID: <Pi...@koj.rkbloom.net>
--
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by "Paul J. Reder" <re...@raleigh.ibm.com>.
rbb@covalent.net wrote:
> The issue of how to deal with restart processes when one thread is stuck
> serving a long-lived request. That was the major reason for not releasing
> 2.0.18 as a beta AFAICR. I posted a design that solves the problem, but
> then other issues got in the way of implementing it. I started the patch
> last Friday, and if I can find a few hours to devote to it, I should have
> a patch this week sometime.
I believe this was the design where I was working to split the scoreboard and
use linked lists to maintain processes and workers vs. Ryan's design of
allowing finer granularity control of process ownership within a fixed block
of workers.
I got side tracked from my patch by my father-in-laws passing away, and some
mod_perl support work. I am now back on this full time. I should have my
patch by the end of this week or early next. Then we can all compare the
two patches.
I'm glad you are back working on this, I was afraid I would have to implement
both sides for arguments sake. :)
On your mark...
--
Paul J. Reder
-----------------------------------------------------------
"The strength of the Constitution lies entirely in the determination of each
citizen to defend it. Only if every single citizen feels duty bound to do
his share in this defense are the constitutional rights secure."
-- Albert Einstein
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by rb...@covalent.net.
> > -1. We already tagged 2.0.18, and rejected releasing it as a beta because
> > of the threaded MPM issues.
>
> I don't know who "we" is but I didn't like it because it didn't work
> nearly as as well as 2.0.16, regardless of the MPM :)
>
> > If we can wait about 1 week, I should have
> > the time to implement the design I posted back when we tagged
> > 2.0.19.
>
> Is that the new-and-improved pod+connect() design? If so, I have a
> patch on new-httpd (and private code that works a bit better). We
> still don't know how to portably kill threads non-gracefully though
> AFAIK.
>
> If it is not the new-and-improved pod+connect() design, what is it?
> (I've looked in new-httpd April-June; I don't doubt its there but a
> hint would be useful.)
The issue of how to deal with restart processes when one thread is stuck
serving a long-lived request. That was the major reason for not releasing
2.0.18 as a beta AFAICR. I posted a design that solves the problem, but
then other issues got in the way of implementing it. I started the patch
last Friday, and if I can find a few hours to devote to it, I should have
a patch this week sometime.
Ryan
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Jeff Trawick <tr...@attglobal.net>.
rbb@covalent.net writes:
> -1. We already tagged 2.0.18, and rejected releasing it as a beta because
> of the threaded MPM issues.
I don't know who "we" is but I didn't like it because it didn't work
nearly as as well as 2.0.16, regardless of the MPM :)
> If we can wait about 1 week, I should have
> the time to implement the design I posted back when we tagged
> 2.0.19.
Is that the new-and-improved pod+connect() design? If so, I have a
patch on new-httpd (and private code that works a bit better). We
still don't know how to portably kill threads non-gracefully though
AFAIK.
If it is not the new-and-improved pod+connect() design, what is it?
(I've looked in new-httpd April-June; I don't doubt its there but a
hint would be useful.)
--
Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
http://www.geocities.com/SiliconValley/Park/9289/
Born in Roswell... married an alien...
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by rb...@covalent.net.
-1. We already tagged 2.0.18, and rejected releasing it as a beta because
of the threaded MPM issues. If we can wait about 1 week, I should have
the time to implement the design I posted back when we tagged 2.0.19.
Then, we can have a chance of releasing as a beta. Otherwise, this tag
will just be another alpha, because the bug that stopped us from being
beta before still exists.
Ryan
On 19 Jun 2001, Jeff Trawick wrote:
> subject says it all...
>
> --
> Jeff Trawick | trawick@attglobal.net | PGP public key at web site:
> http://www.geocities.com/SiliconValley/Park/9289/
> Born in Roswell... married an alien...
>
>
_______________________________________________________________________________
Ryan Bloom rbb@apache.org
406 29th St.
San Francisco, CA 94131
-------------------------------------------------------------------------------
Re: plz vote on tagging current CVS as APACHE_2_0_19
Posted by Cliff Woolley <cl...@yahoo.com>.
On 19 Jun 2001, Jeff Trawick wrote:
> subject says it all...
+1
--------------------------------------------------------------
Cliff Woolley
cliffwoolley@yahoo.com
Charlottesville, VA