You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@httpd.apache.org by Dean Gaudet <dg...@arctic.org> on 1999/06/18 18:29:25 UTC
Re: work in progress: mpm-3.tar.gz (fwd)
On Fri, 18 Jun 1999, Zeev Suraski wrote:
>
> Is new-httpd moderated? Looks like every letter I send gets censored,
> which is really weird, since I see all sorts of crap on the list, whereas
> my posts are usually technical...
> Anyway, I'm sending this to you directly to ensure you get a chance to
> look at it...
It only allows subscribers to post... are you using the same subscription
addr as you are for sending messages to the list?
Brian Behlendorf deals with the non-subscriber posts, and he sometimes
lags by a few days...
> On Thu, 17 Jun 1999, Dean Gaudet wrote:
>
> > We will impose an additional restriction on modules -- if threads are in
> > use, they may not make any assumption that the same thread will be used to
> > process all phases of a request. Put another way -- thread local storage
> > is useless... and there will be no "thread_init_hook" function to tell
> > modules when threads have been created. This restriction is to give us
> > access to hybrid async/sync techniques. Modules needing information
> > persisting between request phases should use request-specific data
> > (or connection-specific data).
>
> Ouch, that's a very agressive restriction. It pretty much requires any
> module that uses local storage to be Apache specific, since it would have
> to save information in Apache's per-request or per-connection structure
> (not to mention it would have to pass pointers to these structures all
> over to any function that may require access to these globals, which is
> terrible).
Apache already passes around a request_rec pretty much everywhere... I
suppose we can implement apache-specific "thread local storage", which we
save and restore if we ever switch threads... but for first implementation
I'm really not going to worry about it...
> I really urge you to reconsider this. For PHP 4.0 (Zend), I've written a
> platform independent local storage resource manager (that works very well
> in the threaded ISAPI/IIS4 environment), but the whole approach will be
> renedered completely useless with such restrictions, since it's based on
> the thread id, and obviously expects that all steps and hooks are called
> within the same thread
You could base your storage off the conn_rec * instead of a thread_id...
Also, I was planning on building a connection_id which is a densely packed
small integer, because the scoreboard will need something like this.
The main case I'm considering this for is the handler phase. In general,
any request goes through a bunch of protocol stages and reaches the
handler, and from there it fits into a few small categories:
1. copy a file fd back to the client
2. copy a pipe/socket fd (from another process) back to the client
3. copy a mmapped region back to the client
4. copy a dynamically generated memory region back to the client
5. the handler writes stuff to a BUFF, and its sent to the client
1, 2, 3, and 4 are very simple cases where if the stuff to be sent doesn't
fit in the socket's send buffer, and we have to block the thread serving
the response. At this point we're potentially consuming an expensive
resource (a thread, stack, kernel memory for the thread, ...) just to wait
for the client.
Instead we can switch to an asynchronous behaviour, all of 1, 2, 3, 4 are
obvious -- the handler is essentially in a loop which we all know the
structure of, because the entire object is already generated somewhere.
The handler at this point sets up a special new record in the conn_rec,
and will return with a special return code indicating the switch to
asynchronous behaviour.
If the MPM supports async stuff, then it will release this thread from
serving the rest of this request... otherwise there'll be a library to
handle the "async" stuff synchronously. This async stuff will be
completed using select/poll/non-blocking i/o (or other similar variants,
there are several other faster methods on other platforms). We've freed
up the expensive resource:
- consume less CPU per client
- handle thousands upon thousands of long haul slow clients, because
they're just a conn_rec/request_rec at this point... no kernel stack, no
context switching... really, just minimal resource consumption
- do better on existing benchmarks, but more importantly, do better on
real world problems
At some point in the future the async stuff will finish, and a caller
supplied "completion" function will be called in another thread. This
gets us back out of the async core, and into protocol code, which will
"resume" the handler. This way the async core really has no knowledge of
the protocols involved -- and we can use this technique for any protocol
(and for variations on 1, 2, 3, 4)...
5. is the case for modules which don't want to take advantage of the async
features. But we can give them help by turning them into case 4 with more
features for BUFF... such as buffer up to 50k responses, and do the async
thing, otherwise do it synchronously.
> Frankly, with such restrictions, I'm not sure how
> something at the complexity of a scripting language can be implemented as
> an Apache module. If it can, it would have to tie the implementation to
> Apache very closely (PHP 4.0's implementation actually allows the same DLL
> or library to be used for the CGI version, Apache version and IIS version,
> with thin server-specific wrappers; with such restrictions, it doesn't
> seem possible).
>
> If I'm missing something obvious, please enlighten me :)
I think you're missing something slightly non-obvious :) Or I'm still
missing your point...
In essence I'm saying that the "thread local storage" is part of the
conn_rec (or request_rec, whichever is most convenient for you). All
entry points into your module include a request_rec structure -- you can
fetch a void * pointer from your request_data entry; you can store
whatever you need there.
So think of the thread as a resource which happens to execute your code
for a while, but think of the conn_rec/request_rec as your indication of
what is going on.
Apache *could* support "thread local storage", and if this really bothers
you then I'll encourage you to supply a patch. We can change the
requirement this way:
- MPMs which do not guarantee to use the same thread for all request
phases must save and restore the thread local storage across such changes
... but this is really hard to do portably -- unless we require all
modules to go through an apache, portable thread local storage API. Which
means I'd rather it wait for APR, or rather someone else take care of
it... 'cause the stuff which I'm working on is busy enough already :)
Dean
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Zeev Suraski <bo...@netvision.net.il>.
On Mon, 21 Jun 1999, Ben Laurie wrote:
> Zeev Suraski wrote:
> > It does affect us. We change the configuration directives for the current
> > thread and expect them to take affect when the PHP callback is called;
> > Under these restrictions, we can't really do it.
>
> Eeep! But that's just a dodgy thing to do - you should set stuff under
> the current request or connection, not change configuration directives.
> Clearly, regardless of what is going on with threads, this technique
> would be safe. Can't remember if Apache provides appropriate hooks but
> if it doesn't it should.
With Apache's directives, we actually do set the configuration directives
in the special pointer given to us by Apache's configuration mechanism.
But since PHP 4.0 has a completely generalized configuration mechanism (we
don't support a limited set of directives, but any possible directive), we
have to use dynamic allocation. And like any part of PHP, dynamic
allocation uses PHP's memory manager. This memory manager uses globals,
which in thread safe mode are in TLS; And there's your problem.
I'd suspend that discussion for now, though, until Dean's library is
mature enough to see whether we actually have a problem, and if we do end
up having a problem, see how we can solve it.
Zeev
--
-----------------------------------------------------
Zeev Suraski <ze...@zend.com>
For a PGP public key, finger bourbon@netvision.net.il
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Ben Laurie <be...@algroup.co.uk>.
Zeev Suraski wrote:
> It does affect us. We change the configuration directives for the current
> thread and expect them to take affect when the PHP callback is called;
> Under these restrictions, we can't really do it.
Eeep! But that's just a dodgy thing to do - you should set stuff under
the current request or connection, not change configuration directives.
Clearly, regardless of what is going on with threads, this technique
would be safe. Can't remember if Apache provides appropriate hooks but
if it doesn't it should.
Cheers,
Ben.
--
http://www.apache-ssl.org/ben.html
"My grandfather once told me that there are two kinds of people: those
who work and those who take the credit. He told me to try to be in the
first group; there was less competition there."
- Indira Gandhi
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Zeev Suraski <bo...@netvision.net.il>.
On Sat, 19 Jun 1999, Dean Gaudet wrote:
> On Sun, 20 Jun 1999, Zeev Suraski wrote:
>
> To be honest, this sounds like a bug in your code. Apache never
> guaranteed to call your code from the same thread -- in 1.x it built your
> structures in one process, and then executed them in another process...
> nevermind thread...
It's not a bug, it's just that this code never ran in a multithreaded
Apache environment. That sort of stuff is completely transparent in the
multiprocess environment in Apache 1.x.
> Yeah I know it's not the end of the world -- this is me saying "if you
> follow me down this path, we can kick ass on performance". But there are
> many many ways to handle this while leaving my restriction in the API...
> for example, you could choose to manage a pool of PHP threads within each
> apache process -- and your module would simply synchronize the apache
> thread with your threads. That's not as far-fetched as it sounds --
> that's really similar to the jserv model, where the processing occurs
> in another threaded process. You won't be the only people with this
> requirement, which is why a generic server-server protocol would be
> nice... and hey, an HTTP/1.1 proxy would solve that :)
>
> But I really want the php module to be able to take advantage of the async
> stuff -- because php is a real world application, unlike the benchmark
> crap that got me off my butt and started me working on this again.
> For example, I want to provide the primatives so that php can buffer
> responses up to 64k (configurable), and send those out with async support;
> for longer responses I'm happy consuming a thread forever.
>
> It'll require a little bit of work in the php response handler, and it'll
> mean that the log_request phase will almost certainly happen in a
> different thread. But I don't expect the changes to be difficult...
>
> Just so that we're all on the same page, this is what I'm aiming for:
>
> - new connection arrives at port 80
> - mpm does an accept and builds a conn_rec
> - thread 123 is chosen to process conn_rec
> - thread 123 goes through the apache API phases:
> post_read_request
> header_parser
> translate_handler
> ap_check_user_id
> auth_checker
> access_checker
> type_checker
> fixer_upper
> - thread 123 calls the response handlers
> - a response handler determines that the async engine can
> do the job
> - the stack unwinds from the apache core up into the mpm
> (thread 123 is no longer handling this connection, it may
> immediately begin servicing another connection)
> - the mpm sends the response asynchronously
> - at some point in the future the response has been sent
> - thread 234 is chosen to handle the "completion" event
> - thread 234 notices that the entire response has been sent, so
> it calls the logger (it may have been doing a range-request,
> and decided to send more ranges asynchronously)
> - thread 234 proceeds with keep-alive processing, perhaps this is
> a persistent connection
> - if it is, we could repeat the above process, and send another
> async response and end up back here again
>
> So we have asynchronous behaviour only in the handler -- and even then
> it only happens with modules that support async behaviour. The logger
> is the most obvious thing which is going to be called "out of thread".
>
> And since HTTP doesn't have any state from request to request within
> one connection, I don't think we need to debate about the fact that
> different requests on one connection may occur within different threads.
>
> That's what I'm aiming for. The only reason I stated that the thread
> may switch in any phase was to try to help folks like Tony who asked for
> async DNS...
>
> I think we're all on the same page now :) I'm not going to break your
> stuff in terrible ways, from what you've said your stuff will work just
> fine within this model. We just need to tweak our API a bit.
Well, if you're optimistic, I'm optimistic as well :) I'll wait a bit
until the model is more mature (and until I have more time) and see what
can be done to help PHP take advantage of this asynchornous behavior. If
it can be done, I don't mind giving users better performance in Apache
than they'd get from IIS (to say the least :)
Zeev
--
-----------------------------------------------------
Zeev Suraski <ze...@zend.com>
For a PGP public key, finger bourbon@netvision.net.il
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by "Michael H. Voase" <mv...@midcoast.com.au>.
Dean Gaudet wrote:
>
>
> I also had another big goal -- I wanted to show a roadmap which has lots
> of open projects still left in it... so that we can take advantage of some
> parallel development. So yup, MPM lacks all the stuff you mentioned...
> for a good reason... it was a design which we've been solidifying over an
> 18 month period and then I spent a day or so hacking to lay out the
> framework... and if I've convinced everyone it's going in the right
> direction multiple of us can spend time polishing up the rough edges.
>
> Yeah ok I'm procrastinating from cleaning up the mess I've made of buff.c.
>
> Dean
>
I would like to comment that Dean has missed another
point that wasnt brought up . The reason why MPM can satisfy
the needs of many different models is that it is now generalised
to the point that any application can be supported with this .
Quite aside from a http server , with a little polishing and
extraction of http dependent code ,this framework could IMHO
support just about any application you can throw at it .
So the result is that you now have a bootstrap / support
architecture that is far, far more than just a humble web server .
Cheers Mik Voase.
--
----------------------------------------------------------------------------
/~\ /~\ CASTLE INDUSTRIES PTY. LTD.
| |_____| | Incorporated 1969. in N.S.W., Australia
| | Phone +612 6567 1227 Fax +612 6567 1449
| /~\ | Web http://www.midcoast.com.au/~mvoase
| [ ] | Michael H. Voase. Director.
~~~~~~~~~~~~~~ Castle Industries - Industrial Strength
Solutions.
----------------------------------------------------------------------------
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Dean Gaudet <dg...@arctic.org>.
Dude, as I mentioned in my first post, grep for the TODO comments. I
already listed all of this.
But my reasons for starting again with apache-1.3 go like so:
- I know apache-1.3 works, I don't know your code works; I knew I was
going to break everything and rather wanted to start with a code base I
knew worked.
- I don't believe it is wise to mix the prefork and mpmt models as you
guys did in apache-apr -- this does nothing to help with reducing the
number of #defines; and does nothing to help people who really do have
older unixes... my method gives them a much easier "port", they've got
something which looks like apache-1.3 with a little more use of
non-blocking sockets. No other new tricks are in use.
- not everything is going to look like an fdqueue. In Zach Brown's model
fds are delivered via real-time queued signals -- a lightweight message
passing protocol in essence. Similarly in my ASH model. Your accept
queue stuff looked impossibly hard to integrate my code in... I tried.
- you guys made little or no attempt to abstract and define the
method/ordering of configuring the server; in my opinion we can't even
begin to talk about different mpms without specifying the order in which
modules will be invoked during configuration and run-time. Module
authors need to know their execution environment.
- I wanted to lay the groundwork for abstracting the protocol from the
MPM. It's a simple change to MPM now to call (*protocol_handler)()
instead of ap_process_connection. (This is required for folks wanting
to play with the transport layer... such as mux and ssl.)
- The scoreboard can't just be hidden in another file, it needs a complete
redesign. I decided to just lose the functionality for now until we can
figure out all the pieces needed to put it back together.
In short, I tried to work with your apache-apr/pthreads code for the ASH
stuff I was doing. I couldn't. This is why I stepped back and tried to
figure out why, and that's what resulted in the MPM code.
There's nothing stopping you guys from taking the apache-apr/pthreads
stuff and bringing it over to the MPM model -- all the hooks are there...
and it's totally welcome! And it looks like manoj has done that :)
In short, I don't think we've lost anything. Every one of these attempts
-- RST's age old threaded work; my first pthread port; my NSPR port; the
manoj/ryan pthread port -- has taught us something. You guys in
particular gave me much more insight into the gnarly problems of
restart/shutdown. MPM is just my view on how it all fits together.
I look forward to all the abstractions you list, because I'll need those
pieces as well for my work. You can see I started -- with splitting out
the "unix daemon" code -- the detach/uid/gid stuff. Yup Listen needs to
happen soon. Ralf's MM stuff should go in to satisfy the unix
multiprocess shared mem/semaphore needs.
At that point mpm_prefork will amount to a dynamic process pool management
module... and hey maybe even that might want to be abstracted. But I'm
not sure I need it -- my model will have a fixed number of processes; and
I don't want any signals used. But as I said back up there, I want
mpm_prefork as close to apache-1.3 as possible because then we have
something we know works (and that should keep the non-modern unix folks
appeased :)
I also had another big goal -- I wanted to show a roadmap which has lots
of open projects still left in it... so that we can take advantage of some
parallel development. So yup, MPM lacks all the stuff you mentioned...
for a good reason... it was a design which we've been solidifying over an
18 month period and then I spent a day or so hacking to lay out the
framework... and if I've convinced everyone it's going in the right
direction multiple of us can spend time polishing up the rough edges.
Yeah ok I'm procrastinating from cleaning up the mess I've made of buff.c.
Dean
On Tue, 22 Jun 1999, Ryan Bloom wrote:
>
> I have finally had a chance to review the mpm code. I have a few
> questions/concerens about it.
>
> If the goal was to clean up http_main, I think the code should have been
> split up into multiple files, not moved as one big chunk to a new file.
> The sections I am thinking about are
>
> 1) The accept_mutex logic. This stuff is large and was already moved out
> of http_main in the hybrid apache work. Why leave it with the mpm stuff
> now? It seems to me that regardless of the mpm, we will need a method to
> lock the acceptors. Whether that be cross-process or cross-thread is
> another question, but that one is best answered with a #define or a flag
> to a common function, not re-writing this section of code for each mpm.
>
> 2) The scoreboard logic. Again, this was moved out in the hybrid apache
> work, because we wanted to make http_main smaller. Are you saying that we
> will no longer need a segment of shared memory in ALL mpm's? I don't
> believe it. Yes, a lot of the scoreboard information is there for users
> consumption only (mod_status), and the non-user information could easily
> be done using other IPC mechanisms, but mod_status is incredibly useful,
> and having to rebuild that information each time somebody asks for it
> seems like a wasted expense.
>
> 3) all the *_listeners functions. ALL mpm's need some way to setup a
> list of the listeners we care about. The only method that doesn't, is one
> where each socket has it's own thread listenening on it. We discussed
> moving this logic out of http_main and into our accept-loop abstraction in
> the hybrid work, but decided against it, because we couldn't find a model
> that didn't require this logic.
>
> This code basically has each mpm duplicating ALOT of code, IMHO. I will
> agree that taking the child management logic out of http_main, and either
> modularizing it, or just abstracting it based on platform is a good idea.
>
> I get the feeling however, that a bunch of code was ripped out of
> http_main and moved to this new mpm file, and that bothers me, because it
> doesn't clean up the code, it just hides it in another file.
>
> I believe the correct way to do this, is to follow the path we took in the
> hybrid work. Move all the code that is related to each other to separate
> files, so that each mpm can use those functions easily instead of
> re-inventing the wheel each time (More of this is needed in the hybrid
> work). I think if that is done, IMHO, we will find that the accept-loop
> abstraction that Bill started is very close to what is needed.
>
> :)
>
> Just my $0.02
>
> Ryan
>
> _______________________________________________________________________
> Ryan Bloom rbb@raleigh.ibm.com
> 4205 S Miami Blvd
> RTP, NC 27709 It's a beautiful sight to see good dancers
> doing simple steps. It's a painful sight to
> see beginners doing complicated patterns.
>
>
>
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Ryan Bloom <rb...@raleigh.ibm.com>.
I have finally had a chance to review the mpm code. I have a few
questions/concerens about it.
If the goal was to clean up http_main, I think the code should have been
split up into multiple files, not moved as one big chunk to a new file.
The sections I am thinking about are
1) The accept_mutex logic. This stuff is large and was already moved out
of http_main in the hybrid apache work. Why leave it with the mpm stuff
now? It seems to me that regardless of the mpm, we will need a method to
lock the acceptors. Whether that be cross-process or cross-thread is
another question, but that one is best answered with a #define or a flag
to a common function, not re-writing this section of code for each mpm.
2) The scoreboard logic. Again, this was moved out in the hybrid apache
work, because we wanted to make http_main smaller. Are you saying that we
will no longer need a segment of shared memory in ALL mpm's? I don't
believe it. Yes, a lot of the scoreboard information is there for users
consumption only (mod_status), and the non-user information could easily
be done using other IPC mechanisms, but mod_status is incredibly useful,
and having to rebuild that information each time somebody asks for it
seems like a wasted expense.
3) all the *_listeners functions. ALL mpm's need some way to setup a
list of the listeners we care about. The only method that doesn't, is one
where each socket has it's own thread listenening on it. We discussed
moving this logic out of http_main and into our accept-loop abstraction in
the hybrid work, but decided against it, because we couldn't find a model
that didn't require this logic.
This code basically has each mpm duplicating ALOT of code, IMHO. I will
agree that taking the child management logic out of http_main, and either
modularizing it, or just abstracting it based on platform is a good idea.
I get the feeling however, that a bunch of code was ripped out of
http_main and moved to this new mpm file, and that bothers me, because it
doesn't clean up the code, it just hides it in another file.
I believe the correct way to do this, is to follow the path we took in the
hybrid work. Move all the code that is related to each other to separate
files, so that each mpm can use those functions easily instead of
re-inventing the wheel each time (More of this is needed in the hybrid
work). I think if that is done, IMHO, we will find that the accept-loop
abstraction that Bill started is very close to what is needed.
:)
Just my $0.02
Ryan
_______________________________________________________________________
Ryan Bloom rbb@raleigh.ibm.com
4205 S Miami Blvd
RTP, NC 27709 It's a beautiful sight to see good dancers
doing simple steps. It's a painful sight to
see beginners doing complicated patterns.
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Dean Gaudet <dg...@arctic.org>.
On Sun, 20 Jun 1999, Zeev Suraski wrote:
> PHP interfaces with Apache in two stages - configuration directive parsing
> and the actual scripting execution callback. Lets assume these two are
> called in separate threads.
This isn't a new assumption, this is how it has always been -- config
parsing at config time is in a completely different context from how it is
used at run-time. There's a confusion right now about the two "faces" of
configuration:
- one face is read-only, essentially everything read from the config file
is used in a read-only manner at run-time. This stuff can be shared
without synchronization... the vast majority of config stuff fits in
this category (all of apache's core modules fit entirely in this
category)
- the other face is shared, dynamic information which results as a
function of the configuration. For example, a dynamic cache of files,
or a pool of mysql db connections. This stuff can be shared only if
synchronization occurs.
In some sense it would be valid to mprotect(PROT_READ) all the pages
used in pconf -- any dynamic data you need should be allocated in
pchild during child_init. But, the second face is new, it's only really
interesting when you've got a threaded server.
> PHP's configuration parsing mechanism, in this case, reading from
> .htaccess/.conf files using registered Apache callbacks, stores those
> directives in PHP dynamic structures that use PHP's memory manager. PHP's
> memory manager, in turn, uses TLS.
>
> Since the data structures that were allocated in the configuration parsing
> phase are used on another phase (script execution) - their deallocation may
> occur in a different thread from that in which they were
> allocated. Undoubtfully, that would cause problems since the TLS structure
> used by the memory manager for deallocation won't be the right structure.
To be honest, this sounds like a bug in your code. Apache never
guaranteed to call your code from the same thread -- in 1.x it built your
structures in one process, and then executed them in another process...
nevermind thread...
> Now, I understand your point more or less. What I'm saying is that such a
> restriction is very aggressive and somewhat unnatural, and is likely to
> cause a lot of headache to module developers. In PHP, it would be possible
> to get around it by moving the TLS code into a separate DLL/.so that would
> be server dependent, and would be implemented as standard TLS for all web
> servers except for apache, and implement it as request-local-storage in
> Apache. It would be a pain, though, so if you could keep a background task
> to think about how to allow modules to ask for all of their execution path
> to occur within the same thread (without preventing the whole server from
> using asynchronous operations), it would be great.
Yeah I know it's not the end of the world -- this is me saying "if you
follow me down this path, we can kick ass on performance". But there are
many many ways to handle this while leaving my restriction in the API...
for example, you could choose to manage a pool of PHP threads within each
apache process -- and your module would simply synchronize the apache
thread with your threads. That's not as far-fetched as it sounds --
that's really similar to the jserv model, where the processing occurs
in another threaded process. You won't be the only people with this
requirement, which is why a generic server-server protocol would be
nice... and hey, an HTTP/1.1 proxy would solve that :)
But I really want the php module to be able to take advantage of the async
stuff -- because php is a real world application, unlike the benchmark
crap that got me off my butt and started me working on this again.
For example, I want to provide the primatives so that php can buffer
responses up to 64k (configurable), and send those out with async support;
for longer responses I'm happy consuming a thread forever.
It'll require a little bit of work in the php response handler, and it'll
mean that the log_request phase will almost certainly happen in a
different thread. But I don't expect the changes to be difficult...
Just so that we're all on the same page, this is what I'm aiming for:
- new connection arrives at port 80
- mpm does an accept and builds a conn_rec
- thread 123 is chosen to process conn_rec
- thread 123 goes through the apache API phases:
post_read_request
header_parser
translate_handler
ap_check_user_id
auth_checker
access_checker
type_checker
fixer_upper
- thread 123 calls the response handlers
- a response handler determines that the async engine can
do the job
- the stack unwinds from the apache core up into the mpm
(thread 123 is no longer handling this connection, it may
immediately begin servicing another connection)
- the mpm sends the response asynchronously
- at some point in the future the response has been sent
- thread 234 is chosen to handle the "completion" event
- thread 234 notices that the entire response has been sent, so
it calls the logger (it may have been doing a range-request,
and decided to send more ranges asynchronously)
- thread 234 proceeds with keep-alive processing, perhaps this is
a persistent connection
- if it is, we could repeat the above process, and send another
async response and end up back here again
So we have asynchronous behaviour only in the handler -- and even then
it only happens with modules that support async behaviour. The logger
is the most obvious thing which is going to be called "out of thread".
And since HTTP doesn't have any state from request to request within
one connection, I don't think we need to debate about the fact that
different requests on one connection may occur within different threads.
That's what I'm aiming for. The only reason I stated that the thread
may switch in any phase was to try to help folks like Tony who asked for
async DNS...
I think we're all on the same page now :) I'm not going to break your
stuff in terrible ways, from what you've said your stuff will work just
fine within this model. We just need to tweak our API a bit.
Dean
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Zeev Suraski <ze...@zend.com>.
At 00:13 20/06/99 , Dean Gaudet wrote:
>On Sat, 19 Jun 1999, Rasmus Lerdorf wrote:
>
> > > > Hrm, when I first read Dean's description, I thought about this
> too, but
> > > > figured we wouldn't have a problem because we really just have a single
> > > > entry point. Our only other hook is the config stuff and we don't do
> > > > anything that I think would require it to be done from the same thread.
> > > > However, I could see this restriction being a pain if we wanted to
> be able
> > > > to enter PHP from more that just the content-handling request phase.
> > > > I would think this would be a big problem for mod_perl stuff.
> > >
> > > It does affect us. We change the configuration directives for the
> current
> > > thread and expect them to take affect when the PHP callback is called;
> > > Under these restrictions, we can't really do it.
> >
> > Hrm.. True, and we store it TLS-style keyed on the thread id. How the
> > heck is it useful to have a module's init handler be in a separate thread
> > from the actual invocation later on? Well, ok, the answer was already
> > given, and that is to use Apache's memory pool mechanism to store things.
> > We can abstract that away in SAPI, I guess, but 3rd party things that
> > don't know about Apache's memory management mechanism and do TLS-style
> > things to be thread-safe are going to be confused. So yes, I agree with
> > you, it would be a PITA for us.
>
>I really don't understand the difficulty.
>
>I return to my earlier example:
>
> FILE *f = fopen("foo", "r");
>
>You may now use f in any thread. There's no requirement that f be used
>only in the thread which created it. I don't see why such a requirement
>is necessary! If libc didn't keep everything it needed to know in the
>structure pointed to by f, why does it have a structure to begin with?
>
>Now, you're complaining about TLS. Apache knows nothing about TLS.
>It won't know anything about it until that mythical portable run-time
>appears. It's not like I'm breaking anything, there's nothing to break.
Other modules use TLS though, regardless of Apache, and expect some sort of
a continuous execution model, where each part of the execution phase occurs
within the same thread. For example, the PHP MySQL module defines a global
structure where it stores information that's used in all of its
functions. A pointer to that structure isn't passed between all of the
functions, since some of them are called from PHP's engine, that knows
nothing about the MySQL module's globals.
In a multithreaded environment, obviously, you can't really have a global
structure like that, but you need to pick some TLS-style alternative. So,
when you compile PHP 4.0 as thread safe, it accesses that structure using
the thread safe resource manager. Notice that the MySQL module doesn't
know and doesn't need to know that it's running within Apache - all of its
interface with the outside world is done through PHP's API functions, which
may wrap around Apache, ISAPI or plain old libc for the CGI binary
version. It doesn't know about request_rec or any server-side
structures. The only thing it expects is that the execution model is
linear, and occurs within the same thread.
PHP interfaces with Apache in two stages - configuration directive parsing
and the actual scripting execution callback. Lets assume these two are
called in separate threads.
PHP's configuration parsing mechanism, in this case, reading from
.htaccess/.conf files using registered Apache callbacks, stores those
directives in PHP dynamic structures that use PHP's memory manager. PHP's
memory manager, in turn, uses TLS.
Since the data structures that were allocated in the configuration parsing
phase are used on another phase (script execution) - their deallocation may
occur in a different thread from that in which they were
allocated. Undoubtfully, that would cause problems since the TLS structure
used by the memory manager for deallocation won't be the right structure.
Now, I understand your point more or less. What I'm saying is that such a
restriction is very aggressive and somewhat unnatural, and is likely to
cause a lot of headache to module developers. In PHP, it would be possible
to get around it by moving the TLS code into a separate DLL/.so that would
be server dependent, and would be implemented as standard TLS for all web
servers except for apache, and implement it as request-local-storage in
Apache. It would be a pain, though, so if you could keep a background task
to think about how to allow modules to ask for all of their execution path
to occur within the same thread (without preventing the whole server from
using asynchronous operations), it would be great.
Zeev
--
Zeev Suraski <ze...@zend.com> http://www.zend.com/
For a PGP public key, finger bourbon@netvision.net.il
Hybrid server ported to MPM
Posted by Manoj Kasichainula <ma...@raleigh.ibm.com>.
On Sat, Jun 19, 1999 at 03:37:48PM -0700, Dean Gaudet wrote:
>
>
> On Sat, 19 Jun 1999, Manoj Kasichainula wrote:
> > Which I'm hoping to have ready soon. I'm attempting to port the
> > http_main.c from the apache-apr repository to Dean's framework.
> > (mpm_mpmt.c?).
Here's a patch:
http://www.io.com/~manoj/patch/mpm_mpmt-1.patch.gz
Other than mod_mpmt.c, the other added files are the same as the
apache-apr versions except for a couple of search-and-replaces for the
unixd stuff. The USE_ACCEPT_QUEUE accept model doesn't work, but
adding fdqueue.c might be enough for that. If this model is kept and
used, maybe (or maybe not) the accept queue stuff should be split into
its own MPM.
Right now mpm_mpmt.c #includes the other .c files it needs. Once the
MPM configuration stuff is hammered down more, this can be fixed.
Any objections before commit?
> mpm_pthread_mpmt ? dunno, your call. There could be a mpmt model on
> windows now that Bill seems to have figured out how to force windows to
> allow multiple processes to accept() on the same socket.
It's still called mpm_mpmt.c, but s/mpmt/pthread_mpmt/g on the patch
would fix that quickly enough. pthread_mpmt sounds good to me. I
don't care too much what it's called; mpm_pmpmpmpm would be fine too.
--
Manoj Kasichainula - manojk@raleigh.ibm.com
IBM, Apache Development
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Bill Stoddard <st...@raleigh.ibm.com>.
Dean Gaudet wrote:
> mpm_pthread_mpmt ? dunno, your call. There could be a mpmt model on
> windows now that Bill seems to have figured out how to force windows to
> allow multiple processes to accept() on the same socket.
Wow, take a few vacation days and the world changes while I'm away :-)
Cool
work Dean! Here is what I'm working on (all Win32, unfortunately part
time):
1. Getting the Win32 socket sharing code working on 95/98 (for 1.3.7)
Almost there...
2. Supporting 16-bit CGIs on Win32 (1.3.7)
This has turned into a compulsion, really. Who really cares if
Apache
supports 16 bit CGIs? I think I have a simple two line hack that will
work...
And we can close out those pesky PRs.
3. Win32 MPM (2.0)
Should be easy given my socket sharing work.
Looking further out... (next couple of months, depending on my schedule)
4. Non-blocking, non-buffering Win32 CGIs (2.0)
5. Exploiting async I/O, completion ports and TransmitFile (2.0)
6. Caching
Win32 specific file handle cache to use with TransmitFile.
Bill
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Dean Gaudet <dg...@arctic.org>.
On Sat, 19 Jun 1999, Manoj Kasichainula wrote:
> On Sat, Jun 19, 1999 at 03:13:18PM -0700, Dean Gaudet wrote:
> > I'm not interested in totally broken 3rd party libraries. If you have
> > to deal with these then use an MPM that uses the same thread for all
> > phases.
>
> Which I'm hoping to have ready soon. I'm attempting to port the
> http_main.c from the apache-apr repository to Dean's framework.
> (mpm_mpmt.c?).
mpm_pthread_mpmt ? dunno, your call. There could be a mpmt model on
windows now that Bill seems to have figured out how to force windows to
allow multiple processes to accept() on the same socket.
Dean
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Manoj Kasichainula <ma...@io.com>.
On Sat, Jun 19, 1999 at 03:13:18PM -0700, Dean Gaudet wrote:
> I'm not interested in totally broken 3rd party libraries. If you have
> to deal with these then use an MPM that uses the same thread for all
> phases.
Which I'm hoping to have ready soon. I'm attempting to port the
http_main.c from the apache-apr repository to Dean's framework.
(mpm_mpmt.c?).
--
Manoj Kasichainula - manojk at io dot com - http://www.io.com/~manojk/
"Everything sucks." - Butthead
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Dean Gaudet <dg...@arctic.org>.
On Sat, 19 Jun 1999, Dean Gaudet wrote:
> FILE *f = fopen("foo", "r");
>
> You may now use f in any thread. There's no requirement that f be used
> only in the thread which created it. I don't see why such a requirement
> is necessary! If libc didn't keep everything it needed to know in the
> structure pointed to by f, why does it have a structure to begin with?
BTW -- I'm not saying anything about f being used simultaneously by
multiple threads. I'm suggesting that f can be used by multiple threads
provided they all take care to avoid using it at the same time. I think
it's totally reasonable for a library to provide an interface which is
thread-safe, but doesn't provide any of its own locking. In fact, that's
totally the right way to do things... because frequently locking isn't
even required.
zlib for example, hangs all of its data off a z_stream. If I want
multiple threads to write through a single z_stream then I have to provide
my own locking... but I know that as long as I'm serializing access to
each z_stream I'm still free to use it from any thread.
In general though, I probably don't need multiple threads writing through
one z_stream... what I probably have are multiple z_streams, and multiple
threads... and how I choose to multiplex the two is up to me. The library
doesn't force me into any model.
That's a good library. It would have to bend over backwards to be stupid
really...
Can you give me an example of a "thread-safe" library which just doesn't
work this way?
Note that I'm not changing the thread running the code in the middle of
anything -- it's at very defined points in the execution, and those points
are definately not in the middle of the library... the stack trace
looks roughly like this:
#0 third_party_library()
#1 third_party_module()
#2 apache_core_code()
#3 mpm_core_code()
#4 http_main_code()
thread switches can only occur at the #3 level... which means the stack
has to unwind, the #0, #1, and #2 functions have to return before #3
can choose to use a different thread... so later it may be:
#0 third_party_library_2()
#1 third_party_module_2()
#2 apache_core_code_2()
#3 mpm_core_code()
#4 http_main_code()
In a different thread, but all the earlier stuff has returned... there's
no longjmping...
Does it make sense?
It actually may never happen either -- the only phase I want it for
is the handler phase, and the subsequent log_handler. But Tony asked
about DNS, and to have async support for DNS means that this unwinding
and switching can happen in any phase. But if it makes you happier I
can just say that the log_transaction phase may occur in a different
thread from the rest of the request... but I'm loathe to do that until
someone actually comes up with a library which requires this.
Dean
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Dean Gaudet <dg...@arctic.org>.
On Sat, 19 Jun 1999, Rasmus Lerdorf wrote:
> > > Hrm, when I first read Dean's description, I thought about this too, but
> > > figured we wouldn't have a problem because we really just have a single
> > > entry point. Our only other hook is the config stuff and we don't do
> > > anything that I think would require it to be done from the same thread.
> > > However, I could see this restriction being a pain if we wanted to be able
> > > to enter PHP from more that just the content-handling request phase.
> > > I would think this would be a big problem for mod_perl stuff.
> >
> > It does affect us. We change the configuration directives for the current
> > thread and expect them to take affect when the PHP callback is called;
> > Under these restrictions, we can't really do it.
>
> Hrm.. True, and we store it TLS-style keyed on the thread id. How the
> heck is it useful to have a module's init handler be in a separate thread
> from the actual invocation later on? Well, ok, the answer was already
> given, and that is to use Apache's memory pool mechanism to store things.
> We can abstract that away in SAPI, I guess, but 3rd party things that
> don't know about Apache's memory management mechanism and do TLS-style
> things to be thread-safe are going to be confused. So yes, I agree with
> you, it would be a PITA for us.
I really don't understand the difficulty.
I return to my earlier example:
FILE *f = fopen("foo", "r");
You may now use f in any thread. There's no requirement that f be used
only in the thread which created it. I don't see why such a requirement
is necessary! If libc didn't keep everything it needed to know in the
structure pointed to by f, why does it have a structure to begin with?
Now, you're complaining about TLS. Apache knows nothing about TLS.
It won't know anything about it until that mythical portable run-time
appears. It's not like I'm breaking anything, there's nothing to break.
As far as I understand your point, you're saying that there may exist
some library such that when you do:
FILE *f = fopen("foo", "r");
it stuffs something into the TLS for the thread calling fopen.
Guys, that's dumb. That's more complicated than it needs to be.
Now, if your complaint is:
the dir_create/dir_merge methods don't include a conn_rec or
request_rec when used at run-time
I'm with you all the way. What you're saying is that for the
dir_create/dir_merge functions, when used at run time, you have no way
of knowing the context in which they're being used. That we can fix.
Suppose that's fixed. We can make this guarantee:
whenever apache calls a module at run-time, it guarantees to pass
it a conn_rec * in some form or another
Ok, so apache enters your module at entry point foo. You do this:
foo(conn_rec *c)
{
void *tls;
tls = find_my_tls(c);
set the tls for whatever thread library you're using = tls;
do whatever foo() would have done in apache 1.x
}
Your module really has no excuse, we're giving you a piece of context
(the conn_rec) which we guarantee to be the same for every phase of
the request. Why do you need the thread to be the same?
I'm not interested in totally broken 3rd party libraries. If you have
to deal with these then use an MPM that uses the same thread for all
phases.
Dean
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Rasmus Lerdorf <ra...@lerdorf.on.ca>.
> > Hrm, when I first read Dean's description, I thought about this too, but
> > figured we wouldn't have a problem because we really just have a single
> > entry point. Our only other hook is the config stuff and we don't do
> > anything that I think would require it to be done from the same thread.
> > However, I could see this restriction being a pain if we wanted to be able
> > to enter PHP from more that just the content-handling request phase.
> > I would think this would be a big problem for mod_perl stuff.
>
> It does affect us. We change the configuration directives for the current
> thread and expect them to take affect when the PHP callback is called;
> Under these restrictions, we can't really do it.
Hrm.. True, and we store it TLS-style keyed on the thread id. How the
heck is it useful to have a module's init handler be in a separate thread
from the actual invocation later on? Well, ok, the answer was already
given, and that is to use Apache's memory pool mechanism to store things.
We can abstract that away in SAPI, I guess, but 3rd party things that
don't know about Apache's memory management mechanism and do TLS-style
things to be thread-safe are going to be confused. So yes, I agree with
you, it would be a PITA for us.
-Rasmus
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Zeev Suraski <bo...@netvision.net.il>.
On Sat, 19 Jun 1999, Rasmus Lerdorf wrote:
> > And my general hunch about it is still negative, in the sense that I think
> > we'd bump into many problems, especially in modules that use 3rd party
> > libraries that are also thread safe (for example, say you initialize the
> > MySQL client library in one phase, and then use it in another - there's a
> > good chance there's thread-specific initialization in that library, and it
> > won't happen in such a case; In MySQL's case, I don't think we have a
> > problem, but I'm almost sure there'd be others in which we would). And
> > there's also ISAPI.
> >
> > How feasible would it be to be able to mark a module as one that wants all
> > of its request steps to be performed within the same thread?
>
> Hrm, when I first read Dean's description, I thought about this too, but
> figured we wouldn't have a problem because we really just have a single
> entry point. Our only other hook is the config stuff and we don't do
> anything that I think would require it to be done from the same thread.
> However, I could see this restriction being a pain if we wanted to be able
> to enter PHP from more that just the content-handling request phase.
> I would think this would be a big problem for mod_perl stuff.
It does affect us. We change the configuration directives for the current
thread and expect them to take affect when the PHP callback is called;
Under these restrictions, we can't really do it.
Zeev
--
-----------------------------------------------------
Zeev Suraski <ze...@zend.com>
For a PGP public key, finger bourbon@netvision.net.il
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Rasmus Lerdorf <ra...@lerdorf.on.ca>.
> And my general hunch about it is still negative, in the sense that I think
> we'd bump into many problems, especially in modules that use 3rd party
> libraries that are also thread safe (for example, say you initialize the
> MySQL client library in one phase, and then use it in another - there's a
> good chance there's thread-specific initialization in that library, and it
> won't happen in such a case; In MySQL's case, I don't think we have a
> problem, but I'm almost sure there'd be others in which we would). And
> there's also ISAPI.
>
> How feasible would it be to be able to mark a module as one that wants all
> of its request steps to be performed within the same thread?
Hrm, when I first read Dean's description, I thought about this too, but
figured we wouldn't have a problem because we really just have a single
entry point. Our only other hook is the config stuff and we don't do
anything that I think would require it to be done from the same thread.
However, I could see this restriction being a pain if we wanted to be able
to enter PHP from more that just the content-handling request phase.
I would think this would be a big problem for mod_perl stuff.
-Rasmus
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Dean Gaudet <dg...@arctic.org>.
On Fri, 18 Jun 1999, Zeev Suraski wrote:
> Correct me if I'm wrong, but you can't obtain the conn_rec from anywhere
> in the code by calling a simple get_conn_rec_ptr()..?
Everywhere which apache calls your module, it passes a request_rec, or a
conn_rec, or some other token by which you can figure out your context...
beyond that, it's really up to you I think. If you need a simple
get_conn_rec_ptr() routine, you can implement one rather easily, using
whatever portable, non-portable, etc. method you need.
Yes, you won't be the only one needing this... but my point is more along
these lines: it's not a difficult problem to solve, and I've got way more
difficult problems to solve at the moment.
> I guess it'll be
> possible to implement such a function (if you keep some mapping between
> thread id's and their current corresponding conn_rec's, even though I'm
> not sure if you need it for other purposes or whether it would be just
> pure added overhead).
> Without such a function, you still have to pass the conn_rec pointer
> around everywhere, to any function that may possibly need access to a
> global per-thread variable, which is exactly what TLS comes to solve...
Yeah we're saying the same thing -- I'm saying I'm not worrying about it
because it's solveable. TLS is really just a void * that magically
changes on context switches.
> Well, I guess it all depends on what kind of stages we're talking about.
> If the model is remotely similar to Apache 1.x, then we're not in too much
The model is apache 1.x with a few extra config-time hooks at the moment.
> Well then, I wasn't missing that, it's just not enough for our purposes :)
I still think it's enough, so maybe I'm not explaining myself well enough?
> One question I raised is whether I can get to that resource from anywhere
> in the code even though it wasn't passed on to me through the stack.
Yeah -- as I said above, the interface between you and apache are all
those methods in the module structure... and every one includes a
context... if you need that context everywhere in your code and don't pass
it around as a parameter to every one of your functions (as we do within
apache), then you'll need to set up some TLS -- but reset it on every
entry point to your module.
> Also, it requires the TLS code to be Apache specific, instead of just
> platform specific. As I said in my previous post, under Win32, for
> instance, the same PHP DLL is used for the CGI (which is just a 16KB big
> .exe) and the IIS module (which is a 90KB DLL). Both use the very same
> language DLL. This is more important that one may think, since the fact
> that all interfaces work with the same DLL allows you to link other DLLs
> against this DLL, and have these other DLLs work with any Win32 interface
> of PHP. For example, if we want to distribute a MySQL extension DLL for
> PHP, we won't have to distribute one MySQL extension for CGI, one for IIS,
> another for Apache and another for NSAPI - but just one extension DLL,
> that's linked against the PHP DLL, that every interface uses.
Surely your DLL has a special entry point for each apache method you hook,
right? Make that entry point set up your TLS.
> Which brings me to another consideration - while I haven't read the full
> ISAPI spec anywhere (if it even exists) - I think you can rely on all of
> the request stages in ISAPI happening within the same thread (Microsoft
> certainly uses it in their examples as far as I recall). If you won't be
> able to rely on it in Apache 2.0, it'll pretty much mean that ISAPI will
> not be supported..?
That's just too unfortunate then. i.e. I don't care. I'm not going to
stop the progress of apache just for some stupidity in ISAPI.
If ISAPI requires that, then the WINNT MPM will have to guarantee it.
It's easy to guarantee it by not implementing any of the async stuff. Too
bad for WINNT users, they're going to be stuck a generation behind in
technology (as if that should bother them, they're using NT after all).
> I'm not exactly sure how we could implement Apache local storage
> equivalent, unless we have access to the conn_rec from anywhere in the
> code, regardless of function arguments.
I know I'm repeating myself a bunch... you have it at all the boundaries
between apache and your code. That's enough.
> If you want TLS support within
> Apache you can probably use the TLS code from PHP 4.0 (platform
> independant and more powerful than the TLS in Win32), you'd just implement
> the function that returns the thread id as a function that returns the
> conn_rec pointer or identifier somehow.
Ryan or someone working on APR might be interested in this.
> Two 'still's remain here though. It still means that the module would
> have to use Apache-specific TLS, which is less than optimal from a point
> of view of a module writer that wants the same code to be used on multiple
> servers (that would be me in our case:).
Nope it wouldn't require apache-specific TLS...
> And my general hunch about it is still negative, in the sense that I think
> we'd bump into many problems, especially in modules that use 3rd party
> libraries that are also thread safe (for example, say you initialize the
> MySQL client library in one phase, and then use it in another - there's a
> good chance there's thread-specific initialization in that library, and it
> won't happen in such a case; In MySQL's case, I don't think we have a
> problem, but I'm almost sure there'd be others in which we would). And
> there's also ISAPI.
>
> How feasible would it be to be able to mark a module as one that wants
> all of its request steps to be performed within the same thread?
To be honest, at the moment I really only care about this requirement
between the handler phase, and the subsuquent logging phase.
But I still say that such libraries are broken. When you "initialize the
mysql client library" it passes you back a pointer, right? A pointer
which you then pass to all other mysql library functions, right? If their
code doesn't hang all the info they need off that pointer, then their code
is broken. Sorry, but that's just how it is. Why pass back a pointer if
it's not all the info they need?
That'd be like saying FILE *f = fopen("foo", "r"); can only be used within
the thread that the fopen occurs. Which seems pretty silly to me...
However, I can relax the requirement a small amount: It's only MPMs which
support async behaviour which have this requirement. On all unix
platforms there will be the prefork MPM which won't be async; and it's a
small exercise to build a pthread MPM which won't have async support.
Async is an extra feature for the MPM, not a requirement. So we can cater
to such brokenness on all unix platforms simply by choosing a less than
optimal MPM.
But for the majority of users, I want async MPM support.
Dean
Re: work in progress: mpm-3.tar.gz (fwd)
Posted by Zeev Suraski <bo...@netvision.net.il>.
This will probably be rejected on new-httpd, as my subscription address is
bourbon-new-httpd@bourbon.netvision.net.il, whereas my From field is
either bourbon@netvision.net.il or zeev@zend.com...
On Fri, 18 Jun 1999, Dean Gaudet wrote:
> Apache already passes around a request_rec pretty much everywhere... I
> suppose we can implement apache-specific "thread local storage", which we
> save and restore if we ever switch threads... but for first implementation
> I'm really not going to worry about it...
>
> You could base your storage off the conn_rec * instead of a thread_id...
> Also, I was planning on building a connection_id which is a densely packed
> small integer, because the scoreboard will need something like this.
Correct me if I'm wrong, but you can't obtain the conn_rec from anywhere
in the code by calling a simple get_conn_rec_ptr()..? I guess it'll be
possible to implement such a function (if you keep some mapping between
thread id's and their current corresponding conn_rec's, even though I'm
not sure if you need it for other purposes or whether it would be just
pure added overhead).
Without such a function, you still have to pass the conn_rec pointer
around everywhere, to any function that may possibly need access to a
global per-thread variable, which is exactly what TLS comes to solve...
> The main case I'm considering this for is the handler phase. In general,
> any request goes through a bunch of protocol stages and reaches the
> handler, and from there it fits into a few small categories:
>
> 1. copy a file fd back to the client
> 2. copy a pipe/socket fd (from another process) back to the client
> 3. copy a mmapped region back to the client
> 4. copy a dynamically generated memory region back to the client
> 5. the handler writes stuff to a BUFF, and its sent to the client
>
> 1, 2, 3, and 4 are very simple cases where if the stuff to be sent doesn't
> fit in the socket's send buffer, and we have to block the thread serving
> the response. At this point we're potentially consuming an expensive
> resource (a thread, stack, kernel memory for the thread, ...) just to wait
> for the client.
>
> Instead we can switch to an asynchronous behaviour, all of 1, 2, 3, 4 are
> obvious -- the handler is essentially in a loop which we all know the
> structure of, because the entire object is already generated somewhere.
> The handler at this point sets up a special new record in the conn_rec,
> and will return with a special return code indicating the switch to
> asynchronous behaviour.
>
> If the MPM supports async stuff, then it will release this thread from
> serving the rest of this request... otherwise there'll be a library to
> handle the "async" stuff synchronously. This async stuff will be
> completed using select/poll/non-blocking i/o (or other similar variants,
> there are several other faster methods on other platforms). We've freed
> up the expensive resource:
>
> - consume less CPU per client
> - handle thousands upon thousands of long haul slow clients, because
> they're just a conn_rec/request_rec at this point... no kernel stack, no
> context switching... really, just minimal resource consumption
> - do better on existing benchmarks, but more importantly, do better on
> real world problems
>
> At some point in the future the async stuff will finish, and a caller
> supplied "completion" function will be called in another thread. This
> gets us back out of the async core, and into protocol code, which will
> "resume" the handler. This way the async core really has no knowledge of
> the protocols involved -- and we can use this technique for any protocol
> (and for variations on 1, 2, 3, 4)...
>
> 5. is the case for modules which don't want to take advantage of the async
> features. But we can give them help by turning them into case 4 with more
> features for BUFF... such as buffer up to 50k responses, and do the async
> thing, otherwise do it synchronously.
Well, I guess it all depends on what kind of stages we're talking about.
If the model is remotely similar to Apache 1.x, then we're not in too much
trouble, since PHP does mostly all of its job in one function call (one
hook). Several other hooks are the configuration directives which rely on
Apache passing around the configuration struct pointer anyway, so it may
be ok (even though there might be thread-specific resources attached to
this configuration structure, I haven't thought about it thoroughly yet).
> I think you're missing something slightly non-obvious :) Or I'm still
> missing your point...
>
> In essence I'm saying that the "thread local storage" is part of the
> conn_rec (or request_rec, whichever is most convenient for you). All
> entry points into your module include a request_rec structure -- you can
> fetch a void * pointer from your request_data entry; you can store
> whatever you need there.
Well then, I wasn't missing that, it's just not enough for our purposes :)
One question I raised is whether I can get to that resource from anywhere
in the code even though it wasn't passed on to me through the stack.
Also, it requires the TLS code to be Apache specific, instead of just
platform specific. As I said in my previous post, under Win32, for
instance, the same PHP DLL is used for the CGI (which is just a 16KB big
.exe) and the IIS module (which is a 90KB DLL). Both use the very same
language DLL. This is more important that one may think, since the fact
that all interfaces work with the same DLL allows you to link other DLLs
against this DLL, and have these other DLLs work with any Win32 interface
of PHP. For example, if we want to distribute a MySQL extension DLL for
PHP, we won't have to distribute one MySQL extension for CGI, one for IIS,
another for Apache and another for NSAPI - but just one extension DLL,
that's linked against the PHP DLL, that every interface uses.
Which brings me to another consideration - while I haven't read the full
ISAPI spec anywhere (if it even exists) - I think you can rely on all of
the request stages in ISAPI happening within the same thread (Microsoft
certainly uses it in their examples as far as I recall). If you won't be
able to rely on it in Apache 2.0, it'll pretty much mean that ISAPI will
not be supported..?
> So think of the thread as a resource which happens to execute your code
> for a while, but think of the conn_rec/request_rec as your indication of
> what is going on.
>
> Apache *could* support "thread local storage", and if this really bothers
> you then I'll encourage you to supply a patch. We can change the
> requirement this way:
>
> - MPMs which do not guarantee to use the same thread for all request
> phases must save and restore the thread local storage across such changes
>
> ... but this is really hard to do portably -- unless we require all
> modules to go through an apache, portable thread local storage API. Which
> means I'd rather it wait for APR, or rather someone else take care of
> it... 'cause the stuff which I'm working on is busy enough already :)
I'm not exactly sure how we could implement Apache local storage
equivalent, unless we have access to the conn_rec from anywhere in the
code, regardless of function arguments. If you want TLS support within
Apache you can probably use the TLS code from PHP 4.0 (platform
independant and more powerful than the TLS in Win32), you'd just implement
the function that returns the thread id as a function that returns the
conn_rec pointer or identifier somehow.
Two 'still's remain here though. It still means that the module would
have to use Apache-specific TLS, which is less than optimal from a point
of view of a module writer that wants the same code to be used on multiple
servers (that would be me in our case:).
And my general hunch about it is still negative, in the sense that I think
we'd bump into many problems, especially in modules that use 3rd party
libraries that are also thread safe (for example, say you initialize the
MySQL client library in one phase, and then use it in another - there's a
good chance there's thread-specific initialization in that library, and it
won't happen in such a case; In MySQL's case, I don't think we have a
problem, but I'm almost sure there'd be others in which we would). And
there's also ISAPI.
How feasible would it be to be able to mark a module as one that wants all
of its request steps to be performed within the same thread?
Zeev
--
-----------------------------------------------------
Zeev Suraski <ze...@zend.com>
For a PGP public key, finger bourbon@netvision.net.il