You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mod_python-dev@quetz.apache.org by David Fraser <da...@sjsoft.com> on 2006/08/24 15:05:02 UTC
Server Shutdown and register_cleanup
David Fraser wrote:
> Graham Dumpleton wrote:
>> On 18/08/2006, at 7:48 PM, Richard Lewis wrote:
>>> On Tuesday 15 August 2006 23:31, Graham Dumpleton wrote:
>>>> Richard Lewis wrote ..
>>>>> Do you mean that there will be no opportunity to have code run at
>>>>> server
>>>>> shutdown?
>>>> Correct. Reason being that it doesn't actually work most of the time
>>>> anyway.
>>>>
>>>>
>>>> Because of how Apache is implemented, there is no reliable/safe way of
>>>> implementing this feature. If one can't do it properly, it seems
>>>> better not
>>>> to attempt it at all.
>>> Goodness! So, in my mod_python applications I often acquire database
>>> handles
>>> and store them in objects outside of the handler() function so that
>>> they
>>> persist between requests. (This is to avoid the expense of acquiring
>>> a new
>>> handle for every request).
>> And you can still do that.
>>> I then use a cleanup() function to release those
>>> database handles.
>>>
>>> The implication of this is that I will no longer be able to release the
>>> handles at server shutdown, yes?
>> I think you miss what I have been saying. That it doesn't work now.
>> If you
>> were to put calls to apache.log_error() in your cleanup handler and
>> shutdown
>> Apache when it is handling a decent amount of traffic, you will probably
>> find your cleanup functions aren't actually being called, or if they
>> do, they
>> might not exit and will hang at some point. What you are more likely
>> to see
>> in the Apache log is a series of SIGTERM signals being sent and then a
>> SIGKILL signal which is forcibly killing off the process.
>>> Can anyone suggest an alternative method of
>>> doing this? Are there any Python tricks where you can execute code
>>> when the
>>> interpreter itself is about to stop?
>> Python does have means of calling code on application shutdown, but
>> because of how Apache uses signals to shutdown processes and how
>> the Apache main loop works, with the main loop being managed
>> by Apache and not Python they can't be used either.
>>> Or could I have a Python script running
>>> in another process which "looks in" to the mod_python process and
>>> periodically cleanly releases database handles?
>> No.
>>
>> In the greater scheme of things it shouldn't ultimately matter. This
>> is because
>> your database server is going to notice that the connections to it
>> have been
>> dropped and will cleanup the resources on its side anyway. It has to
>> do this
>> as there is nothing to say that Apache or any client process connecting
>> to it will not simply crash without cleanly disconnecting. In other
>> words, you
>> will not get resource leaks. That things didn't get cleanup in the
>> Apache child
>> process doesn't matter as it has been killed anyway, with all its
>> memory being
>> released back to operating system.
> So you are saying:
> 1) There is a mechanism for cleaning up code
> 2) This mechanism is not reliable
> 3) Since databases have to assume clients are not reliable, they clean
> up for them anyway
> 4) Therefore we should not even try to clean up
>
> I'm with you on points 1, 2, and 3, but I think point 4 is taking it a
> bit too far...
> Surely there must be *some* value in trying to clean up behind
> yourself, sometimes?
Hi
I thought it would be good to take this across to python-dev. I've read
through
https://issues.apache.org/jira/browse/MODPYTHON-109?page=all
and the discussion in
http://www.modpython.org/pipermail/mod_python/2006-January/019865.html
http://www.modpython.org/pipermail/mod_python/2006-January/019866.html and
http://www.modpython.org/pipermail/mod_python/2006-January/019870.html
again, and I'm just not sure about this.
Basically, Apache seems to provide some sort of mechanism for child
processes to clean themselves up, and for modules to clean up their
resources in a particular child.
The argument to remove the ability to clean up Python objects seems to
be that:
A) The finalize method was been called in an awkward place (from inside
a signal handler) and other code may be running and have the GIL, so it
may not be called at all, even in a graceful shutdown.
B) A normal restart will just send a TERM signal, which doesn't give
proper opportunity for cleanup
C) If the graceful shutdown doesn't work or respond quickly, Apache will
just kill the process anyway, so we may as will live with being killed
(talk about mixed metaphors...)
D) Since databases etc have to deal with the client process being
killed, they generally will handle this
I accept that problem A with the finalizing methods is a real problem,
but wonder if there are alternate solutions that can be provided to
allow cleanups to be attempted.
I don't think that B or C is a good argument - in that case why would
Apache be providing the hooks to clean up anyway? It feels like throwing
in the towel...
And D just seems impolite - if we can try and clean up we should.
Of course, if we can't manage to call finalize methods even in a
graceful shutdown none of this may be possible...
Trying to find relevant info on this from the Apache docs and other
module documentation:
http://httpd.apache.org/docs/2.2/stopping.html#gracefulstop
talks about advising children to exit after their current request. In
this case it would seem the cleanup methods should get called at the end
of the request processing, and thus shouldn't be in a signal handler
(and there should be no other Python code executing...)
http://www.apachetutor.org/dev/pools
talks about using pools to allocate/deallocate resources other than
memory - could we provide a way to register Python objects that need
cleanup using this mechanism?
Am I barking up the wrong tree or is this worth investigating further?
David
Re: Server Shutdown and register_cleanup
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
On 26/08/2006, at 6:22 AM, Jim Gallacher wrote:
> There was a question on the mod_python list regarding mpcp, which
> provides a mod_python handler for cherrypy. Out of curiosity I took
> a look at mpcp. Low and behold, they use
> req.server.register_cleanup to stop the cherrypy server.
>
> I'm becoming increasingly concerned about dropping
> server.register_cleanup. I suspect that we may end up breaking code
> all over the place. MODPYTHON-109 could end up being a bigger PITA
> than it already is.
Let us for the time being then simply work out for the various MPMs
under what
circumstances it may or may not work and document the various cases
in the
mod_python documentation. It may be the case that the worst of the
problems
have in fact already been solved by at least not calling Py_Finalize
() when
Apache is being shutdown. It may also be the case that perhaps
problems are
not as prevalent or do not exist in newer versions of Apache.
Anyway, I will try and do a proper analysis of what actually happens
by putting
some debug in the httpd source code and tracking through what gets
called when.
Graham
Re: Server Shutdown and register_cleanup
Posted by Jim Gallacher <jp...@jgassociates.ca>.
Jim Gallacher wrote:
> There was a question on the mod_python list regarding mpcp, which
> provides a mod_python handler for cherrypy. Out of curiosity I took a
> look at mpcp. Low and behold, they use req.server.register_cleanup to
> stop the cherrypy server.
>
> I'm becoming increasingly concerned about dropping
> server.register_cleanup. I suspect that we may end up breaking code all
> over the place. MODPYTHON-109 could end up being a bigger PITA than it
> already is.
>
> Jim
>
>
How about this as a kludge. Make server.register_cleanup an option
either at compile time or at startup via a PythonOption. (I haven't
looked at the code, but I'm assuming that it is possible).
Forcing people that *really* want to use the cleanup to explicitly turn
it on will at least give them fair warning that it might cause some
ugly problems. Something like this could be used:
PythonOption mod_python.enable_server_cleanup "yes, I did read the docs"
Jim
Re: Server Shutdown and register_cleanup
Posted by Jim Gallacher <jp...@jgassociates.ca>.
There was a question on the mod_python list regarding mpcp, which
provides a mod_python handler for cherrypy. Out of curiosity I took a
look at mpcp. Low and behold, they use req.server.register_cleanup to
stop the cherrypy server.
I'm becoming increasingly concerned about dropping
server.register_cleanup. I suspect that we may end up breaking code all
over the place. MODPYTHON-109 could end up being a bigger PITA than it
already is.
Jim
Re: Server Shutdown and register_cleanup
Posted by David Fraser <da...@sjsoft.com>.
Jim Gallacher wrote:
> David Fraser wrote:
>> Jim Gallacher wrote:
>>> David Fraser wrote:
>>>> Hi
>>>>
>>>> I thought it would be good to take this across to python-dev. I've
>>>> read
>>>> through
>>>> https://issues.apache.org/jira/browse/MODPYTHON-109?page=all
>>>> and the discussion in
>>>> http://www.modpython.org/pipermail/mod_python/2006-January/019865.html
>>>> http://www.modpython.org/pipermail/mod_python/2006-January/019866.html
>>>> and
>>>> http://www.modpython.org/pipermail/mod_python/2006-January/019870.html
>>>> again, and I'm just not sure about this.
>>>>
>>>> Basically, Apache seems to provide some sort of mechanism for child
>>>> processes to clean themselves up, and for modules to clean up their
>>>> resources in a particular child.
>>>>
>>>> The argument to remove the ability to clean up Python objects seems to
>>>> be that:
>>>> A) The finalize method was been called in an awkward place (from
>>>> inside
>>>> a signal handler) and other code may be running and have the GIL,
>>>> so it
>>>> may not be called at all, even in a graceful shutdown.
>>>> B) A normal restart will just send a TERM signal, which doesn't give
>>>> proper opportunity for cleanup
>>>> C) If the graceful shutdown doesn't work or respond quickly, Apache
>>>> will
>>>> just kill the process anyway, so we may as will live with being killed
>>>> (talk about mixed metaphors...)
>>>> D) Since databases etc have to deal with the client process being
>>>> killed, they generally will handle this
>>>>
>>>> I accept that problem A with the finalizing methods is a real problem,
>>>> but wonder if there are alternate solutions that can be provided to
>>>> allow cleanups to be attempted.
>>>> I don't think that B or C is a good argument - in that case why would
>>>> Apache be providing the hooks to clean up anyway? It feels like
>>>> throwing
>>>> in the towel...
>>>> And D just seems impolite - if we can try and clean up we should.
>>>> Of course, if we can't manage to call finalize methods even in a
>>>> graceful shutdown none of this may be possible...
>>>>
>>>> Trying to find relevant info on this from the Apache docs and other
>>>> module documentation:
>>>> http://httpd.apache.org/docs/2.2/stopping.html#gracefulstop
>>>> talks about advising children to exit after their current
>>>> request. In
>>>> this case it would seem the cleanup methods should get called at
>>>> the end
>>>> of the request processing, and thus shouldn't be in a signal handler
>>>> (and there should be no other Python code executing...)
>>> Except that the parent "advises" it's children by sending a signal,
>>> doesn't it?
>> On Unix it does, but I'm not sure about Win32.
> I'm not sure about Win32 either, since it doesn't have any child
> processes...
>> Anyway if the exit is not
>> actually not from the signal handler, but the signal handler is simply
>> flagging that an exit should be done after the current request, then the
>> cleanup could be done alongside the exit and outside of the signal
>> handler...
>>>> http://www.apachetutor.org/dev/pools
>>>> talks about using pools to allocate/deallocate resources other than
>>>> memory - could we provide a way to register Python objects that need
>>>> cleanup using this mechanism?
>>> That *is* the mechanism that mod_python uses to register cleanups.
>>> req.register_cleanup uses the request pool, and
>>> apache.register_cleanup uses the server pool (child_init_pool).
>> Good then :-)
>>>> Am I barking up the wrong tree or is this worth investigating further?
>>>> David
>>> It's worth investigating. There may be a solution, but we just can't
>>> see it. I don't think anyone would argue that the current proposal to
>>> drop the server cleanup is sub-optimal, but the current implementation
>>> is worse than having no cleanup at all.
>> OK great that's reassuring. I forgot to mention in the above email the
>> mod_perl documentation that seems to indicate that mod_perl does this:
>> http://modperlbook.org/html/ch05_03.html
> Interesting, in as much as it touches on the problem we are trying to
> solve here. See section 5.3.2.
>> http://162.105.203.19/apache-doc/24.htm#BIN67
>> http://162.105.203.19/apache-doc/79.htm#BIN172
> I've been reading this book, "Writing Apache Modules with Perl and C",
> the last couple of days. :) It's a darn good yarn, even if I did
> figure out who-done-it by the end of the first chapter. As I'm reading
> I keep having a recurring fantasy... "wouldn't it be great to have
> this kind of resource for mod_python"? I think I need to get out more. :)
>
> What you need to realize is that mod_python is not doing anything
> exotic. We are all playing in the same sandbox by the same rules
> imposed by apache. Callbacks for things like child initialization and
> exit, or any other phase get triggered the same way in any module.
> What we are bumping into with this particular bug is a limitation of
> the python interpreter, and the whole GIL problem.
Right, I've understood that now. Will try see if I can get anything done
on it, but at least I understand the position more clearly now. Thanks
for the feedback
David
Re: Server Shutdown and register_cleanup
Posted by Jim Gallacher <jp...@jgassociates.ca>.
David Fraser wrote:
> Jim Gallacher wrote:
>> David Fraser wrote:
>>> Hi
>>>
>>> I thought it would be good to take this across to python-dev. I've read
>>> through
>>> https://issues.apache.org/jira/browse/MODPYTHON-109?page=all
>>> and the discussion in
>>> http://www.modpython.org/pipermail/mod_python/2006-January/019865.html
>>> http://www.modpython.org/pipermail/mod_python/2006-January/019866.html
>>> and
>>> http://www.modpython.org/pipermail/mod_python/2006-January/019870.html
>>> again, and I'm just not sure about this.
>>>
>>> Basically, Apache seems to provide some sort of mechanism for child
>>> processes to clean themselves up, and for modules to clean up their
>>> resources in a particular child.
>>>
>>> The argument to remove the ability to clean up Python objects seems to
>>> be that:
>>> A) The finalize method was been called in an awkward place (from inside
>>> a signal handler) and other code may be running and have the GIL, so it
>>> may not be called at all, even in a graceful shutdown.
>>> B) A normal restart will just send a TERM signal, which doesn't give
>>> proper opportunity for cleanup
>>> C) If the graceful shutdown doesn't work or respond quickly, Apache will
>>> just kill the process anyway, so we may as will live with being killed
>>> (talk about mixed metaphors...)
>>> D) Since databases etc have to deal with the client process being
>>> killed, they generally will handle this
>>>
>>> I accept that problem A with the finalizing methods is a real problem,
>>> but wonder if there are alternate solutions that can be provided to
>>> allow cleanups to be attempted.
>>> I don't think that B or C is a good argument - in that case why would
>>> Apache be providing the hooks to clean up anyway? It feels like throwing
>>> in the towel...
>>> And D just seems impolite - if we can try and clean up we should.
>>> Of course, if we can't manage to call finalize methods even in a
>>> graceful shutdown none of this may be possible...
>>>
>>> Trying to find relevant info on this from the Apache docs and other
>>> module documentation:
>>> http://httpd.apache.org/docs/2.2/stopping.html#gracefulstop
>>> talks about advising children to exit after their current request. In
>>> this case it would seem the cleanup methods should get called at the end
>>> of the request processing, and thus shouldn't be in a signal handler
>>> (and there should be no other Python code executing...)
>> Except that the parent "advises" it's children by sending a signal,
>> doesn't it?
> On Unix it does, but I'm not sure about Win32.
I'm not sure about Win32 either, since it doesn't have any child
processes...
> Anyway if the exit is not
> actually not from the signal handler, but the signal handler is simply
> flagging that an exit should be done after the current request, then the
> cleanup could be done alongside the exit and outside of the signal
> handler...
>>> http://www.apachetutor.org/dev/pools
>>> talks about using pools to allocate/deallocate resources other than
>>> memory - could we provide a way to register Python objects that need
>>> cleanup using this mechanism?
>> That *is* the mechanism that mod_python uses to register cleanups.
>> req.register_cleanup uses the request pool, and
>> apache.register_cleanup uses the server pool (child_init_pool).
> Good then :-)
>>> Am I barking up the wrong tree or is this worth investigating further?
>>> David
>> It's worth investigating. There may be a solution, but we just can't
>> see it. I don't think anyone would argue that the current proposal to
>> drop the server cleanup is sub-optimal, but the current implementation
>> is worse than having no cleanup at all.
> OK great that's reassuring. I forgot to mention in the above email the
> mod_perl documentation that seems to indicate that mod_perl does this:
> http://modperlbook.org/html/ch05_03.html
Interesting, in as much as it touches on the problem we are trying to
solve here. See section 5.3.2.
> http://162.105.203.19/apache-doc/24.htm#BIN67
> http://162.105.203.19/apache-doc/79.htm#BIN172
I've been reading this book, "Writing Apache Modules with Perl and C",
the last couple of days. :) It's a darn good yarn, even if I did figure
out who-done-it by the end of the first chapter. As I'm reading I keep
having a recurring fantasy... "wouldn't it be great to have this kind of
resource for mod_python"? I think I need to get out more. :)
What you need to realize is that mod_python is not doing anything
exotic. We are all playing in the same sandbox by the same rules imposed
by apache. Callbacks for things like child initialization and exit, or
any other phase get triggered the same way in any module. What we are
bumping into with this particular bug is a limitation of the python
interpreter, and the whole GIL problem.
>> Really though, isn't this whole discussion actually about database
>> connection pooling? Doesn't that cover 99% of the cases people care
>> about? If so maybe our energies would be better focused on what may be
>> required to support mod_dbd within mod_python.
> Database connection pooling does seem a large amount of it, but we also
> do other things from within Apache like launching separate index
> processes or running things like Excel via COM. At the moment the
> indexing process watches the parent process and exit when it does, but
> it might be quite nice to be able to tell the child process it should
> exit explicitly.
>> http://httpd.apache.org/docs/2.2/mod/mod_dbd.html
> Although it may be awkward to use mod_dbd with its limited set of
> database drivers and functionality, when there is the Python DB-API ...
> I've looked at the mod_dbd documentation before - how do you even
> execute a SQL statement and retrieve the rows? Maybe I'm missing how it
> works...
> But if we could get mod_dbd to manage Python DB-API connections and pool
> them, now that would be cool as it would require minimal changes to
> existing Python code...
I've only taken a cursory glance at mod_dbd and the underlying apr_dbd,
and only in a few stolen moments during the day today, but my gut tells
me that it may not be a simple as I had hoped this morning. I have a
feeling that we might actually need to write a python DB-API wrapper
around the apr_dbd_* calls. This would certainly be a non-trivial thing,
but would be kinda cool. Having this wrapper would allow us to add sql
database functionality (I'm thinking about the often discussed
SQL-Session subclass) without worrying about any particular database
dependency, and would likely be a real boon for mod_python.
My guess is we won't see a Python DB-API wrapper magically appearing
from outside of the mod_python community, so if we want it... start
hacking. :) It might be something to consider for 3.4. It could also
make a nice Google Summer of Code project for next year if nothing
happens in the interim.
Jim
Re: Server Shutdown and register_cleanup
Posted by David Fraser <da...@sjsoft.com>.
Jim Gallacher wrote:
> David Fraser wrote:
>> Hi
>>
>> I thought it would be good to take this across to python-dev. I've read
>> through
>> https://issues.apache.org/jira/browse/MODPYTHON-109?page=all
>> and the discussion in
>> http://www.modpython.org/pipermail/mod_python/2006-January/019865.html
>> http://www.modpython.org/pipermail/mod_python/2006-January/019866.html
>> and
>> http://www.modpython.org/pipermail/mod_python/2006-January/019870.html
>> again, and I'm just not sure about this.
>>
>> Basically, Apache seems to provide some sort of mechanism for child
>> processes to clean themselves up, and for modules to clean up their
>> resources in a particular child.
>>
>> The argument to remove the ability to clean up Python objects seems to
>> be that:
>> A) The finalize method was been called in an awkward place (from inside
>> a signal handler) and other code may be running and have the GIL, so it
>> may not be called at all, even in a graceful shutdown.
>> B) A normal restart will just send a TERM signal, which doesn't give
>> proper opportunity for cleanup
>> C) If the graceful shutdown doesn't work or respond quickly, Apache will
>> just kill the process anyway, so we may as will live with being killed
>> (talk about mixed metaphors...)
>> D) Since databases etc have to deal with the client process being
>> killed, they generally will handle this
>>
>> I accept that problem A with the finalizing methods is a real problem,
>> but wonder if there are alternate solutions that can be provided to
>> allow cleanups to be attempted.
>> I don't think that B or C is a good argument - in that case why would
>> Apache be providing the hooks to clean up anyway? It feels like throwing
>> in the towel...
>> And D just seems impolite - if we can try and clean up we should.
>> Of course, if we can't manage to call finalize methods even in a
>> graceful shutdown none of this may be possible...
>>
>> Trying to find relevant info on this from the Apache docs and other
>> module documentation:
>> http://httpd.apache.org/docs/2.2/stopping.html#gracefulstop
>> talks about advising children to exit after their current request. In
>> this case it would seem the cleanup methods should get called at the end
>> of the request processing, and thus shouldn't be in a signal handler
>> (and there should be no other Python code executing...)
> Except that the parent "advises" it's children by sending a signal,
> doesn't it?
On Unix it does, but I'm not sure about Win32. Anyway if the exit is not
actually not from the signal handler, but the signal handler is simply
flagging that an exit should be done after the current request, then the
cleanup could be done alongside the exit and outside of the signal
handler...
>> http://www.apachetutor.org/dev/pools
>> talks about using pools to allocate/deallocate resources other than
>> memory - could we provide a way to register Python objects that need
>> cleanup using this mechanism?
> That *is* the mechanism that mod_python uses to register cleanups.
> req.register_cleanup uses the request pool, and
> apache.register_cleanup uses the server pool (child_init_pool).
Good then :-)
>> Am I barking up the wrong tree or is this worth investigating further?
>> David
> It's worth investigating. There may be a solution, but we just can't
> see it. I don't think anyone would argue that the current proposal to
> drop the server cleanup is sub-optimal, but the current implementation
> is worse than having no cleanup at all.
OK great that's reassuring. I forgot to mention in the above email the
mod_perl documentation that seems to indicate that mod_perl does this:
http://modperlbook.org/html/ch05_03.html
http://162.105.203.19/apache-doc/24.htm#BIN67
http://162.105.203.19/apache-doc/79.htm#BIN172
> Really though, isn't this whole discussion actually about database
> connection pooling? Doesn't that cover 99% of the cases people care
> about? If so maybe our energies would be better focused on what may be
> required to support mod_dbd within mod_python.
Database connection pooling does seem a large amount of it, but we also
do other things from within Apache like launching separate index
processes or running things like Excel via COM. At the moment the
indexing process watches the parent process and exit when it does, but
it might be quite nice to be able to tell the child process it should
exit explicitly.
> http://httpd.apache.org/docs/2.2/mod/mod_dbd.html
Although it may be awkward to use mod_dbd with its limited set of
database drivers and functionality, when there is the Python DB-API ...
I've looked at the mod_dbd documentation before - how do you even
execute a SQL statement and retrieve the rows? Maybe I'm missing how it
works...
But if we could get mod_dbd to manage Python DB-API connections and pool
them, now that would be cool as it would require minimal changes to
existing Python code...
David
Re: Server Shutdown and register_cleanup
Posted by Jim Gallacher <jp...@jgassociates.ca>.
David Fraser wrote:
> David Fraser wrote:
>> Graham Dumpleton wrote:
>>> On 18/08/2006, at 7:48 PM, Richard Lewis wrote:
>>>> On Tuesday 15 August 2006 23:31, Graham Dumpleton wrote:
>>>>> Richard Lewis wrote ..
>>>>>> Do you mean that there will be no opportunity to have code run at
>>>>>> server
>>>>>> shutdown?
>>>>> Correct. Reason being that it doesn't actually work most of the time
>>>>> anyway.
>>>>>
>>>>>
>>>>> Because of how Apache is implemented, there is no reliable/safe way of
>>>>> implementing this feature. If one can't do it properly, it seems
>>>>> better not
>>>>> to attempt it at all.
>>>> Goodness! So, in my mod_python applications I often acquire database
>>>> handles
>>>> and store them in objects outside of the handler() function so that
>>>> they
>>>> persist between requests. (This is to avoid the expense of acquiring
>>>> a new
>>>> handle for every request).
>>> And you can still do that.
>>>> I then use a cleanup() function to release those
>>>> database handles.
>>>>
>>>> The implication of this is that I will no longer be able to release the
>>>> handles at server shutdown, yes?
>>> I think you miss what I have been saying. That it doesn't work now.
>>> If you
>>> were to put calls to apache.log_error() in your cleanup handler and
>>> shutdown
>>> Apache when it is handling a decent amount of traffic, you will probably
>>> find your cleanup functions aren't actually being called, or if they
>>> do, they
>>> might not exit and will hang at some point. What you are more likely
>>> to see
>>> in the Apache log is a series of SIGTERM signals being sent and then a
>>> SIGKILL signal which is forcibly killing off the process.
>>>> Can anyone suggest an alternative method of
>>>> doing this? Are there any Python tricks where you can execute code
>>>> when the
>>>> interpreter itself is about to stop?
>>> Python does have means of calling code on application shutdown, but
>>> because of how Apache uses signals to shutdown processes and how
>>> the Apache main loop works, with the main loop being managed
>>> by Apache and not Python they can't be used either.
>>>> Or could I have a Python script running
>>>> in another process which "looks in" to the mod_python process and
>>>> periodically cleanly releases database handles?
>>> No.
>>>
>>> In the greater scheme of things it shouldn't ultimately matter. This
>>> is because
>>> your database server is going to notice that the connections to it
>>> have been
>>> dropped and will cleanup the resources on its side anyway. It has to
>>> do this
>>> as there is nothing to say that Apache or any client process connecting
>>> to it will not simply crash without cleanly disconnecting. In other
>>> words, you
>>> will not get resource leaks. That things didn't get cleanup in the
>>> Apache child
>>> process doesn't matter as it has been killed anyway, with all its
>>> memory being
>>> released back to operating system.
>> So you are saying:
>> 1) There is a mechanism for cleaning up code
>> 2) This mechanism is not reliable
>> 3) Since databases have to assume clients are not reliable, they clean
>> up for them anyway
>> 4) Therefore we should not even try to clean up
>>
>> I'm with you on points 1, 2, and 3, but I think point 4 is taking it a
>> bit too far...
>> Surely there must be *some* value in trying to clean up behind
>> yourself, sometimes?
> Hi
>
> I thought it would be good to take this across to python-dev. I've read
> through
> https://issues.apache.org/jira/browse/MODPYTHON-109?page=all
> and the discussion in
> http://www.modpython.org/pipermail/mod_python/2006-January/019865.html
> http://www.modpython.org/pipermail/mod_python/2006-January/019866.html and
> http://www.modpython.org/pipermail/mod_python/2006-January/019870.html
> again, and I'm just not sure about this.
>
> Basically, Apache seems to provide some sort of mechanism for child
> processes to clean themselves up, and for modules to clean up their
> resources in a particular child.
>
> The argument to remove the ability to clean up Python objects seems to
> be that:
> A) The finalize method was been called in an awkward place (from inside
> a signal handler) and other code may be running and have the GIL, so it
> may not be called at all, even in a graceful shutdown.
> B) A normal restart will just send a TERM signal, which doesn't give
> proper opportunity for cleanup
> C) If the graceful shutdown doesn't work or respond quickly, Apache will
> just kill the process anyway, so we may as will live with being killed
> (talk about mixed metaphors...)
> D) Since databases etc have to deal with the client process being
> killed, they generally will handle this
>
> I accept that problem A with the finalizing methods is a real problem,
> but wonder if there are alternate solutions that can be provided to
> allow cleanups to be attempted.
> I don't think that B or C is a good argument - in that case why would
> Apache be providing the hooks to clean up anyway? It feels like throwing
> in the towel...
> And D just seems impolite - if we can try and clean up we should.
> Of course, if we can't manage to call finalize methods even in a
> graceful shutdown none of this may be possible...
>
> Trying to find relevant info on this from the Apache docs and other
> module documentation:
> http://httpd.apache.org/docs/2.2/stopping.html#gracefulstop
> talks about advising children to exit after their current request. In
> this case it would seem the cleanup methods should get called at the end
> of the request processing, and thus shouldn't be in a signal handler
> (and there should be no other Python code executing...)
Except that the parent "advises" it's children by sending a signal,
doesn't it?
> http://www.apachetutor.org/dev/pools
> talks about using pools to allocate/deallocate resources other than
> memory - could we provide a way to register Python objects that need
> cleanup using this mechanism?
That *is* the mechanism that mod_python uses to register cleanups.
req.register_cleanup uses the request pool, and apache.register_cleanup
uses the server pool (child_init_pool).
> Am I barking up the wrong tree or is this worth investigating further?
> David
It's worth investigating. There may be a solution, but we just can't see
it. I don't think anyone would argue that the current proposal to drop
the server cleanup is sub-optimal, but the current implementation is
worse than having no cleanup at all.
Really though, isn't this whole discussion actually about database
connection pooling? Doesn't that cover 99% of the cases people care
about? If so maybe our energies would be better focused on what may be
required to support mod_dbd within mod_python.
http://httpd.apache.org/docs/2.2/mod/mod_dbd.html
Jim