You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mod_python-dev@quetz.apache.org by Graham Dumpleton <gr...@dscpl.com.au> on 2006/10/19 02:47:26 UTC
Status of mod_python 3.3.
On JIRA, the following issues are still marked as incomplete for mod_python
version 3.3. I have noted my own comments about where they are up to and
what I think still needs to be done.
MODPYTHON-93 Improve util.FieldStorage efficiency.
This was actually marked as resolved but reopened because it was discovered
that changes meant that Trac <=0.9.6 would no longer work. The changes were
also backed out of mod_python 3.2.X branch and not released in 3.2.10.
At this point I believe we have agreed that code in 3.3 would be left as is and
people would need to use Trac >=0.10, which has now been release, with
mod_python 3.3 or later.
There was comments as to whether util.FieldStorage needs to have more
dictionary like access, but at this point I believe we should mark this issue
as resolved and if people want dictionary like access, they can open a
separate JIRA issue for that and we deal with it in a future release.
In summary, I believe we should mark this as resolved.
MODPYTHON-104 Allow Python code callouts with mod_include (SSI).
The code for all this has been done for some time. The only reason it hasn't
been marked as resolved as no documentation has been added into core
mod_python documentation. I have separately written a article on the new
feature which is available at:
http://www.dscpl.com.au/wiki/ModPython/Articles/BasicsOfServerSideIncludes
I have no problem as this being used as basis for core documentation.
I had been holding off integrating it because of contention over whether we
could use wiki for documentation or not.
In summary, need to still keep this open until some documentation added
to core mod_python documentation.
MODPYTHON-127 Use namespace for mod_python PythonOption settings.
I have made code changes but not committed them back to repository. Jim
has committed some documentation changes related to it already though.
Some more documentation changes are probably required where options are
mentioned in relation to features they affect.
I had posed question about whether mod_python.session.database_directory
should also be added as a general fallback in cases where which type of
filesystem based session was not going to be known. Jim responded with +1,
but his explicit vs implicit comment made me unsure which proposal he
was agreeing with. Thus nothing done about that yet.
In summary, bit more work to do.
MODPYTHON-143 Implement and integrate a new module importer.
Code has been done, except for extra bit more logging of exceptions for when
modules hooks are called and a problem occurs. Also need to update the
documentation.
In summary, more work to do but mainly documentation.
MODPYTHON-186 Build process not using correct values from Python
config Makefile.
This is only known to be an issue on Mac OS X when the very latest
compiler tool chain software given out at Mac developer conference
is used. It may only affect new Intel Macs. Someone did suggest they
would come back with required changes but that hasn't happened.
Personally I would say we don't attempt to address this in 3.3 and defer
it till later.
MODPYTHON-190 Python 2.5 support.
>From what I have seen, people are already using Python 2.5, thus is there
any urgency on this? All I can figure is that by not making changes you will
not be able to work with really large data, or will it all crash badly on a 64
bit platform with 64 bit support compiled in.
I don't know the answers to this and no one else (doesn't even have to be
one of the committers) has stepped up to do the work and work out what
the required changes are.
Personally I would say we don't attempt to address this in 3.3 and defer
it till later.
MODPYTHON-193 Add req.hlist.location to mirror req.hlist.directory.
I have done most of the code for this and now just sorting out some
problems with trailing slashes getting added when they shouldn't. Also
need to still update Session code and ensure None is returned for handler
directory when not used in a directory.
In summary, a bit more work to do.
So that is where we are at. Can I get some agreement that we will:
1. Mark MODPYTHON-93 as resolved.
2. Defer MODPYTHON-186 and MODPYTHON-190 till later.
Finally, is there anything else in outstanding issues (not listed here)
that people believe need to be address for mod_python 3.3?
Graham
Re: Status of mod_python 3.3.
Posted by Jim Gallacher <jp...@jgassociates.ca>.
Graham Dumpleton wrote:
> On JIRA, the following issues are still marked as incomplete for mod_python
> version 3.3. I have noted my own comments about where they are up to and
> what I think still needs to be done.
>
> MODPYTHON-93 Improve util.FieldStorage efficiency.
>
> This was actually marked as resolved but reopened because it was discovered
> that changes meant that Trac <=0.9.6 would no longer work. The changes were
> also backed out of mod_python 3.2.X branch and not released in 3.2.10.
>
> At this point I believe we have agreed that code in 3.3 would be left as is and
> people would need to use Trac >=0.10, which has now been release, with
> mod_python 3.3 or later.
I know we've hashed this over a couple of times, but creating this
dependency still makes me nervous.
> There was comments as to whether util.FieldStorage needs to have more
> dictionary like access, but at this point I believe we should mark this issue
> as resolved and if people want dictionary like access, they can open a
> separate JIRA issue for that and we deal with it in a future release.
>
> In summary, I believe we should mark this as resolved.
>
> MODPYTHON-104 Allow Python code callouts with mod_include (SSI).
>
> The code for all this has been done for some time. The only reason it hasn't
> been marked as resolved as no documentation has been added into core
> mod_python documentation. I have separately written a article on the new
> feature which is available at:
>
> http://www.dscpl.com.au/wiki/ModPython/Articles/BasicsOfServerSideIncludes
>
> I have no problem as this being used as basis for core documentation.
> I had been holding off integrating it because of contention over whether we
> could use wiki for documentation or not.
>
> In summary, need to still keep this open until some documentation added
> to core mod_python documentation.
>
> MODPYTHON-127 Use namespace for mod_python PythonOption settings.
>
> I have made code changes but not committed them back to repository. Jim
> has committed some documentation changes related to it already though.
> Some more documentation changes are probably required where options are
> mentioned in relation to features they affect.
>
> I had posed question about whether mod_python.session.database_directory
> should also be added as a general fallback in cases where which type of
> filesystem based session was not going to be known. Jim responded with +1,
> but his explicit vs implicit comment made me unsure which proposal he
> was agreeing with. Thus nothing done about that yet.
Sorry about that. :) I 100% agree with your proposal. By explicit I
meant using your proposed scheme makes it more obvious to the user what
they are configuring. So yes, we should include the additional
PythonOption settings.
> In summary, bit more work to do.
>
> MODPYTHON-143 Implement and integrate a new module importer.
>
> Code has been done, except for extra bit more logging of exceptions for when
> modules hooks are called and a problem occurs. Also need to update the
> documentation.
>
> In summary, more work to do but mainly documentation.
>
> MODPYTHON-186 Build process not using correct values from Python
> config Makefile.
>
> This is only known to be an issue on Mac OS X when the very latest
> compiler tool chain software given out at Mac developer conference
> is used. It may only affect new Intel Macs. Someone did suggest they
> would come back with required changes but that hasn't happened.
>
> Personally I would say we don't attempt to address this in 3.3 and defer
> it till later.
And I'm Mac-less, so I can't help there anyway.
> MODPYTHON-190 Python 2.5 support.
>
>>>From what I have seen, people are already using Python 2.5, thus is there
> any urgency on this? All I can figure is that by not making changes you will
> not be able to work with really large data, or will it all crash badly on a 64
> bit platform with 64 bit support compiled in.
>
> I don't know the answers to this and no one else (doesn't even have to be
> one of the committers) has stepped up to do the work and work out what
> the required changes are.
>
> Personally I would say we don't attempt to address this in 3.3 and defer
> it till later.
I've actually being going through the code in the last couple of days
and was about to ask some questions about this. So I've added comments
to code sections identified by ssizecheck.py. Should I check in the code
with these comments so we can keep track, or just attach a patch to the
JIRA issue?
Personally, I would like to see this fixed for 3.3. The biggest problem
I have is the lack of a 64-bit machine to test on. On the other hand, I
don't have a 64-bit machine, so it doesn't affect me personally if we
don't deal with it.
At the very least we should make sure we include 32-bit python 2.5 in
our testing round.
> MODPYTHON-193 Add req.hlist.location to mirror req.hlist.directory.
>
> I have done most of the code for this and now just sorting out some
> problems with trailing slashes getting added when they shouldn't. Also
> need to still update Session code and ensure None is returned for handler
> directory when not used in a directory.
>
> In summary, a bit more work to do.
>
> So that is where we are at. Can I get some agreement that we will:
>
> 1. Mark MODPYTHON-93 as resolved.
-0
As I said above, I'm still nervous about this, but I wouldn't veto the
release if it is not.
> 2. Defer MODPYTHON-186 till later
+1
> 3. Defer MODPYTHON-190 till later.
-0
Again, I'd prefer to see this resolved, but I wouldn't veto the release
if it is not fixed.
> Finally, is there anything else in outstanding issues (not listed here)
> that people believe need to be address for mod_python 3.3?
There are a couple of things that can be cleaned up and marked as closed.
Add get_session() method to request object.
http://issues.apache.org/jira/browse/MODPYTHON-59
This idea was pretty much shot down, but there is still a bit of
residual code that should be cleaned up. I'll do that and mark it as
closed. I think the idea still has merit, but it would be better to
start from scratch at some future date.
Add a security section to the documentation.
http://issues.apache.org/jira/browse/MODPYTHON-19
Most of what I've put together is really not mod_python specific. I
think this would be better off in the wiki, so I propose that we do that
and mark this issue as closed.
Although there is no JIRA issue for it, I'd like to see us do a quick
code cleanup. I see lots of complier warnings about unused variables and
it would be nice to excise the offending bits of code. I figure we are
more likely to spot real problems if the compiler is spewing less noise.
If there are no objections I'll do this over the weekend.
Jim
Re: Status of mod_python 3.3.
Posted by Mike Looijmans <nl...@natlab.research.philips.com>.
Graham Dumpleton wrote:
...
> MODPYTHON-93 Improve util.FieldStorage efficiency.
>
> This was actually marked as resolved but reopened because it was discovered
> that changes meant that Trac <=0.9.6 would no longer work. The changes were
> also backed out of mod_python 3.2.X branch and not released in 3.2.10.
>
> At this point I believe we have agreed that code in 3.3 would be left as is and
> people would need to use Trac >=0.10, which has now been release, with
> mod_python 3.3 or later.
>
> There was comments as to whether util.FieldStorage needs to have more
> dictionary like access, but at this point I believe we should mark this issue
> as resolved and if people want dictionary like access, they can open a
> separate JIRA issue for that and we deal with it in a future release.
I give it a +1 to mark it resolved.
There's more than enough good stuff in the 93 fix (such as being able to upload large amounts of
data to file-like objects) to justify breaking just one hack (which is easy to fix on the Trac side
anyway). It would be logical to upgrade both the mod_python and trac releases on the machine
simultaneously, so I doubt there will be any real hits in the field.
Mike.
Re: two mod_python leaks
Posted by Jeff Robbins <je...@livedata.com>.
Yes regarding placement of PyThreadState_Clear...sorry about that. I
rebuilt and ran my simple test and it shows no leaks.
static void release_interpreter(void)
{
PyThreadState *tstate = PyThreadState_Get();
// JSR
PyThreadState_Clear(tstate);
#ifdef WITH_THREAD
PyEval_ReleaseThread(tstate);
#else
PyThreadState_Swap(NULL);
#endif
PyThreadState_Delete(tstate);
}
----- Original Message -----
From: "Graham Dumpleton" <gr...@dscpl.com.au>
To: <py...@httpd.apache.org>
Cc: "Jeff Robbins" <je...@livedata.com>
Sent: Thursday, October 19, 2006 7:53 AM
Subject: Re: two mod_python leaks
> Comments below.
>
> On 19/10/2006, at 8:55 PM, Graham Dumpleton wrote:
>
>> For such issues, by all means just send the email direct to the mailing
>> list.
>> I have copied this back onto the list and I'll do some research into
>> what
>> you describe.
>>
>> BTW, I don't use Windows, but there are others on the list who do and it
>> may help if you can explain what methodology you are using to see the
>> Win32 handle leaks. This will help others to replicate it and then track
>> it
>> down.
>>
>> Thanks.
>>
>> Graham
>>
>> On 19/10/2006, at 8:50 PM, Jeff Robbins wrote:
>>
>>> Graham,
>>>
>>> I apologize in advance for not understanding the email etiquette for
>>> the list, so I'm emailing you directly. (I subscribed to the
>>> python-dev list and email the faq magic subject but the reply said "no
>>> info".) Anyway...
>>>
>>> I know of two leaks in Python 3.2.10 / Apache 2.2.3 that I would dearly
>>> love to get fixed. One of them I have identified the source and made a
>>> change to my copy and it works (and comports with the python embedding
>>> documentation) so I was hoping you could either get it in or help me
>>> understand how I can get it applied.
>>>
>>> The first leak is if you use the _repr_ function of a dict or list
>>> (which is as simple as using str([]) in your mod_python handler!) The
>>> repr functions use a dict in the python threadstate to avoid some
>>> recursion problem. This dict is allocated only on demand, so while I
>>> found str([]) (or a str of any dict or list) to cause its allocation,
>>> there may be other user code things you can do to get it allocated. To
>>> free it, the python embedder need to call "PyThreadState_Clear" before
>>> calling "PyThreadState_Delete". Yet mod_python.c release_interpreter()
>>> does not call "PyThreadState_Clear". The fixed version of
>>> release_interpreter is this:
>>>
>>> static void release_interpreter(void)
>>> {
>>> PyThreadState *tstate = PyThreadState_Get();
>>> #ifdef WITH_THREAD
>>> PyEval_ReleaseThread(tstate);
>>> #else
>>> PyThreadState_Swap(NULL);
>>> #endif
>>> // JSR
>>> PyThreadState_Clear(tstate);
>>> PyThreadState_Delete(tstate);
>>> }
>
> Still looking at this, but where you do it looks to be wrong. The
> documentation
> below for PyThreadState_Clear() says:
>
> The interpreter lock must be held.
>
> Where you are calling that function, it isn't. It may need to be moved to
> before
> the call to PyEval_ReleaseThread(). Yes/No?
>
>>> The Python embedded doc is clear that this is the correct thing to do:
>>> http://docs.python.org/api/threads.html
>>> ...
>>> void PyThreadState_Clear( PyThreadState *tstate)
>>>
>>> Reset all information in a thread state object. The interpreter lock
>>> must be held.
>>>
>>> void PyThreadState_Delete( PyThreadState *tstate)
>>>
>>> Destroy a thread state object. The interpreter lock need not be held.
>>> The thread state must have been reset with a previous call to
>>> PyThreadState_Clear().
>
> Graham
>
Re: two mod_python leaks
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
Comments below.
On 19/10/2006, at 8:55 PM, Graham Dumpleton wrote:
> For such issues, by all means just send the email direct to the
> mailing list.
> I have copied this back onto the list and I'll do some research
> into what
> you describe.
>
> BTW, I don't use Windows, but there are others on the list who do
> and it
> may help if you can explain what methodology you are using to see the
> Win32 handle leaks. This will help others to replicate it and then
> track it
> down.
>
> Thanks.
>
> Graham
>
> On 19/10/2006, at 8:50 PM, Jeff Robbins wrote:
>
>> Graham,
>>
>> I apologize in advance for not understanding the email etiquette
>> for the list, so I'm emailing you directly. (I subscribed to the
>> python-dev list and email the faq magic subject but the reply said
>> "no info".) Anyway...
>>
>> I know of two leaks in Python 3.2.10 / Apache 2.2.3 that I would
>> dearly love to get fixed. One of them I have identified the
>> source and made a change to my copy and it works (and comports
>> with the python embedding documentation) so I was hoping you could
>> either get it in or help me understand how I can get it applied.
>>
>> The first leak is if you use the _repr_ function of a dict or list
>> (which is as simple as using str([]) in your mod_python handler!)
>> The repr functions use a dict in the python threadstate to avoid
>> some recursion problem. This dict is allocated only on demand, so
>> while I found str([]) (or a str of any dict or list) to cause its
>> allocation, there may be other user code things you can do to get
>> it allocated. To free it, the python embedder need to call
>> "PyThreadState_Clear" before calling "PyThreadState_Delete". Yet
>> mod_python.c release_interpreter() does not call
>> "PyThreadState_Clear". The fixed version of release_interpreter
>> is this:
>>
>> static void release_interpreter(void)
>> {
>> PyThreadState *tstate = PyThreadState_Get();
>> #ifdef WITH_THREAD
>> PyEval_ReleaseThread(tstate);
>> #else
>> PyThreadState_Swap(NULL);
>> #endif
>> // JSR
>> PyThreadState_Clear(tstate);
>> PyThreadState_Delete(tstate);
>> }
Still looking at this, but where you do it looks to be wrong. The
documentation
below for PyThreadState_Clear() says:
The interpreter lock must be held.
Where you are calling that function, it isn't. It may need to be
moved to before
the call to PyEval_ReleaseThread(). Yes/No?
>> The Python embedded doc is clear that this is the correct thing to
>> do:
>> http://docs.python.org/api/threads.html
>> ...
>> void PyThreadState_Clear( PyThreadState *tstate)
>>
>> Reset all information in a thread state object. The interpreter
>> lock must be held.
>>
>> void PyThreadState_Delete( PyThreadState *tstate)
>>
>> Destroy a thread state object. The interpreter lock need not be
>> held. The thread state must have been reset with a previous call
>> to PyThreadState_Clear().
Graham
Re: two mod_python leaks
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
For such issues, by all means just send the email direct to the
mailing list.
I have copied this back onto the list and I'll do some research into
what
you describe.
BTW, I don't use Windows, but there are others on the list who do and it
may help if you can explain what methodology you are using to see the
Win32 handle leaks. This will help others to replicate it and then
track it
down.
Thanks.
Graham
On 19/10/2006, at 8:50 PM, Jeff Robbins wrote:
> Graham,
>
> I apologize in advance for not understanding the email etiquette
> for the list, so I'm emailing you directly. (I subscribed to the
> python-dev list and email the faq magic subject but the reply said
> "no info".) Anyway...
>
> I know of two leaks in Python 3.2.10 / Apache 2.2.3 that I would
> dearly love to get fixed. One of them I have identified the source
> and made a change to my copy and it works (and comports with the
> python embedding documentation) so I was hoping you could either
> get it in or help me understand how I can get it applied.
>
> The first leak is if you use the _repr_ function of a dict or list
> (which is as simple as using str([]) in your mod_python handler!)
> The repr functions use a dict in the python threadstate to avoid
> some recursion problem. This dict is allocated only on demand, so
> while I found str([]) (or a str of any dict or list) to cause its
> allocation, there may be other user code things you can do to get
> it allocated. To free it, the python embedder need to call
> "PyThreadState_Clear" before calling "PyThreadState_Delete". Yet
> mod_python.c release_interpreter() does not call
> "PyThreadState_Clear". The fixed version of release_interpreter is
> this:
>
> static void release_interpreter(void)
> {
> PyThreadState *tstate = PyThreadState_Get();
> #ifdef WITH_THREAD
> PyEval_ReleaseThread(tstate);
> #else
> PyThreadState_Swap(NULL);
> #endif
> // JSR
> PyThreadState_Clear(tstate);
> PyThreadState_Delete(tstate);
> }
>
>
>
> The Python embedded doc is clear that this is the correct thing to do:
> http://docs.python.org/api/threads.html
> ...
> void PyThreadState_Clear( PyThreadState *tstate)
>
> Reset all information in a thread state object. The interpreter
> lock must be held.
>
> void PyThreadState_Delete( PyThreadState *tstate)
>
> Destroy a thread state object. The interpreter lock need not be
> held. The thread state must have been reset with a previous call to
> PyThreadState_Clear().
> ...
>
>
>
>
>
> The second leak, unfortunately, I have not been able to fix. It is
> in the parent httpd process on Win32 when you do a "restart" of
> apache. mod_python causes the leak of 4 Win32 Events (which are
> likely allocated via use of various apr_ functions like the mutex
> functions) This is reproducible and easily visible in the windows
> task manager if you look at the "handles" column and use either the
> command line or the apache monitor to restart apache. On my
> installation, mod_python contributes 4 of the 5 leaking handles.
> (Someone else leaks one also!!!) This harms a long-term deployment
> which uses the apache maxrequests feature to restart the child
> every million requests.
>
> Thank you in advance for you patience with this long-winded email!
>
> Best regards,
>
> Jeff Robbins
>
>
> ----- Original Message ----- From: "Graham Dumpleton"
> <gr...@dscpl.com.au>
> To: <py...@httpd.apache.org>
> Sent: Wednesday, October 18, 2006 20:47
> Subject: Status of mod_python 3.3.
>
>
>> On JIRA, the following issues are still marked as incomplete for
>> mod_python
>> version 3.3. I have noted my own comments about where they are up
>> to and
>> what I think still needs to be done.
>>
>> MODPYTHON-93 Improve util.FieldStorage efficiency.
>>
>> This was actually marked as resolved but reopened because it was
>> discovered
>> that changes meant that Trac <=0.9.6 would no longer work. The
>> changes were
>> also backed out of mod_python 3.2.X branch and not released in
>> 3.2.10.
>>
>> At this point I believe we have agreed that code in 3.3 would be
>> left as is and
>> people would need to use Trac >=0.10, which has now been release,
>> with
>> mod_python 3.3 or later.
>>
>> There was comments as to whether util.FieldStorage needs to have more
>> dictionary like access, but at this point I believe we should mark
>> this issue
>> as resolved and if people want dictionary like access, they can
>> open a
>> separate JIRA issue for that and we deal with it in a future release.
>>
>> In summary, I believe we should mark this as resolved.
>>
>> MODPYTHON-104 Allow Python code callouts with mod_include (SSI).
>>
>> The code for all this has been done for some time. The only reason
>> it hasn't
>> been marked as resolved as no documentation has been added into core
>> mod_python documentation. I have separately written a article on
>> the new
>> feature which is available at:
>>
>> http://www.dscpl.com.au/wiki/ModPython/Articles/
>> BasicsOfServerSideIncludes
>>
>> I have no problem as this being used as basis for core documentation.
>> I had been holding off integrating it because of contention over
>> whether we
>> could use wiki for documentation or not.
>>
>> In summary, need to still keep this open until some documentation
>> added
>> to core mod_python documentation.
>>
>> MODPYTHON-127 Use namespace for mod_python PythonOption settings.
>>
>> I have made code changes but not committed them back to
>> repository. Jim
>> has committed some documentation changes related to it already
>> though.
>> Some more documentation changes are probably required where
>> options are
>> mentioned in relation to features they affect.
>>
>> I had posed question about whether
>> mod_python.session.database_directory
>> should also be added as a general fallback in cases where which
>> type of
>> filesystem based session was not going to be known. Jim responded
>> with +1,
>> but his explicit vs implicit comment made me unsure which proposal he
>> was agreeing with. Thus nothing done about that yet.
>>
>> In summary, bit more work to do.
>>
>> MODPYTHON-143 Implement and integrate a new module importer.
>>
>> Code has been done, except for extra bit more logging of
>> exceptions for when
>> modules hooks are called and a problem occurs. Also need to update
>> the
>> documentation.
>>
>> In summary, more work to do but mainly documentation.
>>
>> MODPYTHON-186 Build process not using correct values from Python
>> config Makefile.
>>
>> This is only known to be an issue on Mac OS X when the very latest
>> compiler tool chain software given out at Mac developer conference
>> is used. It may only affect new Intel Macs. Someone did suggest they
>> would come back with required changes but that hasn't happened.
>>
>> Personally I would say we don't attempt to address this in 3.3 and
>> defer
>> it till later.
>>
>> MODPYTHON-190 Python 2.5 support.
>>
>> From what I have seen, people are already using Python 2.5, thus
>> is there
>> any urgency on this? All I can figure is that by not making
>> changes you will
>> not be able to work with really large data, or will it all crash
>> badly on a 64
>> bit platform with 64 bit support compiled in.
>>
>> I don't know the answers to this and no one else (doesn't even
>> have to be
>> one of the committers) has stepped up to do the work and work out
>> what
>> the required changes are.
>>
>> Personally I would say we don't attempt to address this in 3.3 and
>> defer
>> it till later.
>>
>> MODPYTHON-193 Add req.hlist.location to mirror req.hlist.directory.
>>
>> I have done most of the code for this and now just sorting out some
>> problems with trailing slashes getting added when they shouldn't.
>> Also
>> need to still update Session code and ensure None is returned for
>> handler
>> directory when not used in a directory.
>>
>> In summary, a bit more work to do.
>>
>> So that is where we are at. Can I get some agreement that we will:
>>
>> 1. Mark MODPYTHON-93 as resolved.
>> 2. Defer MODPYTHON-186 and MODPYTHON-190 till later.
>>
>> Finally, is there anything else in outstanding issues (not listed
>> here)
>> that people believe need to be address for mod_python 3.3?
>>
>> Graham