You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mod_python-dev@quetz.apache.org by "Graham Dumpleton (JIRA)" <ji...@apache.org> on 2005/11/05 05:54:19 UTC
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Graham Dumpleton updated MODPYTHON-77:
--------------------------------------
Attachment: grahamd_20051105.tar.gz
Here is my first go at an alternate patch for this problem. Patch was made against SVN head, believed to be 3.2.4b.
All the change in effect does is save the first interpreter as main_interpreter, but most importantly it does this in Apache child process and not in the parent process before the fork occurs as was the case before. This avoids problems on Mac OS X where Apache would crash on "restart" and on Linux where Apache would crash after the request had been handled.
Note that this change doesn't use any of the PEP GIL specific calls nor does it do anything specific to make anything work on Python 2.3.5. Except for moving one thread state swap call from the parent process context to the child process context, all thread management code is the same.
The changes work fine on:
Mac OS X (10.3.9) / Apache 2.0.51 (worker) / Python 2.3 (Apple OS Installed)
Linux Fedora Code 2 / Apache 2.0.55 (prefork) / Python 2.3.5
Test example was gilstate.tar.gz attached to MODPYTHON-77.
Also passed on mod_python/test suite on Mac OS X. There were failures of test suite on Linux, but those failures occurred before patches were applied as well.
The changes need to be tested on Win32 Apache as well as system where no thread support compiled into Python.
For those of you following this issue, if you can test this change, indicate if it works or fails and if it fails indicate specifically how it is failing. From any failures can then start to understand the other changes Boyan has made and what is required and what isn't.
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, grahamd_20051105.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: Linux FC 2 Test Failures (3.2.4b)
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
On 06/11/2005, at 11:55 AM, Gregory (Grisha) Trubetskoy wrote:
>
> On Sun, 6 Nov 2005, Graham Dumpleton wrote:
>
>> Haven't had a chance to investigate yet and ensure they aren't caused
>> by me
>> using versions of both Python and Apache not in standard locations.
>> Most
>> tests work though. The tests that fail are:
>>
>> ======================================================================
>> FAIL: test_connectionhandler (__main__.PerRequestTestCase)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> File "test.py", line 962, in test_connectionhandler
>> self.fail(`rsp`)
>> File "/home/grahamd/testing/lib/python2.3/unittest.py", line 270, in
>> fail
>> raise self.failureException, msg
>> AssertionError: '/home/grahamd/src/mod_python-3.2.4b/test/htdocs'
>>
>> ----------------------------------------------------------------------
>> Ran 38 tests in 10.173s
>
> If this is in an OpenVPS, then it has to do with 127.0.0.1 being
> transparently redirected to your IP, so what's in the log is not what
> is expected, or something to that extent - don't remember now, but
> it's all good, I wouldn't worry about this one.
That could possibly be it then. Had wandered if the OpenVPS environment
might have something to do with it, but didn't then understand why only
certain tests would be affected and not others. When I get a chance I
will still try and look into it, extracting the particular tests and
running them through a normal running version of Apache and not in a
loopback arrangement.
Graham
Re: Linux FC 2 Test Failures (3.2.4b)
Posted by "Gregory (Grisha) Trubetskoy" <gr...@apache.org>.
On Sun, 6 Nov 2005, Graham Dumpleton wrote:
> Haven't had a chance to investigate yet and ensure they aren't caused by me
> using versions of both Python and Apache not in standard locations. Most
> tests work though. The tests that fail are:
>
> ======================================================================
> FAIL: test_connectionhandler (__main__.PerRequestTestCase)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "test.py", line 962, in test_connectionhandler
> self.fail(`rsp`)
> File "/home/grahamd/testing/lib/python2.3/unittest.py", line 270, in fail
> raise self.failureException, msg
> AssertionError: '/home/grahamd/src/mod_python-3.2.4b/test/htdocs'
>
> ----------------------------------------------------------------------
> Ran 38 tests in 10.173s
If this is in an OpenVPS, then it has to do with 127.0.0.1 being
transparently redirected to your IP, so what's in the log is not what is
expected, or something to that extent - don't remember now, but it's all
good, I wouldn't worry about this one.
Grisha
Re: Linux FC 2 Test Failures (3.2.4b)
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
On 06/11/2005, at 2:42 AM, Jim Gallacher wrote:
>> The changes work fine on:
>> Mac OS X (10.3.9) / Apache 2.0.51 (worker) / Python 2.3 (Apple OS
>> Installed)
>> Linux Fedora Code 2 / Apache 2.0.55 (prefork) / Python 2.3.5
>> Test example was gilstate.tar.gz attached to MODPYTHON-77.
>> Also passed on mod_python/test suite on Mac OS X. There were failures
>> of test suite on Linux, but those failures occurred before patches
>> were applied as well.
>
> Failures? What failures?
Haven't had a chance to investigate yet and ensure they aren't caused
by me
using versions of both Python and Apache not in standard locations. Most
tests work though. The tests that fail are:
======================================================================
FAIL: test_connectionhandler (__main__.PerRequestTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test.py", line 962, in test_connectionhandler
self.fail(`rsp`)
File "/home/grahamd/testing/lib/python2.3/unittest.py", line 270, in
fail
raise self.failureException, msg
AssertionError: '/home/grahamd/src/mod_python-3.2.4b/test/htdocs'
----------------------------------------------------------------------
Ran 38 tests in 10.173s
FAILED (failures=1)
F Stopping Apache...
/home/grahamd/testing/bin/httpd -k stop -f
/home/grahamd/src/mod_python-3.2.4b/test/conf/test.conf
======================================================================
FAIL: testPerRequestTests (__main__.PerInstanceTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test.py", line 1488, in testPerRequestTests
self.failUnless(result.wasSuccessful())
File "/home/grahamd/testing/lib/python2.3/unittest.py", line 278, in
failUnless
if not expr: raise self.failureException, msg
AssertionError
----------------------------------------------------------------------
Ran 4 tests in 26.620s
FAILED (failures=1)
Re: [jira] Updated: (MODPYTHON-77) The multiple interpreter concept
of mod_python is broken for Python extension modules since Python 2.3
Posted by Jim Gallacher <jp...@jgassociates.ca>.
Graham Dumpleton (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
>
> Graham Dumpleton updated MODPYTHON-77:
> --------------------------------------
>
> Attachment: grahamd_20051105.tar.gz
>
> Here is my first go at an alternate patch for this problem. Patch was made against SVN head, believed to be 3.2.4b.
>
> All the change in effect does is save the first interpreter as main_interpreter, but most importantly it does this in Apache child process and not in the parent process before the fork occurs as was the case before. This avoids problems on Mac OS X where Apache would crash on "restart" and on Linux where Apache would crash after the request had been handled.
>
> Note that this change doesn't use any of the PEP GIL specific calls nor does it do anything specific to make anything work on Python 2.3.5. Except for moving one thread state swap call from the parent process context to the child process context, all thread management code is the same.
>
> The changes work fine on:
>
> Mac OS X (10.3.9) / Apache 2.0.51 (worker) / Python 2.3 (Apple OS Installed)
> Linux Fedora Code 2 / Apache 2.0.55 (prefork) / Python 2.3.5
>
> Test example was gilstate.tar.gz attached to MODPYTHON-77.
>
> Also passed on mod_python/test suite on Mac OS X. There were failures of test suite on Linux, but those failures occurred before patches were applied as well.
Failures? What failures?
Jim