You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mod_python-dev@quetz.apache.org by "Boyan Boyadjiev (JIRA)" <ji...@apache.org> on 2005/09/02 11:26:18 UTC
[jira] Created: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
------------------------------------------------------------------------------------------------------
Key: MODPYTHON-77
URL: http://issues.apache.org/jira/browse/MODPYTHON-77
Project: mod_python
Type: Bug
Components: core
Versions: 3.1.4
Environment: Python >= 2.3
Reporter: Boyan Boyadjiev
The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
...
Limitations and Exclusions
This proposal identifies a solution for extension authors with
complex multi-threaded requirements, but that only require a
single "PyInterpreterState". There is no attempt to cater for
extensions that require multiple interpreter states. At the time
of writing, no extension has been identified that requires
multiple PyInterpreterStates, and indeed it is not clear if that
facility works correctly in Python itself.
...
For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
apr_pool_t *plog, server_rec *s)
{
...
/* initialize global Python interpreter if necessary */
if (! Py_IsInitialized())
{
/* initialze the interpreter */
Py_Initialize();
#ifdef WITH_THREAD
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
/* create the obCallBack dictionary */
interpreters = PyDict_New();
if (! interpreters) {
ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
"python_init: PyDict_New() failed! No more memory?");
exit(1);
}
{
/*
Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
BEGIN
*/
PyObject *p = 0;
interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
PyThreadState* currentThreadState = PyThreadState_Get();
PyInterpreterState *istate = currentThreadState->interp;
idata->istate = istate;
/* obcallback will be created on first use */
idata->obcallback = NULL;
p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
Py_DECREF(p); /*p->refcout = 1*/
/*
END
Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
*/
}
/* Release the thread state because we will never use
* the main interpreter, only sub interpreters created later. */
PyThreadState_Swap(NULL);
#ifdef WITH_THREAD
/* release the lock; now other threads can run */
PyEval_ReleaseLock();
#endif
}
return OK;
}
Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (MODPYTHON-77) The multiple interpreter concept
of mod_python is broken for Python extension modules since Python 2.3
Posted by Jim Gallacher <jp...@jgassociates.ca>.
Doh!
No, I didn't apply the patches. I had a feeling I was forgetting
*something* when I ran test, but I couldn't put my finger on it. :)
I will patch and retest.
Jim
Graham Dumpleton wrote:
> Jim, is this just confirming that test shows a problem with unpatched 3.2
> or when Boyan's GIL state fixes are also applied? I haven't tried the
> suggested patches yet, as wanted an answer as to why GIL state API had
> to be
> explicitly used, which he has now done so.
>
> Graham
>
> On 03/11/2005, at 2:45 AM, Jim Gallacher (JIRA) wrote:
>
>> [ http://issues.apache.org/jira/browse/MODPYTHON-77?
>> page=comments#action_12356614 ]
>>
>> Jim Gallacher commented on MODPYTHON-77:
>> ----------------------------------------
>>
>> Tested gilstate.tar.gz on mpm-prefork. Error log output:
>>
>> [Wed Nov 02 10:40:20 2005] [notice] Apache/2.0.54 (Debian GNU/Linux)
>> DAV/2 SVN/1.2.0 mod_python/3.2.4b Python/2.3.5 PHP/4.4.0-1 configured
>> -- resuming normal operations
>> [Wed Nov 02 10:40:57 2005] [notice] mod_python: (Re)importing module
>> 'mptest'
>> [Wed Nov 02 10:40:57 2005] [error] BEGIN: CALLBACK
>> [Wed Nov 02 10:40:57 2005] [error] EXCEPTION: file() constructor not
>> accessible in restricted mode
>> [Wed Nov 02 10:40:57 2005] [error] FINISH: CALLBACK
>>
>>
>>> The multiple interpreter concept of mod_python is broken for Python
>>> extension modules since Python 2.3
>>> ----------------------------------------------------------------------
>>> --------------------------------
>>>
>>> Key: MODPYTHON-77
>>> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
>>> Project: mod_python
>>> Type: Bug
>>> Components: core
>>> Versions: 3.1.4
>>> Environment: Python >= 2.3
>>> Reporter: Boyan Boyadjiev
>>> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c,
>>> gilstate.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff,
>>> src.zip
>>>
>>> The multiple interpreter concept of mod_python is broken for Python
>>> extension modules since Python 2.3 because of the PEP 311
>>> (Simplified Global Interpreter Lock Acquisition for Extensions):
>>> ...
>>> Limitations and Exclusions
>>> This proposal identifies a solution for extension authors with
>>> complex multi-threaded requirements, but that only require a
>>> single "PyInterpreterState". There is no attempt to cater for
>>> extensions that require multiple interpreter states. At the time
>>> of writing, no extension has been identified that requires
>>> multiple PyInterpreterStates, and indeed it is not clear if that
>>> facility works correctly in Python itself.
>>> ...
>>> For mod_python this means, that complex Python extensions won't work
>>> any more with Python >= 2.3, because they are supposed to work only
>>> with the first interpreter state initialized for the current process
>>> (a problem we experienced). The first interpreter state is not used
>>> by mod_python after the python_init is called.
>>> One solution, which works fine for me, is to save the first
>>> interpreter state into the "interpreters" dictionary in the function
>>> python_init (MAIN_INTERPRETER is used as a key):
>>> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
>>> apr_pool_t *plog, server_rec *s)
>>> {
>>> ...
>>> /* initialize global Python interpreter if necessary */
>>> if (! Py_IsInitialized())
>>> {
>>> /* initialze the interpreter */
>>> Py_Initialize();
>>> #ifdef WITH_THREAD
>>> /* create and acquire the interpreter lock */
>>> PyEval_InitThreads();
>>> #endif
>>> /* create the obCallBack dictionary */
>>> interpreters = PyDict_New();
>>> if (! interpreters) {
>>> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
>>> "python_init: PyDict_New() failed! No more
>>> memory?");
>>> exit(1);
>>> }
>>> {
>>> /*
>>> Workaround PEP 311 - Simplified Global Interpreter Lock
>>> Acquisition for Extensions
>>> BEGIN
>>> */
>>> PyObject *p = 0;
>>> interpreterdata * idata = (interpreterdata
>>> *)malloc(sizeof(interpreterdata));
>>> PyThreadState* currentThreadState = PyThreadState_Get();
>>> PyInterpreterState *istate = currentThreadState->interp;
>>> idata->istate = istate;
>>> /* obcallback will be created on first use */
>>> idata->obcallback = NULL;
>>> p = PyCObject_FromVoidPtr((void ) idata, NULL);
>>> /*p->refcout = 1*/
>>> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p);
>>> /*p->refcout = 2*/
>>> Py_DECREF(p); /*p->refcout = 1*/
>>> /*
>>> END
>>> Workaround PEP 311 - Simplified Global Interpreter Lock
>>> Acquisition for Extensions
>>> */
>>> }
>>> /* Release the thread state because we will never use
>>> * the main interpreter, only sub interpreters created
>>> later. */
>>> PyThreadState_Swap(NULL);
>>> #ifdef WITH_THREAD
>>> /* release the lock; now other threads can run */
>>> PyEval_ReleaseLock();
>>> #endif
>>> }
>>> return OK;
>>> }
>>> Another change I've made in the attached file is to Py_DECREF(p) in
>>> get_interpreter, which will remove leaky reference to the PyCObject
>>> with the interpreter data. This was not a real problem, but now I
>>> see fewer leaks in BoundsChecker :-).
>>
>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> If you think it was sent incorrectly contact one of the administrators:
>> http://issues.apache.org/jira/secure/Administrators.jspa
>> -
>> For more information on JIRA, see:
>> http://www.atlassian.com/software/jira
>
>
Re: [jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
Jim, is this just confirming that test shows a problem with unpatched
3.2
or when Boyan's GIL state fixes are also applied? I haven't tried the
suggested patches yet, as wanted an answer as to why GIL state API had
to be
explicitly used, which he has now done so.
Graham
On 03/11/2005, at 2:45 AM, Jim Gallacher (JIRA) wrote:
> [
> http://issues.apache.org/jira/browse/MODPYTHON-77?
> page=comments#action_12356614 ]
>
> Jim Gallacher commented on MODPYTHON-77:
> ----------------------------------------
>
> Tested gilstate.tar.gz on mpm-prefork. Error log output:
>
> [Wed Nov 02 10:40:20 2005] [notice] Apache/2.0.54 (Debian GNU/Linux)
> DAV/2 SVN/1.2.0 mod_python/3.2.4b Python/2.3.5 PHP/4.4.0-1 configured
> -- resuming normal operations
> [Wed Nov 02 10:40:57 2005] [notice] mod_python: (Re)importing module
> 'mptest'
> [Wed Nov 02 10:40:57 2005] [error] BEGIN: CALLBACK
> [Wed Nov 02 10:40:57 2005] [error] EXCEPTION: file() constructor not
> accessible in restricted mode
> [Wed Nov 02 10:40:57 2005] [error] FINISH: CALLBACK
>
>
>> The multiple interpreter concept of mod_python is broken for Python
>> extension modules since Python 2.3
>> ----------------------------------------------------------------------
>> --------------------------------
>>
>> Key: MODPYTHON-77
>> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
>> Project: mod_python
>> Type: Bug
>> Components: core
>> Versions: 3.1.4
>> Environment: Python >= 2.3
>> Reporter: Boyan Boyadjiev
>> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c,
>> gilstate.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff,
>> src.zip
>>
>> The multiple interpreter concept of mod_python is broken for Python
>> extension modules since Python 2.3 because of the PEP 311 (Simplified
>> Global Interpreter Lock Acquisition for Extensions):
>> ...
>> Limitations and Exclusions
>> This proposal identifies a solution for extension authors with
>> complex multi-threaded requirements, but that only require a
>> single "PyInterpreterState". There is no attempt to cater for
>> extensions that require multiple interpreter states. At the time
>> of writing, no extension has been identified that requires
>> multiple PyInterpreterStates, and indeed it is not clear if that
>> facility works correctly in Python itself.
>> ...
>> For mod_python this means, that complex Python extensions won't work
>> any more with Python >= 2.3, because they are supposed to work only
>> with the first interpreter state initialized for the current process
>> (a problem we experienced). The first interpreter state is not used
>> by mod_python after the python_init is called.
>> One solution, which works fine for me, is to save the first
>> interpreter state into the "interpreters" dictionary in the function
>> python_init (MAIN_INTERPRETER is used as a key):
>> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
>> apr_pool_t *plog, server_rec *s)
>> {
>> ...
>> /* initialize global Python interpreter if necessary */
>> if (! Py_IsInitialized())
>> {
>> /* initialze the interpreter */
>> Py_Initialize();
>> #ifdef WITH_THREAD
>> /* create and acquire the interpreter lock */
>> PyEval_InitThreads();
>> #endif
>> /* create the obCallBack dictionary */
>> interpreters = PyDict_New();
>> if (! interpreters) {
>> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
>> "python_init: PyDict_New() failed! No more
>> memory?");
>> exit(1);
>> }
>> {
>> /*
>> Workaround PEP 311 - Simplified Global Interpreter Lock
>> Acquisition for Extensions
>> BEGIN
>> */
>> PyObject *p = 0;
>> interpreterdata * idata = (interpreterdata
>> *)malloc(sizeof(interpreterdata));
>> PyThreadState* currentThreadState = PyThreadState_Get();
>> PyInterpreterState *istate = currentThreadState->interp;
>> idata->istate = istate;
>> /* obcallback will be created on first use */
>> idata->obcallback = NULL;
>> p = PyCObject_FromVoidPtr((void ) idata, NULL);
>> /*p->refcout = 1*/
>> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p);
>> /*p->refcout = 2*/
>> Py_DECREF(p); /*p->refcout = 1*/
>> /*
>> END
>> Workaround PEP 311 - Simplified Global Interpreter Lock
>> Acquisition for Extensions
>> */
>> }
>> /* Release the thread state because we will never use
>> * the main interpreter, only sub interpreters created later.
>> */
>> PyThreadState_Swap(NULL);
>> #ifdef WITH_THREAD
>> /* release the lock; now other threads can run */
>> PyEval_ReleaseLock();
>> #endif
>> }
>> return OK;
>> }
>> Another change I've made in the attached file is to Py_DECREF(p) in
>> get_interpreter, which will remove leaky reference to the PyCObject
>> with the interpreter data. This was not a real problem, but now I see
>> fewer leaks in BoundsChecker :-).
>
> --
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the administrators:
> http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:
> http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Jim Gallacher (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12356614 ]
Jim Gallacher commented on MODPYTHON-77:
----------------------------------------
Tested gilstate.tar.gz on mpm-prefork. Error log output:
[Wed Nov 02 10:40:20 2005] [notice] Apache/2.0.54 (Debian GNU/Linux) DAV/2 SVN/1.2.0 mod_python/3.2.4b Python/2.3.5 PHP/4.4.0-1 configured -- resuming normal operations
[Wed Nov 02 10:40:57 2005] [notice] mod_python: (Re)importing module 'mptest'
[Wed Nov 02 10:40:57 2005] [error] BEGIN: CALLBACK
[Wed Nov 02 10:40:57 2005] [error] EXCEPTION: file() constructor not accessible in restricted mode
[Wed Nov 02 10:40:57 2005] [error] FINISH: CALLBACK
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Jim Gallacher (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12320924 ]
Jim Gallacher commented on MODPYTHON-77:
----------------------------------------
-1 for me. Unit tests fail for diff3.txt.
For some reason 1 hunk is rejected when diff3 is applied against svn/trunk. Manually removing PyThreadState_Swap as suggested by Boyan does not help. It seems like apache is segfaulting. The only thing in the error log is the following:
[Fri Sep 02 14:31:04 2005] [notice] mod_python: Creating 8 session mutexes based on 256 max processes and 0 max threads.
at which point the apache process dies.
Which mpm are you using, Boyan?
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: Linux FC 2 Test Failures (3.2.4b)
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
On 06/11/2005, at 11:55 AM, Gregory (Grisha) Trubetskoy wrote:
>
> On Sun, 6 Nov 2005, Graham Dumpleton wrote:
>
>> Haven't had a chance to investigate yet and ensure they aren't caused
>> by me
>> using versions of both Python and Apache not in standard locations.
>> Most
>> tests work though. The tests that fail are:
>>
>> ======================================================================
>> FAIL: test_connectionhandler (__main__.PerRequestTestCase)
>> ----------------------------------------------------------------------
>> Traceback (most recent call last):
>> File "test.py", line 962, in test_connectionhandler
>> self.fail(`rsp`)
>> File "/home/grahamd/testing/lib/python2.3/unittest.py", line 270, in
>> fail
>> raise self.failureException, msg
>> AssertionError: '/home/grahamd/src/mod_python-3.2.4b/test/htdocs'
>>
>> ----------------------------------------------------------------------
>> Ran 38 tests in 10.173s
>
> If this is in an OpenVPS, then it has to do with 127.0.0.1 being
> transparently redirected to your IP, so what's in the log is not what
> is expected, or something to that extent - don't remember now, but
> it's all good, I wouldn't worry about this one.
That could possibly be it then. Had wandered if the OpenVPS environment
might have something to do with it, but didn't then understand why only
certain tests would be affected and not others. When I get a chance I
will still try and look into it, extracting the particular tests and
running them through a normal running version of Apache and not in a
loopback arrangement.
Graham
Re: Linux FC 2 Test Failures (3.2.4b)
Posted by "Gregory (Grisha) Trubetskoy" <gr...@apache.org>.
On Sun, 6 Nov 2005, Graham Dumpleton wrote:
> Haven't had a chance to investigate yet and ensure they aren't caused by me
> using versions of both Python and Apache not in standard locations. Most
> tests work though. The tests that fail are:
>
> ======================================================================
> FAIL: test_connectionhandler (__main__.PerRequestTestCase)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "test.py", line 962, in test_connectionhandler
> self.fail(`rsp`)
> File "/home/grahamd/testing/lib/python2.3/unittest.py", line 270, in fail
> raise self.failureException, msg
> AssertionError: '/home/grahamd/src/mod_python-3.2.4b/test/htdocs'
>
> ----------------------------------------------------------------------
> Ran 38 tests in 10.173s
If this is in an OpenVPS, then it has to do with 127.0.0.1 being
transparently redirected to your IP, so what's in the log is not what is
expected, or something to that extent - don't remember now, but it's all
good, I wouldn't worry about this one.
Grisha
Re: Linux FC 2 Test Failures (3.2.4b)
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
On 06/11/2005, at 2:42 AM, Jim Gallacher wrote:
>> The changes work fine on:
>> Mac OS X (10.3.9) / Apache 2.0.51 (worker) / Python 2.3 (Apple OS
>> Installed)
>> Linux Fedora Code 2 / Apache 2.0.55 (prefork) / Python 2.3.5
>> Test example was gilstate.tar.gz attached to MODPYTHON-77.
>> Also passed on mod_python/test suite on Mac OS X. There were failures
>> of test suite on Linux, but those failures occurred before patches
>> were applied as well.
>
> Failures? What failures?
Haven't had a chance to investigate yet and ensure they aren't caused
by me
using versions of both Python and Apache not in standard locations. Most
tests work though. The tests that fail are:
======================================================================
FAIL: test_connectionhandler (__main__.PerRequestTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test.py", line 962, in test_connectionhandler
self.fail(`rsp`)
File "/home/grahamd/testing/lib/python2.3/unittest.py", line 270, in
fail
raise self.failureException, msg
AssertionError: '/home/grahamd/src/mod_python-3.2.4b/test/htdocs'
----------------------------------------------------------------------
Ran 38 tests in 10.173s
FAILED (failures=1)
F Stopping Apache...
/home/grahamd/testing/bin/httpd -k stop -f
/home/grahamd/src/mod_python-3.2.4b/test/conf/test.conf
======================================================================
FAIL: testPerRequestTests (__main__.PerInstanceTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test.py", line 1488, in testPerRequestTests
self.failUnless(result.wasSuccessful())
File "/home/grahamd/testing/lib/python2.3/unittest.py", line 278, in
failUnless
if not expr: raise self.failureException, msg
AssertionError
----------------------------------------------------------------------
Ran 4 tests in 26.620s
FAILED (failures=1)
Re: [jira] Updated: (MODPYTHON-77) The multiple interpreter concept
of mod_python is broken for Python extension modules since Python 2.3
Posted by Jim Gallacher <jp...@jgassociates.ca>.
Graham Dumpleton (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
>
> Graham Dumpleton updated MODPYTHON-77:
> --------------------------------------
>
> Attachment: grahamd_20051105.tar.gz
>
> Here is my first go at an alternate patch for this problem. Patch was made against SVN head, believed to be 3.2.4b.
>
> All the change in effect does is save the first interpreter as main_interpreter, but most importantly it does this in Apache child process and not in the parent process before the fork occurs as was the case before. This avoids problems on Mac OS X where Apache would crash on "restart" and on Linux where Apache would crash after the request had been handled.
>
> Note that this change doesn't use any of the PEP GIL specific calls nor does it do anything specific to make anything work on Python 2.3.5. Except for moving one thread state swap call from the parent process context to the child process context, all thread management code is the same.
>
> The changes work fine on:
>
> Mac OS X (10.3.9) / Apache 2.0.51 (worker) / Python 2.3 (Apple OS Installed)
> Linux Fedora Code 2 / Apache 2.0.55 (prefork) / Python 2.3.5
>
> Test example was gilstate.tar.gz attached to MODPYTHON-77.
>
> Also passed on mod_python/test suite on Mac OS X. There were failures of test suite on Linux, but those failures occurred before patches were applied as well.
Failures? What failures?
Jim
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Graham Dumpleton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Graham Dumpleton updated MODPYTHON-77:
--------------------------------------
Attachment: grahamd_20051105.tar.gz
Here is my first go at an alternate patch for this problem. Patch was made against SVN head, believed to be 3.2.4b.
All the change in effect does is save the first interpreter as main_interpreter, but most importantly it does this in Apache child process and not in the parent process before the fork occurs as was the case before. This avoids problems on Mac OS X where Apache would crash on "restart" and on Linux where Apache would crash after the request had been handled.
Note that this change doesn't use any of the PEP GIL specific calls nor does it do anything specific to make anything work on Python 2.3.5. Except for moving one thread state swap call from the parent process context to the child process context, all thread management code is the same.
The changes work fine on:
Mac OS X (10.3.9) / Apache 2.0.51 (worker) / Python 2.3 (Apple OS Installed)
Linux Fedora Code 2 / Apache 2.0.55 (prefork) / Python 2.3.5
Test example was gilstate.tar.gz attached to MODPYTHON-77.
Also passed on mod_python/test suite on Mac OS X. There were failures of test suite on Linux, but those failures occurred before patches were applied as well.
The changes need to be tested on Win32 Apache as well as system where no thread support compiled into Python.
For those of you following this issue, if you can test this change, indicate if it works or fails and if it fails indicate specifically how it is failing. From any failures can then start to understand the other changes Boyan has made and what is required and what isn't.
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, grahamd_20051105.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Boyan Boyadjiev (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Boyan Boyadjiev updated MODPYTHON-77:
-------------------------------------
Attachment: src.zip
Jim,
could you please try, if the cahnges in the attaced src.zip are working for you. If so this will be a much better solution.
TIA
Boyan
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, mod_python.c, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Boyan Boyadjiev (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Boyan Boyadjiev updated MODPYTHON-77:
-------------------------------------
Attachment: mod_python.h.diff
mod_python.c.diff
mod_python.h.diff and mod_python.c.diff attached:
Changes:
1. Added some comments regarding the thread state changes
2. #ifdef PythonVer>=235 because og the use of new Python APIs.
3. sequence of PyThreadState_Delete(tstate) and PyEval_ReleaseLock() changed - the lock should be held while delete
So now the thread state handling should be completly compatible to Python >= 2.3.5. and some issues should be fixed
e.g. creating thread from python handler will work without limitations for the main interpreter and if you take http://sourceforge.net/tracker/index.php?func=detail&aid=1163563&group_id=5470&atid=105470 also for all other interpreters.
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
Re: [jira] Commented: (MODPYTHON-77) The multiple interpreter concept
of mod_python is broken for Python extension modules since Python 2.3
Posted by Jim Gallacher <jg...@sympatico.ca>.
Boyan,
How critical is this bug with respect to releasing 3.2? I'm heading out
for the weekend in the next couple of hours and was hoping to release
3.2.1beta beforehand. If this is a must fix I'll wait until next week.
(Either way, I'll fix the memory leak get_interpreter which you detail
in MODPYTHON-77).
Regards,
Jim
Jim Gallacher (JIRA) wrote:
> [ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12320905 ]
>
> Jim Gallacher commented on MODPYTHON-77:
> ----------------------------------------
>
> I tried your patch and the unit tests fail. Apache fails to start, with the following message in the error log:
>
> Fatal Python error: PyThreadState_Get: no current thread
>
> Using
> GCC 4.0.2
> apache mpm-prefork 2.0.54
> python 2.3.5
>
> Also, please submit your patches as unified diffs (diff -u) or even better if you are using code checked out of subversion use "svn diff"
>
>
>>The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
>>------------------------------------------------------------------------------------------------------
>>
>> Key: MODPYTHON-77
>> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
>> Project: mod_python
>> Type: Bug
>> Components: core
>> Versions: 3.1.4
>> Environment: Python >= 2.3
>> Reporter: Boyan Boyadjiev
>> Attachments: diff.txt, mod_python.c
>>
>>The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
>>...
>>Limitations and Exclusions
>> This proposal identifies a solution for extension authors with
>> complex multi-threaded requirements, but that only require a
>> single "PyInterpreterState". There is no attempt to cater for
>> extensions that require multiple interpreter states. At the time
>> of writing, no extension has been identified that requires
>> multiple PyInterpreterStates, and indeed it is not clear if that
>> facility works correctly in Python itself.
>>...
>>For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
>>One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
>>static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
>> apr_pool_t *plog, server_rec *s)
>>{
>> ...
>> /* initialize global Python interpreter if necessary */
>> if (! Py_IsInitialized())
>> {
>> /* initialze the interpreter */
>> Py_Initialize();
>>#ifdef WITH_THREAD
>> /* create and acquire the interpreter lock */
>> PyEval_InitThreads();
>>#endif
>> /* create the obCallBack dictionary */
>> interpreters = PyDict_New();
>> if (! interpreters) {
>> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
>> "python_init: PyDict_New() failed! No more memory?");
>> exit(1);
>> }
>> {
>> /*
>> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
>> BEGIN
>> */
>> PyObject *p = 0;
>> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
>> PyThreadState* currentThreadState = PyThreadState_Get();
>> PyInterpreterState *istate = currentThreadState->interp;
>> idata->istate = istate;
>> /* obcallback will be created on first use */
>> idata->obcallback = NULL;
>> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
>> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
>> Py_DECREF(p); /*p->refcout = 1*/
>> /*
>> END
>> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
>> */
>> }
>> /* Release the thread state because we will never use
>> * the main interpreter, only sub interpreters created later. */
>> PyThreadState_Swap(NULL);
>>#ifdef WITH_THREAD
>> /* release the lock; now other threads can run */
>> PyEval_ReleaseLock();
>>#endif
>> }
>> return OK;
>>}
>>Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
>
>
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Jim Gallacher (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12320905 ]
Jim Gallacher commented on MODPYTHON-77:
----------------------------------------
I tried your patch and the unit tests fail. Apache fails to start, with the following message in the error log:
Fatal Python error: PyThreadState_Get: no current thread
Using
GCC 4.0.2
apache mpm-prefork 2.0.54
python 2.3.5
Also, please submit your patches as unified diffs (diff -u) or even better if you are using code checked out of subversion use "svn diff"
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Nicolas Lehuen (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12320927 ]
Nicolas Lehuen commented on MODPYTHON-77:
-----------------------------------------
Well in fact I applied the patch manually and it runs on Win32 (threaded MPM), at least on the unit tests.
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Boyan Boyadjiev (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12320928 ]
Boyan Boyadjiev commented on MODPYTHON-77:
------------------------------------------
I use mpm_winnt with 2.0.54
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Graham Dumpleton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12356739 ]
Graham Dumpleton commented on MODPYTHON-77:
-------------------------------------------
There are now so many different suggestions on this that it is all too confusing.
In respect of latest suggested change to release_interpreter(), this code wasn't based on latest code from 3.2 betas and therefore doesn't include fix for where Python doesn't support threading at all. Ie., the base code on which it should have been made was:
static void release_interpreter(void)
{
PyThreadState *tstate = PyThreadState_Get();
#ifdef WITH_THREAD
PyEval_ReleaseThread(tstate);
#else
PyThreadState_Swap(NULL);
#endif
PyThreadState_Delete(tstate);
}
Ie., new call to PyThreadState_Swap() not factored in.
I will refrain from posting what it perhaps should be so as not to make things worse.
In terms of the comment "smallest required change", I am currently taking this to mean that the most minimal change is still to install first interpreter as "main_interpreter" and then to change release_interpreter() as suggested in previous post.
Before anyone else posts any code, let me independently work out some changes from scratch and thus verify the fixes or not. :-(
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Graham Dumpleton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12356765 ]
Graham Dumpleton commented on MODPYTHON-77:
-------------------------------------------
Just to make this problem even more complicated, due to how Mac OS X handles unloading of dynamically loaded modules, parts of the solution presented here so far may not actually work on that platform.
On Mac OS X, if one takes the initial basic solution of storing the first interpreter as the "main_interpreter" and nothing else, the original problem goes away, however, Apache will crash when "apachectl restart" is run.
The problem is that when a "restart" occurs, Apache doesn't actually kill the parent process. What it does is to unload any loaded Apache modules, such as "mod_python.so" and then it will be reload them again. This results in all global data within "mod_python.so" being thrown away. The "python_init()" function of the mod_python.so will be called again to reinitialise all the lost global data and start over, but it fails in doing this.
The reason it fails is that although mod_python.so is unloaded, the dynamically loaded Python framework isn't. The first consequence of this is that when calling python_init() the subsequent time, the Python framework is already initialised. Due to a previous workaround incorporated into python_init() specifically because of Mac OS X, this luckily doesn't stop us from at least trying to initialise mod_python again.
In initialising mod_python though, because the previous global data is lost, we need to store the initial interpreter state against main_interpreter again. To do this, "PyThreadState_Get()->interp" is used. For this to work though, it is necessary that the call to "PyEval_InitThreads()" just prior to that point in the code has acquired the global lock and set up a current thread state. This will only occur though for the very first call to PyEval_InitThreads(). Because the Python framework isn't unloaded and Python thus is already in an initialised state, the PyEval_InitThreads() call doesn't do anything. Because there is no valid active thread state, the call to PyThreadState_Get() fails.
One can see this when some extra debugging is added to mod_python. For Mac OS X we get:
# apachectl start
[Fri Nov 04 20:50:17 2005] [notice] mod_python: Creating 8 session mutexes based on 6 max processes and 25 max threads.
[Fri Nov 04 20:50:17 2005] [error] mp_initialized=0
[Fri Nov 04 20:50:17 2005] [error] Py_IsInitialized()=0
[Fri Nov 04 20:50:17 2005] [error] interpreters=0
[Fri Nov 04 20:50:18 2005] [error] save interpreter
[Fri Nov 04 20:50:18 2005] [error] pid=805
[Fri Nov 04 20:50:18 2005] [error] ppid=1
[Fri Nov 04 20:50:18 2005] [error] done save interpreter
[Fri Nov 04 20:50:18 2005] [error] get thread state
[Fri Nov 04 20:50:18 2005] [error] okay got thread state
[Fri Nov 04 20:50:18 2005] [error] released interpreter
[Fri Nov 04 20:50:18 2005] [notice] Apache/2.0.51 (Unix) mod_python/3.2.4b Python/2.3 configured -- resuming normal operations
# apachectl restart
[Fri Nov 04 20:50:26 2005] [warn] child process 807 still did not exit, sending a SIGTERM
[Fri Nov 04 20:50:27 2005] [notice] SIGHUP received. Attempting to restart
[Fri Nov 04 20:50:27 2005] [notice] mod_python: Creating 8 session mutexes based on 2 max processes and 25 max threads.
[Fri Nov 04 20:50:27 2005] [error] mp_initialized=0
[Fri Nov 04 20:50:27 2005] [error] Py_IsInitialized()=1
[Fri Nov 04 20:50:27 2005] [error] interpreters=0
[Fri Nov 04 20:50:27 2005] [error] save interpreter
[Fri Nov 04 20:50:27 2005] [error] pid=805
[Fri Nov 04 20:50:27 2005] [error] ppid=1
Fatal Python error: PyThreadState_Get: no current thread
[Fri Nov 04 20:50:27 2005] [notice] seg fault or similar nasty error detected in the parent process
You can see how it is the same process ID, that Python is already marked as initailised, yet "interpreters" and the "initialized" flag which were global data in mod_python.so have gone back to being 0. Finally, the PyThreadState_Get() function dies.
If one looks at Linux, one instead sees:
# apachectl start
[Fri Nov 04 05:22:46 2005] [notice] mod_python: Creating 8 session mutexes based on 150 max processes and 0 max threads.
[Fri Nov 04 05:22:46 2005] [error] mp_initialized=0
[Fri Nov 04 05:22:46 2005] [error] Py_IsInitialized()=0
[Fri Nov 04 05:22:46 2005] [error] interpreters=0
[Fri Nov 04 05:22:46 2005] [error] save interpreter
[Fri Nov 04 05:22:46 2005] [error] pid=8757
[Fri Nov 04 05:22:46 2005] [error] ppid=1
[Fri Nov 04 05:22:46 2005] [error] done save interpreter
[Fri Nov 04 05:22:46 2005] [error] get thread state
[Fri Nov 04 05:22:46 2005] [error] okay got thread state
[Fri Nov 04 05:22:46 2005] [error] released interpreter
[Fri Nov 04 05:22:46 2005] [notice] Apache/2.0.55 (Unix) mod_python/3.2.4b Python/2.3.5 configured -- resuming normal operations
# apachectl restart
[Fri Nov 04 05:22:55 2005] [notice] SIGHUP received. Attempting to restart
[Fri Nov 04 05:22:55 2005] [notice] mod_python: Creating 8 session mutexes based on 150 max processes and 0 max threads.
[Fri Nov 04 05:22:55 2005] [error] mp_initialized=0
[Fri Nov 04 05:22:55 2005] [error] Py_IsInitialized()=0
[Fri Nov 04 05:22:55 2005] [error] interpreters=0
[Fri Nov 04 05:22:55 2005] [error] save interpreter
[Fri Nov 04 05:22:55 2005] [error] pid=8757
[Fri Nov 04 05:22:55 2005] [error] ppid=1
[Fri Nov 04 05:22:55 2005] [error] done save interpreter
[Fri Nov 04 05:22:55 2005] [error] get thread state
[Fri Nov 04 05:22:55 2005] [error] okay got thread state
[Fri Nov 04 05:22:55 2005] [error] released interpreter
[Fri Nov 04 05:22:55 2005] [notice] Apache/2.0.55 (Unix) mod_python/3.2.4b Python/2.3.5 configured -- resuming normal operations
Note how Python isn't initialised on second time through. This indicates that any instance of Python in Apache is also being unloaded and thus everything starts over from a totally clean slate.
Anyway, that be an additional problem on Mac OS X that will have to be dealt with in finding a solution. Since the issue is in the parent process, am going to investigate whether the initial interpreter should only be stored as main_interpreter in the child process. Ie., in PythonChildInitHandler(). This might avoid this issue.
More reports later. :-)
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Boyan Boyadjiev (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Boyan Boyadjiev updated MODPYTHON-77:
-------------------------------------
Attachment: diff2.txt
Now I've attached the a diff file created with "diff -u". Sorry for the inconvenience.
Jim, please make sure that you have took the complete patch. May be you have not removed the following lines from the python_init function:
- /* Release the thread state because we will never use
- * the main interpreter, only sub interpreters created later. */
- PyThreadState_Swap(NULL);
-
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Graham Dumpleton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12320866 ]
Graham Dumpleton commented on MODPYTHON-77:
-------------------------------------------
There have been no problems up to Python 2.3.4 except that there are some bugs in mod_python which can cause issues. See:
http://www.dscpl.com.au/projects/vampire/patches.html
There are issues with Python 2.3.5 and Python 2.4. This has been identified as a bug in Python itself. See:
http://sourceforge.net/tracker/index.php?func=detail&aid=1163563&group_id=5470&atid=105470
Is this the same issue? Can you clarify what version of Python you are using?
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Boyan Boyadjiev (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Boyan Boyadjiev updated MODPYTHON-77:
-------------------------------------
Attachment: diff.txt
Diff aginst the latest version of mod_python.c
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Closed: (MODPYTHON-77) The multiple interpreter concept of
mod_python is broken for Python extension modules since Python 2.3
Posted by "Graham Dumpleton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MODPYTHON-77?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Graham Dumpleton closed MODPYTHON-77.
-------------------------------------
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: https://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Issue Type: Bug
> Components: core
> Affects Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Assigned To: Graham Dumpleton
> Fix For: 3.3
>
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, grahamd_20051105.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Nicolas Lehuen (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12320868 ]
Nicolas Lehuen commented on MODPYTHON-77:
-----------------------------------------
Boyan,
Can you please post a patch (diff) file ? It looks like the mod_python.c file you posted comes from an old version which is out of sync with the current version found in Subversion. I think I can isolate your changes but it is an error-prone process, so it would be better if you could provide us a patch file.
TIA,
Nicolas
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Jim Gallacher (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12320931 ]
Jim Gallacher commented on MODPYTHON-77:
----------------------------------------
Well, maybe I've just having one of those days, but when when I apply the patch using:
patch -p0 < diff3.txt
I get the following error:
Hunk #1 succeeded at 149 with fuzz 1.
Hunk #2 FAILED at 518.
Hunk #3 succeeded at 509 with fuzz 2 (offset -17 lines).
1 out of 5 hunks FAILED -- saving rejects to file mod_python.c.rej
and in mod_python.c.rej:
***************
*** 517,526 ****
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
- /* Release the thread state because we will never use
- * the main interpreter, only sub interpreters created later. */
- PyThreadState_Swap(NULL);
-
/* create the obCallBack dictionary */
interpreters = PyDict_New();
if (! interpreters) {
--- 518,523 ----
/* create and acquire the interpreter lock */
PyEval_InitThreads();
#endif
/* create the obCallBack dictionary */
interpreters = PyDict_New();
if (! interpreters) {
It does not matter if the failed hunk is applied or not - apache fails to start and so the unit tests fail. This is for mpm_prefork. I don't have time to test it on mpm-worker just now. That will have to wait until Monday.
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of
mod_python is broken for Python extension modules since Python 2.3
Posted by "Graham Dumpleton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Graham Dumpleton updated MODPYTHON-77:
--------------------------------------
Fix Version: 3.3
Assign To: Graham Dumpleton
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Assignee: Graham Dumpleton
> Fix For: 3.3
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, grahamd_20051105.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept
of mod_python is broken for Python extension modules since Python 2.3
Posted by "Graham Dumpleton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12366941 ]
Graham Dumpleton commented on MODPYTHON-77:
-------------------------------------------
Changes as contained in grahamd_20051105.tar.gz commited to mod_python SVN trunk at revision 378803.
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, grahamd_20051105.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Graham Dumpleton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12322710 ]
Graham Dumpleton commented on MODPYTHON-77:
-------------------------------------------
I worked out what was going on a few days ago, but first time with Internet access since. :-(
I agree that what is being proposed is reasonable provided that correct changes which will work for any version of 2.3 or later versions of Python.
Some further clarification on the issue.
1. The new API does not really break the concept of multiple interpreters. What it means is that if someone is lazy enough to use the new API they limit their extension to only being usable with the first created interpreter. Someone can still use the existing APIs and write the extension to work in a multi interpreter context.
2. Even with the patches, an extension will still not work properly. In order to work it is important that PythonImport and
PythonInterpreter be used as appropriate and that "main_interpreter" be explicitly specified. This means that one can't have multiple interpreters for different virtual hosts or applications which is a big shortcoming.
In short, extensions writers should be discouraged from using the new API if want to work with mod_python.
Out ot time, gotta go.
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Boyan Boyadjiev (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12360898 ]
Boyan Boyadjiev commented on MODPYTHON-77:
------------------------------------------
WinXP, Win2K, Win2K3: Patched mod_python/3.2.6-dev-20051217 with diff from grahamd_20051105.tar.gz is working fine. The automated tests for my project are working too:-).
[Tue Dec 20 10:17:02 2005] [notice] mod_python: Creating 8 session mutexes based on 0 max processes and 200 max threads.
[Tue Dec 20 10:17:02 2005] [notice] Apache/2.0.54 (Win32) mod_ssl/2.0.54 OpenSSL/0.9.7g mod_python/3.2.6-dev-20051217 Python/2.3.5 configured -- resuming normal operations
[Tue Dec 20 10:17:02 2005] [notice] Server built: Sep 15 2005 11:42:05
[Tue Dec 20 10:17:02 2005] [notice] Parent: Created child process 4000
[Tue Dec 20 10:17:04 2005] [notice] mod_python: Creating 8 session mutexes based on 0 max processes and 200 max threads.
[Tue Dec 20 10:17:04 2005] [notice] Child 4000: Child process is running
[Tue Dec 20 10:17:04 2005] [notice] Child 4000: Acquired the start mutex.
[Tue Dec 20 10:17:04 2005] [notice] Child 4000: Starting 200 worker threads.
[Tue Dec 20 10:17:10 2005] [notice] mod_python: (Re)importing module 'mptest'
[Tue Dec 20 10:17:10 2005] [error] BEGIN: CALLBACK
[Tue Dec 20 10:17:10 2005] [error] FINISH: CALLBACK
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, grahamd_20051105.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Boyan Boyadjiev (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12356601 ]
Boyan Boyadjiev commented on MODPYTHON-77:
------------------------------------------
There should be no difference between using GIL APIs and saving/reusing the main_interpreter on a Windows system with mpm_winnt where everything (module initialization and request processing) runs in the same process. The first change I've proposed was exactly this, but it was crashing with mpm-prefork on Linux (reported by Jim). I thought that the problem with mpm-prefork might be something like this (I'm neither Linux nor Apache expert, so I just hope that assumptions below are not too ridiculous):
- The main interpreter was saved into the interpreters dictionary in python_init, called in the context of the parent process
- Parent and child processes have different main interpreters.
- Handlers using the 'main_interpreter' are running in a child process but trying to use the parent's process main interpreter (bang!).
- Interpreters others than the main are created in a child process when a handler is being accessed for the first time.
That's why I've proposed the second change, where the use of the GIL state APIs makes the access to the main interpreter safe, transparent and independent from the kind of mpm or OS.
Regards,
Boyan
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Boyan Boyadjiev (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12356692 ]
Boyan Boyadjiev commented on MODPYTHON-77:
------------------------------------------
Server: Apache/2.0.54 (Win32) mod_python/3.2.0-dev-20050901 Python/2.3.5
Tested gilstate.tar.gz with mod_python.c.diff & mod_python.h.diff:
[Thu Nov 03 11:13:48 2005] [notice] mod_python: Creating 8 session mutexes based on 0 max processes and 200 max threads.
[Thu Nov 03 11:13:48 2005] [notice] Child 4068: Child process is running
[Thu Nov 03 11:13:48 2005] [notice] Child 4068: Acquired the start mutex.
[Thu Nov 03 11:13:48 2005] [notice] Child 4068: Starting 200 worker threads.
[Thu Nov 03 11:13:54 2005] [error] BEGIN: CALLBACK
[Thu Nov 03 11:13:54 2005] [error] FINISH: CALLBACK
Why? Because main_interpreter == python's first interpreter == interpreter in the TLS
Tested gilstate.tar.gz without mod_python.c.diff & mod_python.h.diff:
[Thu Nov 03 11:31:01 2005] [notice] mod_python: Creating 8 session mutexes based on 0 max processes and 200 max threads.
[Thu Nov 03 11:31:01 2005] [notice] Child 388: Child process is running
[Thu Nov 03 11:31:01 2005] [notice] Child 388: Acquired the start mutex.
[Thu Nov 03 11:31:01 2005] [notice] Child 388: Starting 200 worker threads.
[Thu Nov 03 11:31:03 2005] [error] BEGIN: CALLBACK
[Thu Nov 03 11:31:03 2005] [error] EXCEPTION: file() constructor not accessible in restricted mode
[Thu Nov 03 11:31:03 2005] [error] FINISH: CALLBACK
Why? Because main_interpreter != python's first interpreter
main_interpreter == interpreter in the TLS
interpreter in the TLS != python's first interpreter
For more just try Py_DEBUG, WITH_THREAD && http://sourceforge.net/tracker/index.php?func=detail&aid=1163563&group_id=5470&atid=105470
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Boyan Boyadjiev (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12356757 ]
Boyan Boyadjiev commented on MODPYTHON-77:
------------------------------------------
Sorry - I didn't saw the change done for MODPYTHON-83. PyThreadState_Swap must be added in the non thread case of release_interpreter.
With "smallest required change" I just wanted to say, that if the GIL stuff is not going to be used in mod_python, there is still somehing to do in order to handle the Python thread state in a proper way since Python 2.3.5.
I think, that the most proper fix will be mod_python.?.diff + PyThreadState_Swap.
Now a different point regarding internal_redirect. What will hapen if interpreter1 call internal_redirect into interpreter2? Could it be, that the block
Py_BEGIN_ALLOW_THREADS
ap_internal_redirect(new_uri, self->request_rec);
Py_END_ALLOW_THREADS
is not sofficient any more and should be changed in something like this
release_interpreter
ap_internal_redirect(new_uri, self->request_rec);
get_interpreter
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Work started: (MODPYTHON-77) The multiple interpreter
concept of mod_python is broken for Python extension modules since Python
2.3
Posted by "Graham Dumpleton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Work on MODPYTHON-77 started by Graham Dumpleton
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Assignee: Graham Dumpleton
> Fix For: 3.3
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, grahamd_20051105.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Resolved: (MODPYTHON-77) The multiple interpreter concept of
mod_python is broken for Python extension modules since Python 2.3
Posted by "Graham Dumpleton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Graham Dumpleton resolved MODPYTHON-77:
---------------------------------------
Resolution: Fixed
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Assignee: Graham Dumpleton
> Fix For: 3.3
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, grahamd_20051105.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Boyan Boyadjiev (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Boyan Boyadjiev updated MODPYTHON-77:
-------------------------------------
Attachment: mod_python.c
changes ware made against 3.1.4
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Boyan Boyadjiev (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12320885 ]
Boyan Boyadjiev commented on MODPYTHON-77:
------------------------------------------
Graham,
Thanks a lot for the prompt response!
I am using currently Python 2.3.5.
As far as I understood, the patch http://www.dscpl.com.au/projects/vampire/patches.html fixes two problems:
-the access to the interpreters dictionary is not multithread safe
-the mod_python import of modules is not multithread safe (for this am wondering why the fuctions imp.acquire_lock and imp.release_lock are not used)
The http://sourceforge.net/tracker/index.php?func=detail&aid=1163563&group_id=5470&atid=105470 could be a solution for my problem. I'll check this today.
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Boyan Boyadjiev (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Boyan Boyadjiev updated MODPYTHON-77:
-------------------------------------
Attachment: gil_test.c
I hope that, the following can help clarifying where the problem comes from.
The root of the evil was the introduction of the new Python APIs PyGILState_Ensure and PyGILState_Release. They are designed to make the process of embedding Python and the extension development much easier, but they also break the idea of having multiple interpreters. If you look at the implementation of the PyGILState_Ensure you will see that the Python code executed in a PyGILState_Ensure/PyGILState_Release scope is using the autoInterpreterState, which is the one initialized by Py_Initialize. So you can easily mix multiple interpreters in the same call stack... Another fact is that more and more essential extensions are using those new Python APIs. That's why I thought, that the most acceptable way of 'un-breaking' the concept of having multiple interpreters and using complex Python extensions is:
- To specify, that the use of Python extensions can work only for interpreters[MAIN_INTERPRETER]
- The interpreters[MAIN_INTERPRETER] must be the same as the pystate.c->autoInterpreterState, so the one created by Py_Initiualize.
I've written a very simplified pace of code doing both embedding and extending Python in order to exemplify what I'm talking about (see the attached gil_test.c).
Case 1: Python - it works:
Python 2.3.5 (#62, Sep 2 2005, 19:32:54) [MSC v.1200 32 bit (Intel)] on win32
>>> import gil_test
>>> attribute_found = gil_test.funct1()
sys.testAttr = testAttr
>>> print 'attribute_found =', attribute_found
attribute_found = 1
>>>
Case 2: mod_python (not patched) - does not work:
import gil_test, apache
def handler(request):
request.write('attribute_found=%s' %( gil_test.funct1(),))
return apache.OK
Case 3: mod_python (patched) - works
import gil_test, apache
def handler(request):
request.write('attribute_found=%s' %( gil_test.funct1(),))
return apache.OK
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Jim Gallacher (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12356864 ]
Jim Gallacher commented on MODPYTHON-77:
----------------------------------------
Patched 3.2.4b with diff from grahamd_20051105.tar.gz. Unit tests and gilstate test pass on Debian stable (sarge) and mpm-prefork.
[Sat Nov 05 17:21:36 2005] [notice] mod_python: Creating 8 session mutexes based on 20 max processes and 0 max threads.
[Sat Nov 05 17:21:37 2005] [notice] Apache/2.0.54 (Debian GNU/Linux) mod_python/3.2.4b Python/2.3.5 configured -- resuming normal operations
[Sat Nov 05 17:37:13 2005] [notice] mod_python: (Re)importing module 'mptest'
[Sat Nov 05 17:37:13 2005] [error] BEGIN: CALLBACK
[Sat Nov 05 17:37:13 2005] [error] FINISH: CALLBACK
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, grahamd_20051105.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Graham Dumpleton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Graham Dumpleton updated MODPYTHON-77:
--------------------------------------
Attachment: gilstate.tar.gz
Here is my own simplified test case for the GIL state API issue which exhibits the problem in a different way. In this case it is where the exception:
file() constructor not accessible in restricted mode
is raised.
Have constructed this test case as this particular problem has come up numerous times before. It is known that in some cases the problem is due to an actual error in Python (when Python 2.3.5 or later is used), ie.:
http://sourceforge.net/tracker/index.php?func=detail&aid=1163563&group_id=5470&atid=105470
The attached test case causes a problem though when using Python on Mac OS X (10.3) which is supposed to be Python 2.3.0, thus cannot be related to the bug in Python and is more likely to be the GIL state API issue.
In respect of the changes already posted here now that I have started looking at it, what I don't understand is why the mod_python.c file has to use the GIL state API functions at all. I would have thought that it was enough to save the initial interpreter as "main_interpreter" and for users to ensure they then explicitly named the interpreter they used as being "main_interpreter".
Can someone please explain the reason why there has to be this special case processing whereby GIL state API is only used when making calls into the main_interpreter. This doesn't make sense, the full API for acquiring the thread lock as it existed should have been sufficient. The only thing that might make sense is if it is done this way as a workaround for the bug in Python listed above on SourceForge.
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Updated: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Boyan Boyadjiev (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=all ]
Boyan Boyadjiev updated MODPYTHON-77:
-------------------------------------
Attachment: diff3.txt
It seems that diff2.txt was not transfered correctly to the server, so diff3.txt attached.
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Boyan Boyadjiev (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12356706 ]
Boyan Boyadjiev commented on MODPYTHON-77:
------------------------------------------
The smalest required change for a correct pythreadstate handling is the following (which is the same as in mod_python.c.diff but without the GIL stuff):
static void release_interpreter(void)
{
PyThreadState *tstate = PyThreadState_Get();
#ifdef WITH_THREAD
PyThreadState_Clear(tstate);
/*
PyThreadState_DeleteCurrent will clear also the TLS,
which will allow re-using the current thread with another
Python interpreter !!!
*/
PyThreadState_DeleteCurrent();
#else /*WITH_THREAD*/
PyThreadState_Delete(tstate);
#endif /*WITH_THREAD*/
}
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Graham Dumpleton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12320870 ]
Graham Dumpleton commented on MODPYTHON-77:
-------------------------------------------
Probably even better if they can check out subversion source code from:
https://svn.apache.org/repos/asf/httpd/mod_python/trunk/
and give diff/patch against that. :-)
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Nicolas Lehuen (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12320920 ]
Nicolas Lehuen commented on MODPYTHON-77:
-----------------------------------------
OK, I've tested your patch against the unit tests, and everything works correctly. I could check this is but I'm not quite sure to understand what it does precisely. I'll read the PEP to learn a bit more about the problem first. Meanwhile, if someone else wants to +1 the patch, feel free !
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Jim Gallacher (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12322779 ]
Jim Gallacher commented on MODPYTHON-77:
----------------------------------------
Boyan,
src.zip (file attachment 8) passes the unit tests for mpm-prefork on linux with python 2.3.5. I haven't tested it on mpm-worker yet.
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Graham Dumpleton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12320935 ]
Graham Dumpleton commented on MODPYTHON-77:
-------------------------------------------
I'm -1 for it going into next beta. I too want to understand it first as use of thread states can be very tricky.
I don't have source to refer to at hand so can't check again this instant, but is there a specific reason the interpreter data has to be stashed into the main interpreter itself? Is this so some specific extension can access it? Am wary of a solution which might be specific to making a particular extension work. Would want to know for sure that the specific extension is doing the correct things in the first place.
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, mod_python.c
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
[jira] Commented: (MODPYTHON-77) The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
Posted by "Graham Dumpleton (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/MODPYTHON-77?page=comments#action_12356842 ]
Graham Dumpleton commented on MODPYTHON-77:
-------------------------------------------
Sorry, forgot to mention that my patch is in attachment "grahamd_20051105.tar.gz".
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3
> ------------------------------------------------------------------------------------------------------
>
> Key: MODPYTHON-77
> URL: http://issues.apache.org/jira/browse/MODPYTHON-77
> Project: mod_python
> Type: Bug
> Components: core
> Versions: 3.1.4
> Environment: Python >= 2.3
> Reporter: Boyan Boyadjiev
> Attachments: diff.txt, diff2.txt, diff3.txt, gil_test.c, gilstate.tar.gz, grahamd_20051105.tar.gz, mod_python.c, mod_python.c.diff, mod_python.h.diff, src.zip
>
> The multiple interpreter concept of mod_python is broken for Python extension modules since Python 2.3 because of the PEP 311 (Simplified Global Interpreter Lock Acquisition for Extensions):
> ...
> Limitations and Exclusions
> This proposal identifies a solution for extension authors with
> complex multi-threaded requirements, but that only require a
> single "PyInterpreterState". There is no attempt to cater for
> extensions that require multiple interpreter states. At the time
> of writing, no extension has been identified that requires
> multiple PyInterpreterStates, and indeed it is not clear if that
> facility works correctly in Python itself.
> ...
> For mod_python this means, that complex Python extensions won't work any more with Python >= 2.3, because they are supposed to work only with the first interpreter state initialized for the current process (a problem we experienced). The first interpreter state is not used by mod_python after the python_init is called.
> One solution, which works fine for me, is to save the first interpreter state into the "interpreters" dictionary in the function python_init (MAIN_INTERPRETER is used as a key):
> static int python_init(apr_pool_t *p, apr_pool_t *ptemp,
> apr_pool_t *plog, server_rec *s)
> {
> ...
> /* initialize global Python interpreter if necessary */
> if (! Py_IsInitialized())
> {
> /* initialze the interpreter */
> Py_Initialize();
> #ifdef WITH_THREAD
> /* create and acquire the interpreter lock */
> PyEval_InitThreads();
> #endif
> /* create the obCallBack dictionary */
> interpreters = PyDict_New();
> if (! interpreters) {
> ap_log_error(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0, s,
> "python_init: PyDict_New() failed! No more memory?");
> exit(1);
> }
> {
> /*
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> BEGIN
> */
> PyObject *p = 0;
> interpreterdata * idata = (interpreterdata *)malloc(sizeof(interpreterdata));
> PyThreadState* currentThreadState = PyThreadState_Get();
> PyInterpreterState *istate = currentThreadState->interp;
> idata->istate = istate;
> /* obcallback will be created on first use */
> idata->obcallback = NULL;
> p = PyCObject_FromVoidPtr((void ) idata, NULL); /*p->refcout = 1*/
> PyDict_SetItemString(interpreters, MAIN_INTERPRETER, p); /*p->refcout = 2*/
> Py_DECREF(p); /*p->refcout = 1*/
> /*
> END
> Workaround PEP 311 - Simplified Global Interpreter Lock Acquisition for Extensions
> */
> }
> /* Release the thread state because we will never use
> * the main interpreter, only sub interpreters created later. */
> PyThreadState_Swap(NULL);
> #ifdef WITH_THREAD
> /* release the lock; now other threads can run */
> PyEval_ReleaseLock();
> #endif
> }
> return OK;
> }
> Another change I've made in the attached file is to Py_DECREF(p) in get_interpreter, which will remove leaky reference to the PyCObject with the interpreter data. This was not a real problem, but now I see fewer leaks in BoundsChecker :-).
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira