You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mod_python-dev@quetz.apache.org by Jim Gallacher <jg...@sympatico.ca> on 2005/06/08 00:42:28 UTC
session handling - the next generation
Hi All,
Sorry for the long post, but I think I have a nice contribution to the
session handling mechanism, but I've got a bug I just can't track down.
Make yourself a cup of coffee, sit down and have a read.
In an effort to make session handling more transparent and less error
prone, I've added a new method called get_session to the request object.
(As an aside, I've also created a new apache config directive
PythonSessionOptions, a req.get_session_options() to read the directive,
and a small internal change to req.internal_redirect() to pass a session
to a redirected request).
req.get_session() returns the session instance if it exists or creates a
new one. Internally it calls some python code,
mod_python.Session.create_session(), which does the actual work. The
user will only use get_session() to access their session, and so all our
deadlocking issues are gone forever.
It mostly works, but I've found that approx 1 in 10000 requests will
segfault. Calling mod_python.Session.create_session directly does not
segfault (tested for 2000000 requests) so that is not the issue.
I really hope someone can spot where the problem is. I've reduced
everything to the simplest possible case. The unlock method in
TestSession very rarely segfault or raise an exception and then
segfault, but very rarely. It only fails 0.004% of the time based on
500000 requests. I suspect some kind of garbage collection issue, maybe
a circular reference caused by req->session->_req, but I don't
understand why it works 99.996 % of the time.
I've included the full error.log for 5000000 requests at the end of this
message, but the ones that caught my eye as unusual are:
[Tue Jun 07 17:52:02 2005] [notice] child pid 15066 exit signal
Segmentation fault (11)
Fatal Python error: deletion of interned string failed
[Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
mptest: Traceback (most recent call last):
[Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
mptest: File "/usr/lib/python2.3/site-packages/mod_python/apache.py",
line 299, in HandlerDispatch\n result = object(req)
[Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
mptest: File "/var/www/mp/mptest.py", line 19, in handler\n sess =
req.get_session()
[Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
mptest: File
"/usr/lib/python2.3/site-packages/mod_python/TestSession.py", line 46,
in create_session\n return TestSession(req,sid)
[Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
mptest: File
"/usr/lib/python2.3/site-packages/mod_python/TestSession.py", line 40,
in unlock\n if self._lock and self._locked:
[Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
mptest: AttributeError: 'TestSession' object has no attribute '_lock'
[Tue Jun 07 18:04:03 2005] [notice] child pid 16170 exit signal
Segmentation fault (11)
Fatal Python error: GC object already tracked
I hope someone can give me some insight to the problem. Note that I'm
using mpm-worker.
Regards,
Jim
------------------------------------------------------------------------
mptest.py
# the test handler
from mod_python import apache
from mod_python import TestSession
def handler(req):
req.content_type = 'text/plain'
req.write('mptest.py')
if req.path_info:
test_case = req.path_info[1:]
else:
test_case = '1'
if test_case == '1':
# the good case - always works
sess = TestSession.create_session(req,None)
elif test_case == '2':
# the bad case - sometimes segfaults
sess = req.get_session()
else:
req.write('no test specified')
return apache.OK
---------------------------------------------------------------------------------------------------
mod_python/TestSession.py
# emulate a simple test session
import _apache
from Session import _new_sid
class TestSession(object):
def __init__(self, req, sid=0, secret=None, timeout=0, lock=1):
req.log_error("TestSession.__init__")
self._req = req
self._lock = 1
self._locked = 0
self._sid = _new_sid(req)
self.lock()
self.unlock()
def lock(self):
if self._lock:
_apache._global_lock(self._req.server, self._sid)
self._locked = 1
def unlock(self):
# unlock will ocassionally segfault
if self._lock and self._locked:
_apache._global_unlock(self._req.server, self._sid)
self._locked = 0
def create_session(req,sid):
return TestSession(req,sid)
-------------------------------------------------------------------------------------------------
src/requestobject.c
In the following code session is a PyObject* defined in the the
requestobject struct.
static PyObject *req_get_session(requestobject *self, PyObject *args)
{
PyObject *m;
PyObject *sid;
if (!self->session) {
m = PyImport_ImportModule("mod_python.TestSession");
/* is this redirected request with an existing session? */
sid = PyObject_CallMethod(self->subprocess_env, "get",
"(ss)","REDIRECT_PYSID", "");
self->session = PyObject_CallMethod(m, "create_session",
"(OO)", self, sid);
Py_DECREF(m);
Py_DECREF(sid);
if (self->session == NULL)
return NULL;
}
return self->session;
}
--------------------------------------------------------------------------------------------------------
/var/log/apache2/error.log
The full error log for approx 500000 requests (generated using grep -v
"TestSession.__init" error.log).
[Tue Jun 07 16:35:32 2005] [notice] mod_python: Creating 32 session
mutexes based on 6 max processes and 25 max threads.
[Tue Jun 07 16:35:32 2005] [notice] Apache/2.0.54 (Debian GNU/Linux)
mod_python/3.2.0-dev-20050519 Python/2.3.5 configured -- resuming normal
operations
[Tue Jun 07 16:35:49 2005] [notice] child pid 11540 exit signal
Segmentation fault (11)
[Tue Jun 07 16:35:49 2005] [notice] child pid 11545 exit signal
Segmentation fault (11)
[Tue Jun 07 16:36:05 2005] [notice] child pid 11601 exit signal
Segmentation fault (11)
[Tue Jun 07 16:46:21 2005] [notice] child pid 11629 exit signal
Segmentation fault (11)
[Tue Jun 07 16:46:35 2005] [notice] child pid 11949 exit signal
Segmentation fault (11)
[Tue Jun 07 17:04:07 2005] [notice] child pid 12454 exit signal
Segmentation fault (11)
[Tue Jun 07 17:11:47 2005] [notice] child pid 12351 exit signal
Segmentation fault (11)
[Tue Jun 07 17:18:08 2005] [notice] child pid 12342 exit signal
Segmentation fault (11)
[Tue Jun 07 17:18:20 2005] [notice] child pid 13057 exit signal
Segmentation fault (11)
[Tue Jun 07 17:34:48 2005] [notice] child pid 12859 exit signal
Segmentation fault (11)
[Tue Jun 07 17:36:35 2005] [notice] child pid 13599 exit signal
Segmentation fault (11)
[Tue Jun 07 17:38:10 2005] [notice] child pid 13698 exit signal
Segmentation fault (11)
[Tue Jun 07 17:39:02 2005] [notice] child pid 13764 exit signal
Segmentation fault (11)
[Tue Jun 07 17:39:13 2005] [notice] child pid 13819 exit signal
Segmentation fault (11)
[Tue Jun 07 17:39:15 2005] [notice] child pid 13810 exit signal
Segmentation fault (11)
[Tue Jun 07 17:40:02 2005] [notice] child pid 13917 exit signal
Segmentation fault (11)
[Tue Jun 07 17:41:03 2005] [notice] child pid 13048 exit signal
Segmentation fault (11)
Fatal Python error: GC object already tracked
[Tue Jun 07 17:42:41 2005] [notice] child pid 14016 exit signal Aborted (6)
Fatal Python error: GC object already tracked
[Tue Jun 07 17:45:10 2005] [notice] child pid 14321 exit signal Aborted (6)
[Tue Jun 07 17:48:01 2005] [notice] child pid 14439 exit signal
Segmentation fault (11)
[Tue Jun 07 17:48:49 2005] [notice] child pid 14508 exit signal
Segmentation fault (11)
[Tue Jun 07 17:50:03 2005] [notice] child pid 14615 exit signal
Segmentation fault (11)
[Tue Jun 07 17:50:03 2005] [notice] child pid 14672 exit signal
Segmentation fault (11)
Fatal Python error: GC object already tracked
[Tue Jun 07 17:50:53 2005] [notice] child pid 14665 exit signal
Segmentation fault (11)
[Tue Jun 07 17:50:53 2005] [notice] child pid 14758 exit signal Aborted (6)
[Tue Jun 07 17:51:18 2005] [notice] child pid 14800 exit signal
Segmentation fault (11)
[Tue Jun 07 17:51:18 2005] [notice] child pid 14832 exit signal
Segmentation fault (11)
[Tue Jun 07 17:51:18 2005] [notice] child pid 14864 exit signal
Segmentation fault (11)
[Tue Jun 07 17:51:20 2005] [notice] child pid 14893 exit signal
Segmentation fault (11)
[Tue Jun 07 17:51:21 2005] [notice] child pid 14921 exit signal
Segmentation fault (11)
[Tue Jun 07 17:51:21 2005] [notice] child pid 14928 exit signal
Segmentation fault (11)
[Tue Jun 07 17:51:23 2005] [notice] child pid 14977 exit signal
Segmentation fault (11)
[Tue Jun 07 17:51:24 2005] [notice] child pid 15006 exit signal
Segmentation fault (11)
[Tue Jun 07 17:52:02 2005] [notice] child pid 15066 exit signal
Segmentation fault (11)
Fatal Python error: deletion of interned string failed
[Tue Jun 07 17:53:06 2005] [notice] child pid 15109 exit signal Aborted (6)
Fatal Python error: deletion of interned string failed
[Tue Jun 07 17:53:55 2005] [notice] child pid 15210 exit signal Aborted (6)
[Tue Jun 07 17:54:03 2005] [notice] child pid 15104 exit signal
Segmentation fault (11)
[Tue Jun 07 17:54:05 2005] [notice] child pid 15312 exit signal
Segmentation fault (11)
[Tue Jun 07 17:54:17 2005] [notice] child pid 15405 exit signal
Segmentation fault (11)
[Tue Jun 07 17:55:04 2005] [notice] child pid 15437 exit signal
Segmentation fault (11)
[Tue Jun 07 17:55:04 2005] [notice] child pid 15483 exit signal
Segmentation fault (11)
[Tue Jun 07 17:55:17 2005] [notice] child pid 15563 exit signal
Segmentation fault (11)
[Tue Jun 07 17:55:54 2005] [error] [client 192.168.1.12] PythonHandler
mptest: Traceback (most recent call last):
[Tue Jun 07 17:55:54 2005] [error] [client 192.168.1.12] PythonHandler
mptest: File "/usr/lib/python2.3/site-packages/mod_python/apache.py",
line 287, in HandlerDispatch\n log=debug)
[Tue Jun 07 17:55:54 2005] [error] [client 192.168.1.12] PythonHandler
mptest: File "/usr/lib/python2.3/site-packages/mod_python/apache.py",
line 422, in import_module\n module = sys.modules[module_name]
[Tue Jun 07 17:55:54 2005] [error] [client 192.168.1.12] PythonHandler
mptest: KeyError: 'mptest'
[Tue Jun 07 17:55:58 2005] [notice] child pid 15474 exit signal
Segmentation fault (11)
[Tue Jun 07 17:56:08 2005] [notice] child pid 15600 exit signal
Segmentation fault (11)
[Tue Jun 07 17:56:08 2005] [notice] child pid 15644 exit signal
Segmentation fault (11)
[Tue Jun 07 17:57:08 2005] [notice] child pid 15730 exit signal
Segmentation fault (11)
[Tue Jun 07 17:57:10 2005] [notice] child pid 15778 exit signal
Segmentation fault (11)
[Tue Jun 07 17:57:44 2005] [notice] child pid 15772 exit signal
Segmentation fault (11)
[Tue Jun 07 17:58:11 2005] [notice] child pid 15902 exit signal
Segmentation fault (11)
[Tue Jun 07 17:58:21 2005] [notice] child pid 15945 exit signal
Segmentation fault (11)
[Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
mptest: Traceback (most recent call last):
[Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
mptest: File "/usr/lib/python2.3/site-packages/mod_python/apache.py",
line 299, in HandlerDispatch\n result = object(req)
[Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
mptest: File "/var/www/mp/mptest.py", line 19, in handler\n sess =
req.get_session()
[Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
mptest: File
"/usr/lib/python2.3/site-packages/mod_python/TestSession.py", line 46,
in create_session\n return TestSession(req,sid)
[Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
mptest: File
"/usr/lib/python2.3/site-packages/mod_python/TestSession.py", line 40,
in unlock\n if self._lock and self._locked:
[Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
mptest: AttributeError: 'TestSession' object has no attribute '_lock'
[Tue Jun 07 17:59:38 2005] [notice] child pid 15939 exit signal
Segmentation fault (11)
[Tue Jun 07 18:02:59 2005] [notice] child pid 15865 exit signal
Segmentation fault (11)
[Tue Jun 07 18:02:59 2005] [notice] child pid 16051 exit signal
Segmentation fault (11)
[Tue Jun 07 18:03:02 2005] [notice] child pid 16142 exit signal
Segmentation fault (11)
[Tue Jun 07 18:04:03 2005] [notice] child pid 16170 exit signal
Segmentation fault (11)
Fatal Python error: GC object already tracked
[Tue Jun 07 18:04:16 2005] [notice] child pid 16248 exit signal Aborted (6)
[Tue Jun 07 18:04:40 2005] [notice] child pid 16285 exit signal
Segmentation fault (11)
[Tue Jun 07 18:05:15 2005] [notice] child pid 16318 exit signal
Segmentation fault (11)
[Tue Jun 07 18:16:49 2005] [notice] child pid 16573 exit signal
Segmentation fault (11)
[Tue Jun 07 18:17:50 2005] [notice] child pid 16924 exit signal
Segmentation fault (11)
Re: session handling - the next generation
Posted by Jim Gallacher <jg...@sympatico.ca>.
Nicolas Lehuen wrote:
> Hi Jim,
>
> Have you added the corresponding code in request_tp_clear and
> request_tp_traverse ?
Yes.
> I think you can check-in your code, since the debate about import
> looks like it is going to take som time before the next release...
Will do.
Jim
> Regards,
> Nicolas
>
> 2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
>
>>Nicolas Lehuen wrote:
>>
>>>Hi Jim,
>>>
>>>After a few checks (unittest + load testing), I've checked in my
>>>modifications ; you might want to update and merge it with your code.
>>
>>I'm still getting a memory leak with the merged code. Should I commit my
>>changes anyway, or maybe we could create new svn branch for this testing?
>>
>>Jim
>>
>>
>>>Regards,
>>>Nicolas
>>>
>>>2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
>>>
>>>
>>>>Nicolas Lehuen wrote:
>>>>
>>>>
>>>>>You should cast self to a requestobject* :
>>>>
>>>>That worked. Running some tests.
>>>>
>>>>Jim
>>>>
>>>>
>>>>
>>>>>((requestobject*)self)->session
>>>>>
>>>>>Anyway, I'm currently rewriting the whole request_tp_traverse /
>>>>>request_tp_clear / request_tp_clear functions like this :
>>>>>
>>>>>static void request_tp_dealloc(requestobject *self)
>>>>>{
>>>>> // de-register the object from the GC
>>>>> // before its deallocation, to prevent the
>>>>> // GC to run on a partially de-allocated object
>>>>> PyObject_GC_UnTrack(self);
>>>>>
>>>>> request_tp_clear(self);
>>>>>
>>>>> PyObject_GC_Del(self);
>>>>>}
>>>>>
>>>>>/**
>>>>>** request_tp_traverse
>>>>>**
>>>>>* Traversal of the request object
>>>>>*/
>>>>>static int request_tp_traverse(requestobject* self, visitproc visit,
>>>>>void *arg) {
>>>>> int result;
>>>>>
>>>>> if(self->dict) {result = visit(self->dict,arg); if(result) return result;}
>>>>> if(self->connection) {result = visit(self->connection,arg);
>>>>>if(result) return result;}
>>>>> if(self->server) {result = visit(self->server,arg); if(result)
>>>>>return result;}
>>>>> if(self->next) {result = visit(self->next,arg); if(result) return result;}
>>>>> if(self->prev) {result = visit(self->prev,arg); if(result) return result;}
>>>>> if(self->main) {result = visit(self->main,arg); if(result) return result;}
>>>>> if(self->headers_in) {result = visit(self->headers_in,arg);
>>>>>if(result) return result;}
>>>>> if(self->headers_out) {result = visit(self->headers_out,arg);
>>>>>if(result) return result;}
>>>>> if(self->err_headers_out) {result =
>>>>>visit(self->err_headers_out,arg); if(result) return result;}
>>>>> if(self->subprocess_env) {result =
>>>>>visit(self->subprocess_env,arg); if(result) return result;}
>>>>> if(self->notes) {result = visit(self->notes,arg); if(result) return result;}
>>>>> if(self->phase) {result = visit(self->phase,arg); if(result) return result;}
>>>>>
>>>>> // no need to Py_DECREF(dict) since the reference is borrowed
>>>>> return 0;
>>>>>}
>>>>>
>>>>>static int request_tp_clear(requestobject *self)
>>>>>{
>>>>> PyObject* tmp;
>>>>> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->connection; self->connection=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->server; self->server=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->next; self->next=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->prev; self->prev=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->main; self->main=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->headers_in; self->headers_in=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->headers_out; self->headers_out=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->err_headers_out; self->err_headers_out=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->subprocess_env; self->subprocess_env=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->notes; self->notes=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->phase; self->phase=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->hlo; self->hlo=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
>>>>> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
>>>>>
>>>>> return 0;
>>>>>}
>>>>>
>>>>>As you can guess from the source code, we'll have to add some code for
>>>>>the session member when you'll integrate your code in subversion.
>>>>>
>>>>>Regards,
>>>>>Nicolas
>>>>>
>>>>>2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
>>>>>
>>>>>
>>>>>
>>>>>>Nicolas,
>>>>>>
>>>>>>It fails to compile when I add your bit of code. Sure looks like it
>>>>>>should work though.
>>>>>>
>>>>>>requestobject.c: In function `request_tp_traverse':
>>>>>>requestobject.c:1539: error: structure has no member named `session'
>>>>>>requestobject.c:1540: error: structure has no member named `session'
>>>>>>
>>>>>>Just to be clear, it compiles and runs just fine without the extra check
>>>>>>added into request_tp_traverse.
>>>>>>
>>>>>>Nicolas Lehuen wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>I'm re-reading the comment I wrote when I implemented tp_traverse :
>>>>>>>
>>>>>>>// only traverse its dictionary since other fields defined in
>>>>>>>request_rec_mbrs with type T_OBJECT
>>>>>>>// cannot be the source of memory leaks (unless you really want it)
>>>>>>>
>>>>>>>Turns out that we should at least traverse the next and prev in case
>>>>>>>someone does something like req.next.my_prev = req. That's what I
>>>>>>>meant by "unless you really want it".
>>>>>>
>>>>>>I wondered about that. :)
>>>>>>
>>>>>>Jim
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>It's pretty difficult to get
>>>>>>>there (you need an internal redirect), but you can. So I'm going to
>>>>>>>have a look at re-implementing the tp_traverse handler.
>>>>>>>Regards,
>>>>>>>Nicolas
>>>>>>>
>>>>>>>2005/6/12, Nicolas Lehuen <ni...@gmail.com>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>Duh, I get it. If you add a member to the request object, and this
>>>>>>>>member is not referenced in the request object's dictionary, then you
>>>>>>>>have to add a special case for it in the tp_traverse handler. In
>>>>>>>>requestobject.c :
>>>>>>>>
>>>>>>>>/**
>>>>>>>>** request_tp_traverse
>>>>>>>>**
>>>>>>>>* Traversal of the request object
>>>>>>>>*/
>>>>>>>>static int request_tp_traverse(PyObject *self, visitproc visit, void *arg) {
>>>>>>>> PyObject *dict,*values,*item,*str;
>>>>>>>> int i,size;
>>>>>>>>
>>>>>>>> // only traverse its dictionary since other fields defined in
>>>>>>>>request_rec_mbrs with type T_OBJECT
>>>>>>>> // cannot be the source of memory leaks (unless you really want it)
>>>>>>>> dict=*_PyObject_GetDictPtr(self);
>>>>>>>> if(dict) {
>>>>>>>> // this check is not needed, I guess, _PyObject_GetDictPtr
>>>>>>>>always give a pointer to a dict object.
>>>>>>>> if(PyDict_Check(dict)) {
>>>>>>>> i = visit(dict,arg);
>>>>>>>> if(i) {
>>>>>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>>>>>>>((requestobject*)self)->request_rec, "%s:%i Call to visit()
>>>>>>>>failed",__LINE__,__FILE__);
>>>>>>>> // no need to Py_DECREF(dict) since the reference is borrowed
>>>>>>>> return i;
>>>>>>>> }
>>>>>>>> }
>>>>>>>> else {
>>>>>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>>>>>>>((requestobject*)self)->request_rec, "%s:%i Expected a
>>>>>>>>dictionary",__LINE__,__FILE__);
>>>>>>>> }
>>>>>>>> }
>>>>>>>> else {
>>>>>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>>>>>>>((requestobject*)self)->request_rec, "%s:%i Expected a
>>>>>>>>dictionary",__LINE__,__FILE__);
>>>>>>>> }
>>>>>>>>
>>>>>>>> /* This is the new code that needs to be added to support the new
>>>>>>>>session member */
>>>>>>>> if(self->session) {
>>>>>>>> i = visit(self->session,arg);
>>>>>>>> if(i) {
>>>>>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>>>>>>>((requestobject*)self)->request_rec, "%s:%i Call to visit()
>>>>>>>>failed",__LINE__,__FILE__);
>>>>>>>> return i;
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>> // no need to Py_DECREF(dict) since the reference is borrowed
>>>>>>>> return 0;
>>>>>>>>}
>>>>>>>>
>>>>>>>>Regards,
>>>>>>>>Nicolas
>>>>>>>>
>>>>>>>>2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>Hi Nicolas,
>>>>>>>>>
>>>>>>>>>Nicolas Lehuen wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>Hi Jim,
>>>>>>>>>>
>>>>>>>>>>How where you creating the session object in requestobject.c ? If you
>>>>>>>>>>were using PythonObject_New, then this may explain the memory leak.
>>>>>>>>>>Objects that must be managed by the garbage collector have to be
>>>>>>>>>>created with PyObject_GC_New.
>>>>>>>>>
>>>>>>>>>The c-code loads the python module and calls a function which generates
>>>>>>>>>the session object.
>>>>>>>>>
>>>>>>>>>src/requestobject.c
>>>>>>>>>static PyObject *req_get_session(requestobject *self, PyObject *args)
>>>>>>>>>{
>>>>>>>>> PyObject *m; // session module
>>>>>>>>> PyObject *sid; // session id
>>>>>>>>> PyObject *result;
>>>>>>>>>
>>>>>>>>> if (!self->session) {
>>>>>>>>> sid = PyObject_CallMethod(self->subprocess_env, "get",
>>>>>>>>> "(ss)","REDIRECT_PYSID", "");
>>>>>>>>>
>>>>>>>>> /********
>>>>>>>>> * This is where the session instance is created
>>>>>>>>> ********/
>>>>>>>>> m = PyImport_ImportModule("mod_python.session2.TestSession");
>>>>>>>>> self->session = PyObject_CallMethod(m, "create_session", "(OO)",
>>>>>>>>> self, sid);
>>>>>>>>>
>>>>>>>>> Py_DECREF(m);
>>>>>>>>> Py_DECREF(sid);
>>>>>>>>> if (self->session == NULL)
>>>>>>>>> return NULL;
>>>>>>>>> }
>>>>>>>>> result = self->session;
>>>>>>>>> Py_INCREF(result);
>>>>>>>>> return result;
>>>>>>>>>}
>>>>>>>>>
>>>>>>>>>----------------------------------------------------------------
>>>>>>>>>mod_python/session2/TestSession.py
>>>>>>>>># emulate a simple test session
>>>>>>>>>import _apache
>>>>>>>>
>>>>>>>>>from mod_python.Session import _new_sid
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>class TestSession(object):
>>>>>>>>>
>>>>>>>>> def __init__(self, req, sid=0, secret=None, timeout=0, lock=1):
>>>>>>>>> req.log_error("TestSession.__init__")
>>>>>>>>> # Circular Reference causes problem
>>>>>>>>> self._req = req
>>>>>>>>>
>>>>>>>>> # keeping a reference to the server object does
>>>>>>>>> # NOT cause a problem
>>>>>>>>> self._server = req.server
>>>>>>>>> self._lock = 1
>>>>>>>>> self._locked = 0
>>>>>>>>> self._sid = _new_sid(req)
>>>>>>>>> self.lock()
>>>>>>>>> self.unlock()
>>>>>>>>>
>>>>>>>>> def lock(self):
>>>>>>>>> if self._lock:
>>>>>>>>> _apache._global_lock(self._req.server, self._sid)
>>>>>>>>> self._locked = 1
>>>>>>>>>
>>>>>>>>> def unlock(self):
>>>>>>>>> # unlock will ocassionally segfault
>>>>>>>>> if self._lock and self._locked:
>>>>>>>>> _apache._global_unlock(self._req.server, self._sid)
>>>>>>>>> self._locked = 0
>>>>>>>>>
>>>>>>>>>def create_session(req,sid):
>>>>>>>>> return TestSession(req,sid)
>>>>>>>>>
>>>>>>>>
>>
>
Re: session handling - the next generation
Posted by Nicolas Lehuen <ni...@gmail.com>.
Hi Jim,
Have you added the corresponding code in request_tp_clear and
request_tp_traverse ?
I think you can check-in your code, since the debate about import
looks like it is going to take som time before the next release...
Regards,
Nicolas
2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
> Nicolas Lehuen wrote:
> > Hi Jim,
> >
> > After a few checks (unittest + load testing), I've checked in my
> > modifications ; you might want to update and merge it with your code.
>
> I'm still getting a memory leak with the merged code. Should I commit my
> changes anyway, or maybe we could create new svn branch for this testing?
>
> Jim
>
> > Regards,
> > Nicolas
> >
> > 2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
> >
> >>Nicolas Lehuen wrote:
> >>
> >>>You should cast self to a requestobject* :
> >>
> >>That worked. Running some tests.
> >>
> >>Jim
> >>
> >>
> >>>((requestobject*)self)->session
> >>>
> >>>Anyway, I'm currently rewriting the whole request_tp_traverse /
> >>>request_tp_clear / request_tp_clear functions like this :
> >>>
> >>>static void request_tp_dealloc(requestobject *self)
> >>>{
> >>> // de-register the object from the GC
> >>> // before its deallocation, to prevent the
> >>> // GC to run on a partially de-allocated object
> >>> PyObject_GC_UnTrack(self);
> >>>
> >>> request_tp_clear(self);
> >>>
> >>> PyObject_GC_Del(self);
> >>>}
> >>>
> >>>/**
> >>> ** request_tp_traverse
> >>> **
> >>> * Traversal of the request object
> >>> */
> >>>static int request_tp_traverse(requestobject* self, visitproc visit,
> >>>void *arg) {
> >>> int result;
> >>>
> >>> if(self->dict) {result = visit(self->dict,arg); if(result) return result;}
> >>> if(self->connection) {result = visit(self->connection,arg);
> >>>if(result) return result;}
> >>> if(self->server) {result = visit(self->server,arg); if(result)
> >>>return result;}
> >>> if(self->next) {result = visit(self->next,arg); if(result) return result;}
> >>> if(self->prev) {result = visit(self->prev,arg); if(result) return result;}
> >>> if(self->main) {result = visit(self->main,arg); if(result) return result;}
> >>> if(self->headers_in) {result = visit(self->headers_in,arg);
> >>>if(result) return result;}
> >>> if(self->headers_out) {result = visit(self->headers_out,arg);
> >>>if(result) return result;}
> >>> if(self->err_headers_out) {result =
> >>>visit(self->err_headers_out,arg); if(result) return result;}
> >>> if(self->subprocess_env) {result =
> >>>visit(self->subprocess_env,arg); if(result) return result;}
> >>> if(self->notes) {result = visit(self->notes,arg); if(result) return result;}
> >>> if(self->phase) {result = visit(self->phase,arg); if(result) return result;}
> >>>
> >>> // no need to Py_DECREF(dict) since the reference is borrowed
> >>> return 0;
> >>>}
> >>>
> >>>static int request_tp_clear(requestobject *self)
> >>>{
> >>> PyObject* tmp;
> >>> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
> >>> tmp=self->connection; self->connection=NULL; Py_XDECREF(tmp);
> >>> tmp=self->server; self->server=NULL; Py_XDECREF(tmp);
> >>> tmp=self->next; self->next=NULL; Py_XDECREF(tmp);
> >>> tmp=self->prev; self->prev=NULL; Py_XDECREF(tmp);
> >>> tmp=self->main; self->main=NULL; Py_XDECREF(tmp);
> >>> tmp=self->headers_in; self->headers_in=NULL; Py_XDECREF(tmp);
> >>> tmp=self->headers_out; self->headers_out=NULL; Py_XDECREF(tmp);
> >>> tmp=self->err_headers_out; self->err_headers_out=NULL; Py_XDECREF(tmp);
> >>> tmp=self->subprocess_env; self->subprocess_env=NULL; Py_XDECREF(tmp);
> >>> tmp=self->notes; self->notes=NULL; Py_XDECREF(tmp);
> >>> tmp=self->phase; self->phase=NULL; Py_XDECREF(tmp);
> >>> tmp=self->hlo; self->hlo=NULL; Py_XDECREF(tmp);
> >>> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
> >>> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
> >>> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
> >>> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
> >>>
> >>> return 0;
> >>>}
> >>>
> >>>As you can guess from the source code, we'll have to add some code for
> >>>the session member when you'll integrate your code in subversion.
> >>>
> >>>Regards,
> >>>Nicolas
> >>>
> >>>2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
> >>>
> >>>
> >>>>Nicolas,
> >>>>
> >>>>It fails to compile when I add your bit of code. Sure looks like it
> >>>>should work though.
> >>>>
> >>>>requestobject.c: In function `request_tp_traverse':
> >>>>requestobject.c:1539: error: structure has no member named `session'
> >>>>requestobject.c:1540: error: structure has no member named `session'
> >>>>
> >>>>Just to be clear, it compiles and runs just fine without the extra check
> >>>> added into request_tp_traverse.
> >>>>
> >>>>Nicolas Lehuen wrote:
> >>>>
> >>>>
> >>>>>I'm re-reading the comment I wrote when I implemented tp_traverse :
> >>>>>
> >>>>>// only traverse its dictionary since other fields defined in
> >>>>>request_rec_mbrs with type T_OBJECT
> >>>>>// cannot be the source of memory leaks (unless you really want it)
> >>>>>
> >>>>>Turns out that we should at least traverse the next and prev in case
> >>>>>someone does something like req.next.my_prev = req. That's what I
> >>>>>meant by "unless you really want it".
> >>>>
> >>>>I wondered about that. :)
> >>>>
> >>>>Jim
> >>>>
> >>>>
> >>>>
> >>>>>It's pretty difficult to get
> >>>>>there (you need an internal redirect), but you can. So I'm going to
> >>>>>have a look at re-implementing the tp_traverse handler.
> >>>>>Regards,
> >>>>>Nicolas
> >>>>>
> >>>>>2005/6/12, Nicolas Lehuen <ni...@gmail.com>:
> >>>>>
> >>>>>
> >>>>>
> >>>>>>Duh, I get it. If you add a member to the request object, and this
> >>>>>>member is not referenced in the request object's dictionary, then you
> >>>>>>have to add a special case for it in the tp_traverse handler. In
> >>>>>>requestobject.c :
> >>>>>>
> >>>>>>/**
> >>>>>>** request_tp_traverse
> >>>>>>**
> >>>>>>* Traversal of the request object
> >>>>>>*/
> >>>>>>static int request_tp_traverse(PyObject *self, visitproc visit, void *arg) {
> >>>>>> PyObject *dict,*values,*item,*str;
> >>>>>> int i,size;
> >>>>>>
> >>>>>> // only traverse its dictionary since other fields defined in
> >>>>>>request_rec_mbrs with type T_OBJECT
> >>>>>> // cannot be the source of memory leaks (unless you really want it)
> >>>>>> dict=*_PyObject_GetDictPtr(self);
> >>>>>> if(dict) {
> >>>>>> // this check is not needed, I guess, _PyObject_GetDictPtr
> >>>>>>always give a pointer to a dict object.
> >>>>>> if(PyDict_Check(dict)) {
> >>>>>> i = visit(dict,arg);
> >>>>>> if(i) {
> >>>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> >>>>>>((requestobject*)self)->request_rec, "%s:%i Call to visit()
> >>>>>>failed",__LINE__,__FILE__);
> >>>>>> // no need to Py_DECREF(dict) since the reference is borrowed
> >>>>>> return i;
> >>>>>> }
> >>>>>> }
> >>>>>> else {
> >>>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> >>>>>>((requestobject*)self)->request_rec, "%s:%i Expected a
> >>>>>>dictionary",__LINE__,__FILE__);
> >>>>>> }
> >>>>>> }
> >>>>>> else {
> >>>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> >>>>>>((requestobject*)self)->request_rec, "%s:%i Expected a
> >>>>>>dictionary",__LINE__,__FILE__);
> >>>>>> }
> >>>>>>
> >>>>>> /* This is the new code that needs to be added to support the new
> >>>>>>session member */
> >>>>>> if(self->session) {
> >>>>>> i = visit(self->session,arg);
> >>>>>> if(i) {
> >>>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> >>>>>>((requestobject*)self)->request_rec, "%s:%i Call to visit()
> >>>>>>failed",__LINE__,__FILE__);
> >>>>>> return i;
> >>>>>> }
> >>>>>> }
> >>>>>>
> >>>>>> // no need to Py_DECREF(dict) since the reference is borrowed
> >>>>>> return 0;
> >>>>>>}
> >>>>>>
> >>>>>>Regards,
> >>>>>>Nicolas
> >>>>>>
> >>>>>>2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>>Hi Nicolas,
> >>>>>>>
> >>>>>>>Nicolas Lehuen wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>>Hi Jim,
> >>>>>>>>
> >>>>>>>>How where you creating the session object in requestobject.c ? If you
> >>>>>>>>were using PythonObject_New, then this may explain the memory leak.
> >>>>>>>>Objects that must be managed by the garbage collector have to be
> >>>>>>>>created with PyObject_GC_New.
> >>>>>>>
> >>>>>>>The c-code loads the python module and calls a function which generates
> >>>>>>>the session object.
> >>>>>>>
> >>>>>>>src/requestobject.c
> >>>>>>>static PyObject *req_get_session(requestobject *self, PyObject *args)
> >>>>>>>{
> >>>>>>> PyObject *m; // session module
> >>>>>>> PyObject *sid; // session id
> >>>>>>> PyObject *result;
> >>>>>>>
> >>>>>>> if (!self->session) {
> >>>>>>> sid = PyObject_CallMethod(self->subprocess_env, "get",
> >>>>>>> "(ss)","REDIRECT_PYSID", "");
> >>>>>>>
> >>>>>>> /********
> >>>>>>> * This is where the session instance is created
> >>>>>>> ********/
> >>>>>>> m = PyImport_ImportModule("mod_python.session2.TestSession");
> >>>>>>> self->session = PyObject_CallMethod(m, "create_session", "(OO)",
> >>>>>>> self, sid);
> >>>>>>>
> >>>>>>> Py_DECREF(m);
> >>>>>>> Py_DECREF(sid);
> >>>>>>> if (self->session == NULL)
> >>>>>>> return NULL;
> >>>>>>> }
> >>>>>>> result = self->session;
> >>>>>>> Py_INCREF(result);
> >>>>>>> return result;
> >>>>>>>}
> >>>>>>>
> >>>>>>>----------------------------------------------------------------
> >>>>>>>mod_python/session2/TestSession.py
> >>>>>>># emulate a simple test session
> >>>>>>>import _apache
> >>>>>>
> >>>>>>>from mod_python.Session import _new_sid
> >>>>>>
> >>>>>>
> >>>>>>>class TestSession(object):
> >>>>>>>
> >>>>>>> def __init__(self, req, sid=0, secret=None, timeout=0, lock=1):
> >>>>>>> req.log_error("TestSession.__init__")
> >>>>>>> # Circular Reference causes problem
> >>>>>>> self._req = req
> >>>>>>>
> >>>>>>> # keeping a reference to the server object does
> >>>>>>> # NOT cause a problem
> >>>>>>> self._server = req.server
> >>>>>>> self._lock = 1
> >>>>>>> self._locked = 0
> >>>>>>> self._sid = _new_sid(req)
> >>>>>>> self.lock()
> >>>>>>> self.unlock()
> >>>>>>>
> >>>>>>> def lock(self):
> >>>>>>> if self._lock:
> >>>>>>> _apache._global_lock(self._req.server, self._sid)
> >>>>>>> self._locked = 1
> >>>>>>>
> >>>>>>> def unlock(self):
> >>>>>>> # unlock will ocassionally segfault
> >>>>>>> if self._lock and self._locked:
> >>>>>>> _apache._global_unlock(self._req.server, self._sid)
> >>>>>>> self._locked = 0
> >>>>>>>
> >>>>>>>def create_session(req,sid):
> >>>>>>> return TestSession(req,sid)
> >>>>>>>
> >>>>>>
> >>
> >
>
>
Re: session handling - the next generation
Posted by Jim Gallacher <jg...@sympatico.ca>.
Nicolas Lehuen wrote:
> Hi Jim,
>
> After a few checks (unittest + load testing), I've checked in my
> modifications ; you might want to update and merge it with your code.
I'm still getting a memory leak with the merged code. Should I commit my
changes anyway, or maybe we could create new svn branch for this testing?
Jim
> Regards,
> Nicolas
>
> 2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
>
>>Nicolas Lehuen wrote:
>>
>>>You should cast self to a requestobject* :
>>
>>That worked. Running some tests.
>>
>>Jim
>>
>>
>>>((requestobject*)self)->session
>>>
>>>Anyway, I'm currently rewriting the whole request_tp_traverse /
>>>request_tp_clear / request_tp_clear functions like this :
>>>
>>>static void request_tp_dealloc(requestobject *self)
>>>{
>>> // de-register the object from the GC
>>> // before its deallocation, to prevent the
>>> // GC to run on a partially de-allocated object
>>> PyObject_GC_UnTrack(self);
>>>
>>> request_tp_clear(self);
>>>
>>> PyObject_GC_Del(self);
>>>}
>>>
>>>/**
>>> ** request_tp_traverse
>>> **
>>> * Traversal of the request object
>>> */
>>>static int request_tp_traverse(requestobject* self, visitproc visit,
>>>void *arg) {
>>> int result;
>>>
>>> if(self->dict) {result = visit(self->dict,arg); if(result) return result;}
>>> if(self->connection) {result = visit(self->connection,arg);
>>>if(result) return result;}
>>> if(self->server) {result = visit(self->server,arg); if(result)
>>>return result;}
>>> if(self->next) {result = visit(self->next,arg); if(result) return result;}
>>> if(self->prev) {result = visit(self->prev,arg); if(result) return result;}
>>> if(self->main) {result = visit(self->main,arg); if(result) return result;}
>>> if(self->headers_in) {result = visit(self->headers_in,arg);
>>>if(result) return result;}
>>> if(self->headers_out) {result = visit(self->headers_out,arg);
>>>if(result) return result;}
>>> if(self->err_headers_out) {result =
>>>visit(self->err_headers_out,arg); if(result) return result;}
>>> if(self->subprocess_env) {result =
>>>visit(self->subprocess_env,arg); if(result) return result;}
>>> if(self->notes) {result = visit(self->notes,arg); if(result) return result;}
>>> if(self->phase) {result = visit(self->phase,arg); if(result) return result;}
>>>
>>> // no need to Py_DECREF(dict) since the reference is borrowed
>>> return 0;
>>>}
>>>
>>>static int request_tp_clear(requestobject *self)
>>>{
>>> PyObject* tmp;
>>> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
>>> tmp=self->connection; self->connection=NULL; Py_XDECREF(tmp);
>>> tmp=self->server; self->server=NULL; Py_XDECREF(tmp);
>>> tmp=self->next; self->next=NULL; Py_XDECREF(tmp);
>>> tmp=self->prev; self->prev=NULL; Py_XDECREF(tmp);
>>> tmp=self->main; self->main=NULL; Py_XDECREF(tmp);
>>> tmp=self->headers_in; self->headers_in=NULL; Py_XDECREF(tmp);
>>> tmp=self->headers_out; self->headers_out=NULL; Py_XDECREF(tmp);
>>> tmp=self->err_headers_out; self->err_headers_out=NULL; Py_XDECREF(tmp);
>>> tmp=self->subprocess_env; self->subprocess_env=NULL; Py_XDECREF(tmp);
>>> tmp=self->notes; self->notes=NULL; Py_XDECREF(tmp);
>>> tmp=self->phase; self->phase=NULL; Py_XDECREF(tmp);
>>> tmp=self->hlo; self->hlo=NULL; Py_XDECREF(tmp);
>>> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
>>> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
>>> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
>>> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
>>>
>>> return 0;
>>>}
>>>
>>>As you can guess from the source code, we'll have to add some code for
>>>the session member when you'll integrate your code in subversion.
>>>
>>>Regards,
>>>Nicolas
>>>
>>>2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
>>>
>>>
>>>>Nicolas,
>>>>
>>>>It fails to compile when I add your bit of code. Sure looks like it
>>>>should work though.
>>>>
>>>>requestobject.c: In function `request_tp_traverse':
>>>>requestobject.c:1539: error: structure has no member named `session'
>>>>requestobject.c:1540: error: structure has no member named `session'
>>>>
>>>>Just to be clear, it compiles and runs just fine without the extra check
>>>> added into request_tp_traverse.
>>>>
>>>>Nicolas Lehuen wrote:
>>>>
>>>>
>>>>>I'm re-reading the comment I wrote when I implemented tp_traverse :
>>>>>
>>>>>// only traverse its dictionary since other fields defined in
>>>>>request_rec_mbrs with type T_OBJECT
>>>>>// cannot be the source of memory leaks (unless you really want it)
>>>>>
>>>>>Turns out that we should at least traverse the next and prev in case
>>>>>someone does something like req.next.my_prev = req. That's what I
>>>>>meant by "unless you really want it".
>>>>
>>>>I wondered about that. :)
>>>>
>>>>Jim
>>>>
>>>>
>>>>
>>>>>It's pretty difficult to get
>>>>>there (you need an internal redirect), but you can. So I'm going to
>>>>>have a look at re-implementing the tp_traverse handler.
>>>>>Regards,
>>>>>Nicolas
>>>>>
>>>>>2005/6/12, Nicolas Lehuen <ni...@gmail.com>:
>>>>>
>>>>>
>>>>>
>>>>>>Duh, I get it. If you add a member to the request object, and this
>>>>>>member is not referenced in the request object's dictionary, then you
>>>>>>have to add a special case for it in the tp_traverse handler. In
>>>>>>requestobject.c :
>>>>>>
>>>>>>/**
>>>>>>** request_tp_traverse
>>>>>>**
>>>>>>* Traversal of the request object
>>>>>>*/
>>>>>>static int request_tp_traverse(PyObject *self, visitproc visit, void *arg) {
>>>>>> PyObject *dict,*values,*item,*str;
>>>>>> int i,size;
>>>>>>
>>>>>> // only traverse its dictionary since other fields defined in
>>>>>>request_rec_mbrs with type T_OBJECT
>>>>>> // cannot be the source of memory leaks (unless you really want it)
>>>>>> dict=*_PyObject_GetDictPtr(self);
>>>>>> if(dict) {
>>>>>> // this check is not needed, I guess, _PyObject_GetDictPtr
>>>>>>always give a pointer to a dict object.
>>>>>> if(PyDict_Check(dict)) {
>>>>>> i = visit(dict,arg);
>>>>>> if(i) {
>>>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>>>>>((requestobject*)self)->request_rec, "%s:%i Call to visit()
>>>>>>failed",__LINE__,__FILE__);
>>>>>> // no need to Py_DECREF(dict) since the reference is borrowed
>>>>>> return i;
>>>>>> }
>>>>>> }
>>>>>> else {
>>>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>>>>>((requestobject*)self)->request_rec, "%s:%i Expected a
>>>>>>dictionary",__LINE__,__FILE__);
>>>>>> }
>>>>>> }
>>>>>> else {
>>>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>>>>>((requestobject*)self)->request_rec, "%s:%i Expected a
>>>>>>dictionary",__LINE__,__FILE__);
>>>>>> }
>>>>>>
>>>>>> /* This is the new code that needs to be added to support the new
>>>>>>session member */
>>>>>> if(self->session) {
>>>>>> i = visit(self->session,arg);
>>>>>> if(i) {
>>>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>>>>>((requestobject*)self)->request_rec, "%s:%i Call to visit()
>>>>>>failed",__LINE__,__FILE__);
>>>>>> return i;
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> // no need to Py_DECREF(dict) since the reference is borrowed
>>>>>> return 0;
>>>>>>}
>>>>>>
>>>>>>Regards,
>>>>>>Nicolas
>>>>>>
>>>>>>2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>Hi Nicolas,
>>>>>>>
>>>>>>>Nicolas Lehuen wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>Hi Jim,
>>>>>>>>
>>>>>>>>How where you creating the session object in requestobject.c ? If you
>>>>>>>>were using PythonObject_New, then this may explain the memory leak.
>>>>>>>>Objects that must be managed by the garbage collector have to be
>>>>>>>>created with PyObject_GC_New.
>>>>>>>
>>>>>>>The c-code loads the python module and calls a function which generates
>>>>>>>the session object.
>>>>>>>
>>>>>>>src/requestobject.c
>>>>>>>static PyObject *req_get_session(requestobject *self, PyObject *args)
>>>>>>>{
>>>>>>> PyObject *m; // session module
>>>>>>> PyObject *sid; // session id
>>>>>>> PyObject *result;
>>>>>>>
>>>>>>> if (!self->session) {
>>>>>>> sid = PyObject_CallMethod(self->subprocess_env, "get",
>>>>>>> "(ss)","REDIRECT_PYSID", "");
>>>>>>>
>>>>>>> /********
>>>>>>> * This is where the session instance is created
>>>>>>> ********/
>>>>>>> m = PyImport_ImportModule("mod_python.session2.TestSession");
>>>>>>> self->session = PyObject_CallMethod(m, "create_session", "(OO)",
>>>>>>> self, sid);
>>>>>>>
>>>>>>> Py_DECREF(m);
>>>>>>> Py_DECREF(sid);
>>>>>>> if (self->session == NULL)
>>>>>>> return NULL;
>>>>>>> }
>>>>>>> result = self->session;
>>>>>>> Py_INCREF(result);
>>>>>>> return result;
>>>>>>>}
>>>>>>>
>>>>>>>----------------------------------------------------------------
>>>>>>>mod_python/session2/TestSession.py
>>>>>>># emulate a simple test session
>>>>>>>import _apache
>>>>>>
>>>>>>>from mod_python.Session import _new_sid
>>>>>>
>>>>>>
>>>>>>>class TestSession(object):
>>>>>>>
>>>>>>> def __init__(self, req, sid=0, secret=None, timeout=0, lock=1):
>>>>>>> req.log_error("TestSession.__init__")
>>>>>>> # Circular Reference causes problem
>>>>>>> self._req = req
>>>>>>>
>>>>>>> # keeping a reference to the server object does
>>>>>>> # NOT cause a problem
>>>>>>> self._server = req.server
>>>>>>> self._lock = 1
>>>>>>> self._locked = 0
>>>>>>> self._sid = _new_sid(req)
>>>>>>> self.lock()
>>>>>>> self.unlock()
>>>>>>>
>>>>>>> def lock(self):
>>>>>>> if self._lock:
>>>>>>> _apache._global_lock(self._req.server, self._sid)
>>>>>>> self._locked = 1
>>>>>>>
>>>>>>> def unlock(self):
>>>>>>> # unlock will ocassionally segfault
>>>>>>> if self._lock and self._locked:
>>>>>>> _apache._global_unlock(self._req.server, self._sid)
>>>>>>> self._locked = 0
>>>>>>>
>>>>>>>def create_session(req,sid):
>>>>>>> return TestSession(req,sid)
>>>>>>>
>>>>>>
>>
>
Re: session handling - the next generation
Posted by Nicolas Lehuen <ni...@gmail.com>.
Hi Jim,
After a few checks (unittest + load testing), I've checked in my
modifications ; you might want to update and merge it with your code.
Regards,
Nicolas
2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
> Nicolas Lehuen wrote:
> > You should cast self to a requestobject* :
>
> That worked. Running some tests.
>
> Jim
>
> >
> > ((requestobject*)self)->session
> >
> > Anyway, I'm currently rewriting the whole request_tp_traverse /
> > request_tp_clear / request_tp_clear functions like this :
> >
> > static void request_tp_dealloc(requestobject *self)
> > {
> > // de-register the object from the GC
> > // before its deallocation, to prevent the
> > // GC to run on a partially de-allocated object
> > PyObject_GC_UnTrack(self);
> >
> > request_tp_clear(self);
> >
> > PyObject_GC_Del(self);
> > }
> >
> > /**
> > ** request_tp_traverse
> > **
> > * Traversal of the request object
> > */
> > static int request_tp_traverse(requestobject* self, visitproc visit,
> > void *arg) {
> > int result;
> >
> > if(self->dict) {result = visit(self->dict,arg); if(result) return result;}
> > if(self->connection) {result = visit(self->connection,arg);
> > if(result) return result;}
> > if(self->server) {result = visit(self->server,arg); if(result)
> > return result;}
> > if(self->next) {result = visit(self->next,arg); if(result) return result;}
> > if(self->prev) {result = visit(self->prev,arg); if(result) return result;}
> > if(self->main) {result = visit(self->main,arg); if(result) return result;}
> > if(self->headers_in) {result = visit(self->headers_in,arg);
> > if(result) return result;}
> > if(self->headers_out) {result = visit(self->headers_out,arg);
> > if(result) return result;}
> > if(self->err_headers_out) {result =
> > visit(self->err_headers_out,arg); if(result) return result;}
> > if(self->subprocess_env) {result =
> > visit(self->subprocess_env,arg); if(result) return result;}
> > if(self->notes) {result = visit(self->notes,arg); if(result) return result;}
> > if(self->phase) {result = visit(self->phase,arg); if(result) return result;}
> >
> > // no need to Py_DECREF(dict) since the reference is borrowed
> > return 0;
> > }
> >
> > static int request_tp_clear(requestobject *self)
> > {
> > PyObject* tmp;
> > tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
> > tmp=self->connection; self->connection=NULL; Py_XDECREF(tmp);
> > tmp=self->server; self->server=NULL; Py_XDECREF(tmp);
> > tmp=self->next; self->next=NULL; Py_XDECREF(tmp);
> > tmp=self->prev; self->prev=NULL; Py_XDECREF(tmp);
> > tmp=self->main; self->main=NULL; Py_XDECREF(tmp);
> > tmp=self->headers_in; self->headers_in=NULL; Py_XDECREF(tmp);
> > tmp=self->headers_out; self->headers_out=NULL; Py_XDECREF(tmp);
> > tmp=self->err_headers_out; self->err_headers_out=NULL; Py_XDECREF(tmp);
> > tmp=self->subprocess_env; self->subprocess_env=NULL; Py_XDECREF(tmp);
> > tmp=self->notes; self->notes=NULL; Py_XDECREF(tmp);
> > tmp=self->phase; self->phase=NULL; Py_XDECREF(tmp);
> > tmp=self->hlo; self->hlo=NULL; Py_XDECREF(tmp);
> > tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
> > tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
> > tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
> > tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
> >
> > return 0;
> > }
> >
> > As you can guess from the source code, we'll have to add some code for
> > the session member when you'll integrate your code in subversion.
> >
> > Regards,
> > Nicolas
> >
> > 2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
> >
> >>Nicolas,
> >>
> >>It fails to compile when I add your bit of code. Sure looks like it
> >>should work though.
> >>
> >>requestobject.c: In function `request_tp_traverse':
> >>requestobject.c:1539: error: structure has no member named `session'
> >>requestobject.c:1540: error: structure has no member named `session'
> >>
> >>Just to be clear, it compiles and runs just fine without the extra check
> >> added into request_tp_traverse.
> >>
> >>Nicolas Lehuen wrote:
> >>
> >>>I'm re-reading the comment I wrote when I implemented tp_traverse :
> >>>
> >>>// only traverse its dictionary since other fields defined in
> >>>request_rec_mbrs with type T_OBJECT
> >>>// cannot be the source of memory leaks (unless you really want it)
> >>>
> >>>Turns out that we should at least traverse the next and prev in case
> >>>someone does something like req.next.my_prev = req. That's what I
> >>>meant by "unless you really want it".
> >>
> >>I wondered about that. :)
> >>
> >>Jim
> >>
> >>
> >>>It's pretty difficult to get
> >>>there (you need an internal redirect), but you can. So I'm going to
> >>>have a look at re-implementing the tp_traverse handler.
> >>>Regards,
> >>>Nicolas
> >>>
> >>>2005/6/12, Nicolas Lehuen <ni...@gmail.com>:
> >>>
> >>>
> >>>>Duh, I get it. If you add a member to the request object, and this
> >>>>member is not referenced in the request object's dictionary, then you
> >>>>have to add a special case for it in the tp_traverse handler. In
> >>>>requestobject.c :
> >>>>
> >>>>/**
> >>>>** request_tp_traverse
> >>>>**
> >>>>* Traversal of the request object
> >>>>*/
> >>>>static int request_tp_traverse(PyObject *self, visitproc visit, void *arg) {
> >>>> PyObject *dict,*values,*item,*str;
> >>>> int i,size;
> >>>>
> >>>> // only traverse its dictionary since other fields defined in
> >>>>request_rec_mbrs with type T_OBJECT
> >>>> // cannot be the source of memory leaks (unless you really want it)
> >>>> dict=*_PyObject_GetDictPtr(self);
> >>>> if(dict) {
> >>>> // this check is not needed, I guess, _PyObject_GetDictPtr
> >>>>always give a pointer to a dict object.
> >>>> if(PyDict_Check(dict)) {
> >>>> i = visit(dict,arg);
> >>>> if(i) {
> >>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> >>>>((requestobject*)self)->request_rec, "%s:%i Call to visit()
> >>>>failed",__LINE__,__FILE__);
> >>>> // no need to Py_DECREF(dict) since the reference is borrowed
> >>>> return i;
> >>>> }
> >>>> }
> >>>> else {
> >>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> >>>>((requestobject*)self)->request_rec, "%s:%i Expected a
> >>>>dictionary",__LINE__,__FILE__);
> >>>> }
> >>>> }
> >>>> else {
> >>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> >>>>((requestobject*)self)->request_rec, "%s:%i Expected a
> >>>>dictionary",__LINE__,__FILE__);
> >>>> }
> >>>>
> >>>> /* This is the new code that needs to be added to support the new
> >>>>session member */
> >>>> if(self->session) {
> >>>> i = visit(self->session,arg);
> >>>> if(i) {
> >>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> >>>>((requestobject*)self)->request_rec, "%s:%i Call to visit()
> >>>>failed",__LINE__,__FILE__);
> >>>> return i;
> >>>> }
> >>>> }
> >>>>
> >>>> // no need to Py_DECREF(dict) since the reference is borrowed
> >>>> return 0;
> >>>>}
> >>>>
> >>>>Regards,
> >>>>Nicolas
> >>>>
> >>>>2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
> >>>>
> >>>>
> >>>>>Hi Nicolas,
> >>>>>
> >>>>>Nicolas Lehuen wrote:
> >>>>>
> >>>>>
> >>>>>>Hi Jim,
> >>>>>>
> >>>>>>How where you creating the session object in requestobject.c ? If you
> >>>>>>were using PythonObject_New, then this may explain the memory leak.
> >>>>>>Objects that must be managed by the garbage collector have to be
> >>>>>>created with PyObject_GC_New.
> >>>>>
> >>>>>The c-code loads the python module and calls a function which generates
> >>>>>the session object.
> >>>>>
> >>>>>src/requestobject.c
> >>>>>static PyObject *req_get_session(requestobject *self, PyObject *args)
> >>>>>{
> >>>>> PyObject *m; // session module
> >>>>> PyObject *sid; // session id
> >>>>> PyObject *result;
> >>>>>
> >>>>> if (!self->session) {
> >>>>> sid = PyObject_CallMethod(self->subprocess_env, "get",
> >>>>> "(ss)","REDIRECT_PYSID", "");
> >>>>>
> >>>>> /********
> >>>>> * This is where the session instance is created
> >>>>> ********/
> >>>>> m = PyImport_ImportModule("mod_python.session2.TestSession");
> >>>>> self->session = PyObject_CallMethod(m, "create_session", "(OO)",
> >>>>> self, sid);
> >>>>>
> >>>>> Py_DECREF(m);
> >>>>> Py_DECREF(sid);
> >>>>> if (self->session == NULL)
> >>>>> return NULL;
> >>>>> }
> >>>>> result = self->session;
> >>>>> Py_INCREF(result);
> >>>>> return result;
> >>>>>}
> >>>>>
> >>>>>----------------------------------------------------------------
> >>>>>mod_python/session2/TestSession.py
> >>>>># emulate a simple test session
> >>>>>import _apache
> >>>>
> >>>>>from mod_python.Session import _new_sid
> >>>>
> >>>>>class TestSession(object):
> >>>>>
> >>>>> def __init__(self, req, sid=0, secret=None, timeout=0, lock=1):
> >>>>> req.log_error("TestSession.__init__")
> >>>>> # Circular Reference causes problem
> >>>>> self._req = req
> >>>>>
> >>>>> # keeping a reference to the server object does
> >>>>> # NOT cause a problem
> >>>>> self._server = req.server
> >>>>> self._lock = 1
> >>>>> self._locked = 0
> >>>>> self._sid = _new_sid(req)
> >>>>> self.lock()
> >>>>> self.unlock()
> >>>>>
> >>>>> def lock(self):
> >>>>> if self._lock:
> >>>>> _apache._global_lock(self._req.server, self._sid)
> >>>>> self._locked = 1
> >>>>>
> >>>>> def unlock(self):
> >>>>> # unlock will ocassionally segfault
> >>>>> if self._lock and self._locked:
> >>>>> _apache._global_unlock(self._req.server, self._sid)
> >>>>> self._locked = 0
> >>>>>
> >>>>>def create_session(req,sid):
> >>>>> return TestSession(req,sid)
> >>>>>
> >>>>
> >>
> >
>
>
Re: session handling - the next generation
Posted by Jim Gallacher <jg...@sympatico.ca>.
Nicolas Lehuen wrote:
> You should cast self to a requestobject* :
That worked. Running some tests.
Jim
>
> ((requestobject*)self)->session
>
> Anyway, I'm currently rewriting the whole request_tp_traverse /
> request_tp_clear / request_tp_clear functions like this :
>
> static void request_tp_dealloc(requestobject *self)
> {
> // de-register the object from the GC
> // before its deallocation, to prevent the
> // GC to run on a partially de-allocated object
> PyObject_GC_UnTrack(self);
>
> request_tp_clear(self);
>
> PyObject_GC_Del(self);
> }
>
> /**
> ** request_tp_traverse
> **
> * Traversal of the request object
> */
> static int request_tp_traverse(requestobject* self, visitproc visit,
> void *arg) {
> int result;
>
> if(self->dict) {result = visit(self->dict,arg); if(result) return result;}
> if(self->connection) {result = visit(self->connection,arg);
> if(result) return result;}
> if(self->server) {result = visit(self->server,arg); if(result)
> return result;}
> if(self->next) {result = visit(self->next,arg); if(result) return result;}
> if(self->prev) {result = visit(self->prev,arg); if(result) return result;}
> if(self->main) {result = visit(self->main,arg); if(result) return result;}
> if(self->headers_in) {result = visit(self->headers_in,arg);
> if(result) return result;}
> if(self->headers_out) {result = visit(self->headers_out,arg);
> if(result) return result;}
> if(self->err_headers_out) {result =
> visit(self->err_headers_out,arg); if(result) return result;}
> if(self->subprocess_env) {result =
> visit(self->subprocess_env,arg); if(result) return result;}
> if(self->notes) {result = visit(self->notes,arg); if(result) return result;}
> if(self->phase) {result = visit(self->phase,arg); if(result) return result;}
>
> // no need to Py_DECREF(dict) since the reference is borrowed
> return 0;
> }
>
> static int request_tp_clear(requestobject *self)
> {
> PyObject* tmp;
> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
> tmp=self->connection; self->connection=NULL; Py_XDECREF(tmp);
> tmp=self->server; self->server=NULL; Py_XDECREF(tmp);
> tmp=self->next; self->next=NULL; Py_XDECREF(tmp);
> tmp=self->prev; self->prev=NULL; Py_XDECREF(tmp);
> tmp=self->main; self->main=NULL; Py_XDECREF(tmp);
> tmp=self->headers_in; self->headers_in=NULL; Py_XDECREF(tmp);
> tmp=self->headers_out; self->headers_out=NULL; Py_XDECREF(tmp);
> tmp=self->err_headers_out; self->err_headers_out=NULL; Py_XDECREF(tmp);
> tmp=self->subprocess_env; self->subprocess_env=NULL; Py_XDECREF(tmp);
> tmp=self->notes; self->notes=NULL; Py_XDECREF(tmp);
> tmp=self->phase; self->phase=NULL; Py_XDECREF(tmp);
> tmp=self->hlo; self->hlo=NULL; Py_XDECREF(tmp);
> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
> tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
>
> return 0;
> }
>
> As you can guess from the source code, we'll have to add some code for
> the session member when you'll integrate your code in subversion.
>
> Regards,
> Nicolas
>
> 2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
>
>>Nicolas,
>>
>>It fails to compile when I add your bit of code. Sure looks like it
>>should work though.
>>
>>requestobject.c: In function `request_tp_traverse':
>>requestobject.c:1539: error: structure has no member named `session'
>>requestobject.c:1540: error: structure has no member named `session'
>>
>>Just to be clear, it compiles and runs just fine without the extra check
>> added into request_tp_traverse.
>>
>>Nicolas Lehuen wrote:
>>
>>>I'm re-reading the comment I wrote when I implemented tp_traverse :
>>>
>>>// only traverse its dictionary since other fields defined in
>>>request_rec_mbrs with type T_OBJECT
>>>// cannot be the source of memory leaks (unless you really want it)
>>>
>>>Turns out that we should at least traverse the next and prev in case
>>>someone does something like req.next.my_prev = req. That's what I
>>>meant by "unless you really want it".
>>
>>I wondered about that. :)
>>
>>Jim
>>
>>
>>>It's pretty difficult to get
>>>there (you need an internal redirect), but you can. So I'm going to
>>>have a look at re-implementing the tp_traverse handler.
>>>Regards,
>>>Nicolas
>>>
>>>2005/6/12, Nicolas Lehuen <ni...@gmail.com>:
>>>
>>>
>>>>Duh, I get it. If you add a member to the request object, and this
>>>>member is not referenced in the request object's dictionary, then you
>>>>have to add a special case for it in the tp_traverse handler. In
>>>>requestobject.c :
>>>>
>>>>/**
>>>>** request_tp_traverse
>>>>**
>>>>* Traversal of the request object
>>>>*/
>>>>static int request_tp_traverse(PyObject *self, visitproc visit, void *arg) {
>>>> PyObject *dict,*values,*item,*str;
>>>> int i,size;
>>>>
>>>> // only traverse its dictionary since other fields defined in
>>>>request_rec_mbrs with type T_OBJECT
>>>> // cannot be the source of memory leaks (unless you really want it)
>>>> dict=*_PyObject_GetDictPtr(self);
>>>> if(dict) {
>>>> // this check is not needed, I guess, _PyObject_GetDictPtr
>>>>always give a pointer to a dict object.
>>>> if(PyDict_Check(dict)) {
>>>> i = visit(dict,arg);
>>>> if(i) {
>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>>>((requestobject*)self)->request_rec, "%s:%i Call to visit()
>>>>failed",__LINE__,__FILE__);
>>>> // no need to Py_DECREF(dict) since the reference is borrowed
>>>> return i;
>>>> }
>>>> }
>>>> else {
>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>>>((requestobject*)self)->request_rec, "%s:%i Expected a
>>>>dictionary",__LINE__,__FILE__);
>>>> }
>>>> }
>>>> else {
>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>>>((requestobject*)self)->request_rec, "%s:%i Expected a
>>>>dictionary",__LINE__,__FILE__);
>>>> }
>>>>
>>>> /* This is the new code that needs to be added to support the new
>>>>session member */
>>>> if(self->session) {
>>>> i = visit(self->session,arg);
>>>> if(i) {
>>>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>>>((requestobject*)self)->request_rec, "%s:%i Call to visit()
>>>>failed",__LINE__,__FILE__);
>>>> return i;
>>>> }
>>>> }
>>>>
>>>> // no need to Py_DECREF(dict) since the reference is borrowed
>>>> return 0;
>>>>}
>>>>
>>>>Regards,
>>>>Nicolas
>>>>
>>>>2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
>>>>
>>>>
>>>>>Hi Nicolas,
>>>>>
>>>>>Nicolas Lehuen wrote:
>>>>>
>>>>>
>>>>>>Hi Jim,
>>>>>>
>>>>>>How where you creating the session object in requestobject.c ? If you
>>>>>>were using PythonObject_New, then this may explain the memory leak.
>>>>>>Objects that must be managed by the garbage collector have to be
>>>>>>created with PyObject_GC_New.
>>>>>
>>>>>The c-code loads the python module and calls a function which generates
>>>>>the session object.
>>>>>
>>>>>src/requestobject.c
>>>>>static PyObject *req_get_session(requestobject *self, PyObject *args)
>>>>>{
>>>>> PyObject *m; // session module
>>>>> PyObject *sid; // session id
>>>>> PyObject *result;
>>>>>
>>>>> if (!self->session) {
>>>>> sid = PyObject_CallMethod(self->subprocess_env, "get",
>>>>> "(ss)","REDIRECT_PYSID", "");
>>>>>
>>>>> /********
>>>>> * This is where the session instance is created
>>>>> ********/
>>>>> m = PyImport_ImportModule("mod_python.session2.TestSession");
>>>>> self->session = PyObject_CallMethod(m, "create_session", "(OO)",
>>>>> self, sid);
>>>>>
>>>>> Py_DECREF(m);
>>>>> Py_DECREF(sid);
>>>>> if (self->session == NULL)
>>>>> return NULL;
>>>>> }
>>>>> result = self->session;
>>>>> Py_INCREF(result);
>>>>> return result;
>>>>>}
>>>>>
>>>>>----------------------------------------------------------------
>>>>>mod_python/session2/TestSession.py
>>>>># emulate a simple test session
>>>>>import _apache
>>>>
>>>>>from mod_python.Session import _new_sid
>>>>
>>>>>class TestSession(object):
>>>>>
>>>>> def __init__(self, req, sid=0, secret=None, timeout=0, lock=1):
>>>>> req.log_error("TestSession.__init__")
>>>>> # Circular Reference causes problem
>>>>> self._req = req
>>>>>
>>>>> # keeping a reference to the server object does
>>>>> # NOT cause a problem
>>>>> self._server = req.server
>>>>> self._lock = 1
>>>>> self._locked = 0
>>>>> self._sid = _new_sid(req)
>>>>> self.lock()
>>>>> self.unlock()
>>>>>
>>>>> def lock(self):
>>>>> if self._lock:
>>>>> _apache._global_lock(self._req.server, self._sid)
>>>>> self._locked = 1
>>>>>
>>>>> def unlock(self):
>>>>> # unlock will ocassionally segfault
>>>>> if self._lock and self._locked:
>>>>> _apache._global_unlock(self._req.server, self._sid)
>>>>> self._locked = 0
>>>>>
>>>>>def create_session(req,sid):
>>>>> return TestSession(req,sid)
>>>>>
>>>>
>>
>
Re: session handling - the next generation
Posted by Nicolas Lehuen <ni...@gmail.com>.
You should cast self to a requestobject* :
((requestobject*)self)->session
Anyway, I'm currently rewriting the whole request_tp_traverse /
request_tp_clear / request_tp_clear functions like this :
static void request_tp_dealloc(requestobject *self)
{
// de-register the object from the GC
// before its deallocation, to prevent the
// GC to run on a partially de-allocated object
PyObject_GC_UnTrack(self);
request_tp_clear(self);
PyObject_GC_Del(self);
}
/**
** request_tp_traverse
**
* Traversal of the request object
*/
static int request_tp_traverse(requestobject* self, visitproc visit,
void *arg) {
int result;
if(self->dict) {result = visit(self->dict,arg); if(result) return result;}
if(self->connection) {result = visit(self->connection,arg);
if(result) return result;}
if(self->server) {result = visit(self->server,arg); if(result)
return result;}
if(self->next) {result = visit(self->next,arg); if(result) return result;}
if(self->prev) {result = visit(self->prev,arg); if(result) return result;}
if(self->main) {result = visit(self->main,arg); if(result) return result;}
if(self->headers_in) {result = visit(self->headers_in,arg);
if(result) return result;}
if(self->headers_out) {result = visit(self->headers_out,arg);
if(result) return result;}
if(self->err_headers_out) {result =
visit(self->err_headers_out,arg); if(result) return result;}
if(self->subprocess_env) {result =
visit(self->subprocess_env,arg); if(result) return result;}
if(self->notes) {result = visit(self->notes,arg); if(result) return result;}
if(self->phase) {result = visit(self->phase,arg); if(result) return result;}
// no need to Py_DECREF(dict) since the reference is borrowed
return 0;
}
static int request_tp_clear(requestobject *self)
{
PyObject* tmp;
tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
tmp=self->connection; self->connection=NULL; Py_XDECREF(tmp);
tmp=self->server; self->server=NULL; Py_XDECREF(tmp);
tmp=self->next; self->next=NULL; Py_XDECREF(tmp);
tmp=self->prev; self->prev=NULL; Py_XDECREF(tmp);
tmp=self->main; self->main=NULL; Py_XDECREF(tmp);
tmp=self->headers_in; self->headers_in=NULL; Py_XDECREF(tmp);
tmp=self->headers_out; self->headers_out=NULL; Py_XDECREF(tmp);
tmp=self->err_headers_out; self->err_headers_out=NULL; Py_XDECREF(tmp);
tmp=self->subprocess_env; self->subprocess_env=NULL; Py_XDECREF(tmp);
tmp=self->notes; self->notes=NULL; Py_XDECREF(tmp);
tmp=self->phase; self->phase=NULL; Py_XDECREF(tmp);
tmp=self->hlo; self->hlo=NULL; Py_XDECREF(tmp);
tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
tmp=self->dict; self->dict=NULL; Py_XDECREF(tmp);
return 0;
}
As you can guess from the source code, we'll have to add some code for
the session member when you'll integrate your code in subversion.
Regards,
Nicolas
2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
> Nicolas,
>
> It fails to compile when I add your bit of code. Sure looks like it
> should work though.
>
> requestobject.c: In function `request_tp_traverse':
> requestobject.c:1539: error: structure has no member named `session'
> requestobject.c:1540: error: structure has no member named `session'
>
> Just to be clear, it compiles and runs just fine without the extra check
> added into request_tp_traverse.
>
> Nicolas Lehuen wrote:
> > I'm re-reading the comment I wrote when I implemented tp_traverse :
> >
> > // only traverse its dictionary since other fields defined in
> > request_rec_mbrs with type T_OBJECT
> > // cannot be the source of memory leaks (unless you really want it)
> >
> > Turns out that we should at least traverse the next and prev in case
> > someone does something like req.next.my_prev = req. That's what I
> > meant by "unless you really want it".
>
> I wondered about that. :)
>
> Jim
>
> > It's pretty difficult to get
> > there (you need an internal redirect), but you can. So I'm going to
> > have a look at re-implementing the tp_traverse handler.
> > Regards,
> > Nicolas
> >
> > 2005/6/12, Nicolas Lehuen <ni...@gmail.com>:
> >
> >>Duh, I get it. If you add a member to the request object, and this
> >>member is not referenced in the request object's dictionary, then you
> >>have to add a special case for it in the tp_traverse handler. In
> >>requestobject.c :
> >>
> >>/**
> >> ** request_tp_traverse
> >> **
> >> * Traversal of the request object
> >> */
> >>static int request_tp_traverse(PyObject *self, visitproc visit, void *arg) {
> >> PyObject *dict,*values,*item,*str;
> >> int i,size;
> >>
> >> // only traverse its dictionary since other fields defined in
> >>request_rec_mbrs with type T_OBJECT
> >> // cannot be the source of memory leaks (unless you really want it)
> >> dict=*_PyObject_GetDictPtr(self);
> >> if(dict) {
> >> // this check is not needed, I guess, _PyObject_GetDictPtr
> >>always give a pointer to a dict object.
> >> if(PyDict_Check(dict)) {
> >> i = visit(dict,arg);
> >> if(i) {
> >> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> >>((requestobject*)self)->request_rec, "%s:%i Call to visit()
> >>failed",__LINE__,__FILE__);
> >> // no need to Py_DECREF(dict) since the reference is borrowed
> >> return i;
> >> }
> >> }
> >> else {
> >> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> >>((requestobject*)self)->request_rec, "%s:%i Expected a
> >>dictionary",__LINE__,__FILE__);
> >> }
> >> }
> >> else {
> >> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> >>((requestobject*)self)->request_rec, "%s:%i Expected a
> >>dictionary",__LINE__,__FILE__);
> >> }
> >>
> >> /* This is the new code that needs to be added to support the new
> >>session member */
> >> if(self->session) {
> >> i = visit(self->session,arg);
> >> if(i) {
> >> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> >>((requestobject*)self)->request_rec, "%s:%i Call to visit()
> >>failed",__LINE__,__FILE__);
> >> return i;
> >> }
> >> }
> >>
> >> // no need to Py_DECREF(dict) since the reference is borrowed
> >> return 0;
> >>}
> >>
> >>Regards,
> >>Nicolas
> >>
> >>2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
> >>
> >>>Hi Nicolas,
> >>>
> >>>Nicolas Lehuen wrote:
> >>>
> >>>>Hi Jim,
> >>>>
> >>>>How where you creating the session object in requestobject.c ? If you
> >>>>were using PythonObject_New, then this may explain the memory leak.
> >>>>Objects that must be managed by the garbage collector have to be
> >>>>created with PyObject_GC_New.
> >>>
> >>>The c-code loads the python module and calls a function which generates
> >>>the session object.
> >>>
> >>>src/requestobject.c
> >>>static PyObject *req_get_session(requestobject *self, PyObject *args)
> >>>{
> >>> PyObject *m; // session module
> >>> PyObject *sid; // session id
> >>> PyObject *result;
> >>>
> >>> if (!self->session) {
> >>> sid = PyObject_CallMethod(self->subprocess_env, "get",
> >>> "(ss)","REDIRECT_PYSID", "");
> >>>
> >>> /********
> >>> * This is where the session instance is created
> >>> ********/
> >>> m = PyImport_ImportModule("mod_python.session2.TestSession");
> >>> self->session = PyObject_CallMethod(m, "create_session", "(OO)",
> >>> self, sid);
> >>>
> >>> Py_DECREF(m);
> >>> Py_DECREF(sid);
> >>> if (self->session == NULL)
> >>> return NULL;
> >>> }
> >>> result = self->session;
> >>> Py_INCREF(result);
> >>> return result;
> >>>}
> >>>
> >>>----------------------------------------------------------------
> >>>mod_python/session2/TestSession.py
> >>># emulate a simple test session
> >>>import _apache
> >>>from mod_python.Session import _new_sid
> >>>
> >>>class TestSession(object):
> >>>
> >>> def __init__(self, req, sid=0, secret=None, timeout=0, lock=1):
> >>> req.log_error("TestSession.__init__")
> >>> # Circular Reference causes problem
> >>> self._req = req
> >>>
> >>> # keeping a reference to the server object does
> >>> # NOT cause a problem
> >>> self._server = req.server
> >>> self._lock = 1
> >>> self._locked = 0
> >>> self._sid = _new_sid(req)
> >>> self.lock()
> >>> self.unlock()
> >>>
> >>> def lock(self):
> >>> if self._lock:
> >>> _apache._global_lock(self._req.server, self._sid)
> >>> self._locked = 1
> >>>
> >>> def unlock(self):
> >>> # unlock will ocassionally segfault
> >>> if self._lock and self._locked:
> >>> _apache._global_unlock(self._req.server, self._sid)
> >>> self._locked = 0
> >>>
> >>>def create_session(req,sid):
> >>> return TestSession(req,sid)
> >>>
> >>
> >
>
>
Re: session handling - the next generation
Posted by Jim Gallacher <jg...@sympatico.ca>.
Nicolas,
It fails to compile when I add your bit of code. Sure looks like it
should work though.
requestobject.c: In function `request_tp_traverse':
requestobject.c:1539: error: structure has no member named `session'
requestobject.c:1540: error: structure has no member named `session'
Just to be clear, it compiles and runs just fine without the extra check
added into request_tp_traverse.
Nicolas Lehuen wrote:
> I'm re-reading the comment I wrote when I implemented tp_traverse :
>
> // only traverse its dictionary since other fields defined in
> request_rec_mbrs with type T_OBJECT
> // cannot be the source of memory leaks (unless you really want it)
>
> Turns out that we should at least traverse the next and prev in case
> someone does something like req.next.my_prev = req. That's what I
> meant by "unless you really want it".
I wondered about that. :)
Jim
> It's pretty difficult to get
> there (you need an internal redirect), but you can. So I'm going to
> have a look at re-implementing the tp_traverse handler.
> Regards,
> Nicolas
>
> 2005/6/12, Nicolas Lehuen <ni...@gmail.com>:
>
>>Duh, I get it. If you add a member to the request object, and this
>>member is not referenced in the request object's dictionary, then you
>>have to add a special case for it in the tp_traverse handler. In
>>requestobject.c :
>>
>>/**
>> ** request_tp_traverse
>> **
>> * Traversal of the request object
>> */
>>static int request_tp_traverse(PyObject *self, visitproc visit, void *arg) {
>> PyObject *dict,*values,*item,*str;
>> int i,size;
>>
>> // only traverse its dictionary since other fields defined in
>>request_rec_mbrs with type T_OBJECT
>> // cannot be the source of memory leaks (unless you really want it)
>> dict=*_PyObject_GetDictPtr(self);
>> if(dict) {
>> // this check is not needed, I guess, _PyObject_GetDictPtr
>>always give a pointer to a dict object.
>> if(PyDict_Check(dict)) {
>> i = visit(dict,arg);
>> if(i) {
>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>((requestobject*)self)->request_rec, "%s:%i Call to visit()
>>failed",__LINE__,__FILE__);
>> // no need to Py_DECREF(dict) since the reference is borrowed
>> return i;
>> }
>> }
>> else {
>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>((requestobject*)self)->request_rec, "%s:%i Expected a
>>dictionary",__LINE__,__FILE__);
>> }
>> }
>> else {
>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>((requestobject*)self)->request_rec, "%s:%i Expected a
>>dictionary",__LINE__,__FILE__);
>> }
>>
>> /* This is the new code that needs to be added to support the new
>>session member */
>> if(self->session) {
>> i = visit(self->session,arg);
>> if(i) {
>> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
>>((requestobject*)self)->request_rec, "%s:%i Call to visit()
>>failed",__LINE__,__FILE__);
>> return i;
>> }
>> }
>>
>> // no need to Py_DECREF(dict) since the reference is borrowed
>> return 0;
>>}
>>
>>Regards,
>>Nicolas
>>
>>2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
>>
>>>Hi Nicolas,
>>>
>>>Nicolas Lehuen wrote:
>>>
>>>>Hi Jim,
>>>>
>>>>How where you creating the session object in requestobject.c ? If you
>>>>were using PythonObject_New, then this may explain the memory leak.
>>>>Objects that must be managed by the garbage collector have to be
>>>>created with PyObject_GC_New.
>>>
>>>The c-code loads the python module and calls a function which generates
>>>the session object.
>>>
>>>src/requestobject.c
>>>static PyObject *req_get_session(requestobject *self, PyObject *args)
>>>{
>>> PyObject *m; // session module
>>> PyObject *sid; // session id
>>> PyObject *result;
>>>
>>> if (!self->session) {
>>> sid = PyObject_CallMethod(self->subprocess_env, "get",
>>> "(ss)","REDIRECT_PYSID", "");
>>>
>>> /********
>>> * This is where the session instance is created
>>> ********/
>>> m = PyImport_ImportModule("mod_python.session2.TestSession");
>>> self->session = PyObject_CallMethod(m, "create_session", "(OO)",
>>> self, sid);
>>>
>>> Py_DECREF(m);
>>> Py_DECREF(sid);
>>> if (self->session == NULL)
>>> return NULL;
>>> }
>>> result = self->session;
>>> Py_INCREF(result);
>>> return result;
>>>}
>>>
>>>----------------------------------------------------------------
>>>mod_python/session2/TestSession.py
>>># emulate a simple test session
>>>import _apache
>>>from mod_python.Session import _new_sid
>>>
>>>class TestSession(object):
>>>
>>> def __init__(self, req, sid=0, secret=None, timeout=0, lock=1):
>>> req.log_error("TestSession.__init__")
>>> # Circular Reference causes problem
>>> self._req = req
>>>
>>> # keeping a reference to the server object does
>>> # NOT cause a problem
>>> self._server = req.server
>>> self._lock = 1
>>> self._locked = 0
>>> self._sid = _new_sid(req)
>>> self.lock()
>>> self.unlock()
>>>
>>> def lock(self):
>>> if self._lock:
>>> _apache._global_lock(self._req.server, self._sid)
>>> self._locked = 1
>>>
>>> def unlock(self):
>>> # unlock will ocassionally segfault
>>> if self._lock and self._locked:
>>> _apache._global_unlock(self._req.server, self._sid)
>>> self._locked = 0
>>>
>>>def create_session(req,sid):
>>> return TestSession(req,sid)
>>>
>>
>
Re: session handling - the next generation
Posted by Nicolas Lehuen <ni...@gmail.com>.
I'm re-reading the comment I wrote when I implemented tp_traverse :
// only traverse its dictionary since other fields defined in
request_rec_mbrs with type T_OBJECT
// cannot be the source of memory leaks (unless you really want it)
Turns out that we should at least traverse the next and prev in case
someone does something like req.next.my_prev = req. That's what I
meant by "unless you really want it". It's pretty difficult to get
there (you need an internal redirect), but you can. So I'm going to
have a look at re-implementing the tp_traverse handler.
Regards,
Nicolas
2005/6/12, Nicolas Lehuen <ni...@gmail.com>:
> Duh, I get it. If you add a member to the request object, and this
> member is not referenced in the request object's dictionary, then you
> have to add a special case for it in the tp_traverse handler. In
> requestobject.c :
>
> /**
> ** request_tp_traverse
> **
> * Traversal of the request object
> */
> static int request_tp_traverse(PyObject *self, visitproc visit, void *arg) {
> PyObject *dict,*values,*item,*str;
> int i,size;
>
> // only traverse its dictionary since other fields defined in
> request_rec_mbrs with type T_OBJECT
> // cannot be the source of memory leaks (unless you really want it)
> dict=*_PyObject_GetDictPtr(self);
> if(dict) {
> // this check is not needed, I guess, _PyObject_GetDictPtr
> always give a pointer to a dict object.
> if(PyDict_Check(dict)) {
> i = visit(dict,arg);
> if(i) {
> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> ((requestobject*)self)->request_rec, "%s:%i Call to visit()
> failed",__LINE__,__FILE__);
> // no need to Py_DECREF(dict) since the reference is borrowed
> return i;
> }
> }
> else {
> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> ((requestobject*)self)->request_rec, "%s:%i Expected a
> dictionary",__LINE__,__FILE__);
> }
> }
> else {
> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> ((requestobject*)self)->request_rec, "%s:%i Expected a
> dictionary",__LINE__,__FILE__);
> }
>
> /* This is the new code that needs to be added to support the new
> session member */
> if(self->session) {
> i = visit(self->session,arg);
> if(i) {
> ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
> ((requestobject*)self)->request_rec, "%s:%i Call to visit()
> failed",__LINE__,__FILE__);
> return i;
> }
> }
>
> // no need to Py_DECREF(dict) since the reference is borrowed
> return 0;
> }
>
> Regards,
> Nicolas
>
> 2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
> > Hi Nicolas,
> >
> > Nicolas Lehuen wrote:
> > > Hi Jim,
> > >
> > > How where you creating the session object in requestobject.c ? If you
> > > were using PythonObject_New, then this may explain the memory leak.
> > > Objects that must be managed by the garbage collector have to be
> > > created with PyObject_GC_New.
> >
> > The c-code loads the python module and calls a function which generates
> > the session object.
> >
> > src/requestobject.c
> > static PyObject *req_get_session(requestobject *self, PyObject *args)
> > {
> > PyObject *m; // session module
> > PyObject *sid; // session id
> > PyObject *result;
> >
> > if (!self->session) {
> > sid = PyObject_CallMethod(self->subprocess_env, "get",
> > "(ss)","REDIRECT_PYSID", "");
> >
> > /********
> > * This is where the session instance is created
> > ********/
> > m = PyImport_ImportModule("mod_python.session2.TestSession");
> > self->session = PyObject_CallMethod(m, "create_session", "(OO)",
> > self, sid);
> >
> > Py_DECREF(m);
> > Py_DECREF(sid);
> > if (self->session == NULL)
> > return NULL;
> > }
> > result = self->session;
> > Py_INCREF(result);
> > return result;
> > }
> >
> > ----------------------------------------------------------------
> > mod_python/session2/TestSession.py
> > # emulate a simple test session
> > import _apache
> > from mod_python.Session import _new_sid
> >
> > class TestSession(object):
> >
> > def __init__(self, req, sid=0, secret=None, timeout=0, lock=1):
> > req.log_error("TestSession.__init__")
> > # Circular Reference causes problem
> > self._req = req
> >
> > # keeping a reference to the server object does
> > # NOT cause a problem
> > self._server = req.server
> > self._lock = 1
> > self._locked = 0
> > self._sid = _new_sid(req)
> > self.lock()
> > self.unlock()
> >
> > def lock(self):
> > if self._lock:
> > _apache._global_lock(self._req.server, self._sid)
> > self._locked = 1
> >
> > def unlock(self):
> > # unlock will ocassionally segfault
> > if self._lock and self._locked:
> > _apache._global_unlock(self._req.server, self._sid)
> > self._locked = 0
> >
> > def create_session(req,sid):
> > return TestSession(req,sid)
> >
>
Re: session handling - the next generation
Posted by Nicolas Lehuen <ni...@gmail.com>.
Duh, I get it. If you add a member to the request object, and this
member is not referenced in the request object's dictionary, then you
have to add a special case for it in the tp_traverse handler. In
requestobject.c :
/**
** request_tp_traverse
**
* Traversal of the request object
*/
static int request_tp_traverse(PyObject *self, visitproc visit, void *arg) {
PyObject *dict,*values,*item,*str;
int i,size;
// only traverse its dictionary since other fields defined in
request_rec_mbrs with type T_OBJECT
// cannot be the source of memory leaks (unless you really want it)
dict=*_PyObject_GetDictPtr(self);
if(dict) {
// this check is not needed, I guess, _PyObject_GetDictPtr
always give a pointer to a dict object.
if(PyDict_Check(dict)) {
i = visit(dict,arg);
if(i) {
ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
((requestobject*)self)->request_rec, "%s:%i Call to visit()
failed",__LINE__,__FILE__);
// no need to Py_DECREF(dict) since the reference is borrowed
return i;
}
}
else {
ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
((requestobject*)self)->request_rec, "%s:%i Expected a
dictionary",__LINE__,__FILE__);
}
}
else {
ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
((requestobject*)self)->request_rec, "%s:%i Expected a
dictionary",__LINE__,__FILE__);
}
/* This is the new code that needs to be added to support the new
session member */
if(self->session) {
i = visit(self->session,arg);
if(i) {
ap_log_rerror(APLOG_MARK, APLOG_NOERRNO|APLOG_ERR, 0,
((requestobject*)self)->request_rec, "%s:%i Call to visit()
failed",__LINE__,__FILE__);
return i;
}
}
// no need to Py_DECREF(dict) since the reference is borrowed
return 0;
}
Regards,
Nicolas
2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
> Hi Nicolas,
>
> Nicolas Lehuen wrote:
> > Hi Jim,
> >
> > How where you creating the session object in requestobject.c ? If you
> > were using PythonObject_New, then this may explain the memory leak.
> > Objects that must be managed by the garbage collector have to be
> > created with PyObject_GC_New.
>
> The c-code loads the python module and calls a function which generates
> the session object.
>
> src/requestobject.c
> static PyObject *req_get_session(requestobject *self, PyObject *args)
> {
> PyObject *m; // session module
> PyObject *sid; // session id
> PyObject *result;
>
> if (!self->session) {
> sid = PyObject_CallMethod(self->subprocess_env, "get",
> "(ss)","REDIRECT_PYSID", "");
>
> /********
> * This is where the session instance is created
> ********/
> m = PyImport_ImportModule("mod_python.session2.TestSession");
> self->session = PyObject_CallMethod(m, "create_session", "(OO)",
> self, sid);
>
> Py_DECREF(m);
> Py_DECREF(sid);
> if (self->session == NULL)
> return NULL;
> }
> result = self->session;
> Py_INCREF(result);
> return result;
> }
>
> ----------------------------------------------------------------
> mod_python/session2/TestSession.py
> # emulate a simple test session
> import _apache
> from mod_python.Session import _new_sid
>
> class TestSession(object):
>
> def __init__(self, req, sid=0, secret=None, timeout=0, lock=1):
> req.log_error("TestSession.__init__")
> # Circular Reference causes problem
> self._req = req
>
> # keeping a reference to the server object does
> # NOT cause a problem
> self._server = req.server
> self._lock = 1
> self._locked = 0
> self._sid = _new_sid(req)
> self.lock()
> self.unlock()
>
> def lock(self):
> if self._lock:
> _apache._global_lock(self._req.server, self._sid)
> self._locked = 1
>
> def unlock(self):
> # unlock will ocassionally segfault
> if self._lock and self._locked:
> _apache._global_unlock(self._req.server, self._sid)
> self._locked = 0
>
> def create_session(req,sid):
> return TestSession(req,sid)
>
Re: A Styling note [was Re: session handling - the next generation]
Posted by Jim Gallacher <jg...@sympatico.ca>.
I shall comply. :)
I would have redacted them from the final version anyway. I hate
comments on variables - if I need a comment I should just have better
variable name in the first place.
Regards,
Jim
Gregory (Grisha) Trubetskoy wrote:
>
>
> On Sun, 12 Jun 2005, Jim Gallacher wrote:
>
>> PyObject *m; // session module
>
>
>
> If I'm not mistaken, the // comments are not strictly C-compliant, we
> should stay away from them.
>
> Generally, we should try our best to adhere to this:
>
> http://httpd.apache.org/dev/styleguide.html
>
> I know mod_python isn't 100% compliant, but there is no reason to make
> it even more non-compliant :-)
>
> Grisha
>
A Styling note [was Re: session handling - the next generation]
Posted by "Gregory (Grisha) Trubetskoy" <gr...@apache.org>.
On Sun, 12 Jun 2005, Jim Gallacher wrote:
> PyObject *m; // session module
If I'm not mistaken, the // comments are not strictly C-compliant, we
should stay away from them.
Generally, we should try our best to adhere to this:
http://httpd.apache.org/dev/styleguide.html
I know mod_python isn't 100% compliant, but there is no reason to make it
even more non-compliant :-)
Grisha
Re: session handling - the next generation
Posted by Jim Gallacher <jg...@sympatico.ca>.
Hi Nicolas,
Nicolas Lehuen wrote:
> Hi Jim,
>
> How where you creating the session object in requestobject.c ? If you
> were using PythonObject_New, then this may explain the memory leak.
> Objects that must be managed by the garbage collector have to be
> created with PyObject_GC_New.
The c-code loads the python module and calls a function which generates
the session object.
src/requestobject.c
static PyObject *req_get_session(requestobject *self, PyObject *args)
{
PyObject *m; // session module
PyObject *sid; // session id
PyObject *result;
if (!self->session) {
sid = PyObject_CallMethod(self->subprocess_env, "get",
"(ss)","REDIRECT_PYSID", "");
/********
* This is where the session instance is created
********/
m = PyImport_ImportModule("mod_python.session2.TestSession");
self->session = PyObject_CallMethod(m, "create_session", "(OO)",
self, sid);
Py_DECREF(m);
Py_DECREF(sid);
if (self->session == NULL)
return NULL;
}
result = self->session;
Py_INCREF(result);
return result;
}
----------------------------------------------------------------
mod_python/session2/TestSession.py
# emulate a simple test session
import _apache
from mod_python.Session import _new_sid
class TestSession(object):
def __init__(self, req, sid=0, secret=None, timeout=0, lock=1):
req.log_error("TestSession.__init__")
# Circular Reference causes problem
self._req = req
# keeping a reference to the server object does
# NOT cause a problem
self._server = req.server
self._lock = 1
self._locked = 0
self._sid = _new_sid(req)
self.lock()
self.unlock()
def lock(self):
if self._lock:
_apache._global_lock(self._req.server, self._sid)
self._locked = 1
def unlock(self):
# unlock will ocassionally segfault
if self._lock and self._locked:
_apache._global_unlock(self._req.server, self._sid)
self._locked = 0
def create_session(req,sid):
return TestSession(req,sid)
Re: session handling - the next generation
Posted by Nicolas Lehuen <ni...@gmail.com>.
Hi Jim,
How where you creating the session object in requestobject.c ? If you
were using PythonObject_New, then this may explain the memory leak.
Objects that must be managed by the garbage collector have to be
created with PyObject_GC_New.
Regards,
Nicolas
2005/6/12, Jim Gallacher <jg...@sympatico.ca>:
> Replying to my own email,
>
> It looks like there was problem with a circular reference in
> req->session->req. This resulted in a memory leak which I had
> mis-interpreted as a reference count problem in req_get_session.
>
> As a result, I was not doing a Py_INCREF(self->session) in
> req_get_session(). This resulted in the intermittent segfault. (I just
> *knew* I was going to screw up the reference counting. :) )
>
> Refactoring TestSession to remove saved reference to the request object
> fixed the memory leak problem. Adding Py_INCREF(self->session) in
> req_get_session() fixed the segfault problem.
>
> I still don't understand why the memory lead only occurs when the
> session object is created in requestobject.c and not when created
> directly in a python script, but there you go.
>
> Jim
>
> Jim Gallacher wrote:
> > Hi All,
> >
> > Sorry for the long post, but I think I have a nice contribution to the
> > session handling mechanism, but I've got a bug I just can't track down.
> > Make yourself a cup of coffee, sit down and have a read.
> >
> > In an effort to make session handling more transparent and less error
> > prone, I've added a new method called get_session to the request object.
> >
> > (As an aside, I've also created a new apache config directive
> > PythonSessionOptions, a req.get_session_options() to read the directive,
> > and a small internal change to req.internal_redirect() to pass a session
> > to a redirected request).
> >
> > req.get_session() returns the session instance if it exists or creates a
> > new one. Internally it calls some python code,
> > mod_python.Session.create_session(), which does the actual work. The
> > user will only use get_session() to access their session, and so all our
> > deadlocking issues are gone forever.
> >
> > It mostly works, but I've found that approx 1 in 10000 requests will
> > segfault. Calling mod_python.Session.create_session directly does not
> > segfault (tested for 2000000 requests) so that is not the issue.
> >
> > I really hope someone can spot where the problem is. I've reduced
> > everything to the simplest possible case. The unlock method in
> > TestSession very rarely segfault or raise an exception and then
> > segfault, but very rarely. It only fails 0.004% of the time based on
> > 500000 requests. I suspect some kind of garbage collection issue, maybe
> > a circular reference caused by req->session->_req, but I don't
> > understand why it works 99.996 % of the time.
> >
> > I've included the full error.log for 5000000 requests at the end of this
> > message, but the ones that caught my eye as unusual are:
> >
> > [Tue Jun 07 17:52:02 2005] [notice] child pid 15066 exit signal
> > Segmentation fault (11)
> > Fatal Python error: deletion of interned string failed
> >
> > [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: Traceback (most recent call last):
> > [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: File "/usr/lib/python2.3/site-packages/mod_python/apache.py",
> > line 299, in HandlerDispatch\n result = object(req)
> > [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: File "/var/www/mp/mptest.py", line 19, in handler\n sess =
> > req.get_session()
> > [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: File
> > "/usr/lib/python2.3/site-packages/mod_python/TestSession.py", line 46,
> > in create_session\n return TestSession(req,sid)
> > [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: File
> > "/usr/lib/python2.3/site-packages/mod_python/TestSession.py", line 40,
> > in unlock\n if self._lock and self._locked:
> > [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: AttributeError: 'TestSession' object has no attribute '_lock'
> >
> > [Tue Jun 07 18:04:03 2005] [notice] child pid 16170 exit signal
> > Segmentation fault (11)
> > Fatal Python error: GC object already tracked
> >
> >
> > I hope someone can give me some insight to the problem. Note that I'm
> > using mpm-worker.
> >
> > Regards,
> > Jim
> >
> > ------------------------------------------------------------------------
> > mptest.py
> > # the test handler
> >
> > from mod_python import apache
> > from mod_python import TestSession
> >
> > def handler(req):
> > req.content_type = 'text/plain'
> > req.write('mptest.py')
> >
> > if req.path_info:
> > test_case = req.path_info[1:]
> > else:
> > test_case = '1'
> >
> > if test_case == '1':
> > # the good case - always works
> > sess = TestSession.create_session(req,None)
> > elif test_case == '2':
> > # the bad case - sometimes segfaults
> > sess = req.get_session()
> > else:
> > req.write('no test specified')
> >
> > return apache.OK
> >
> > ---------------------------------------------------------------------------------------------------
> >
> > mod_python/TestSession.py
> > # emulate a simple test session
> > import _apache
> > from Session import _new_sid
> >
> > class TestSession(object):
> >
> > def __init__(self, req, sid=0, secret=None, timeout=0, lock=1):
> > req.log_error("TestSession.__init__")
>
> ### CIRCULAR REFERENCE self._req = req
> ### causes memory leak
> > self._req = req
> ###
>
> > self._lock = 1
> > self._locked = 0
> > self._sid = _new_sid(req)
> > self.lock()
> > self.unlock()
> >
> > def lock(self):
> > if self._lock:
> > _apache._global_lock(self._req.server, self._sid)
> > self._locked = 1
> >
> > def unlock(self):
> > # unlock will ocassionally segfault
> > if self._lock and self._locked:
> > _apache._global_unlock(self._req.server, self._sid)
> > self._locked = 0
> >
> > def create_session(req,sid):
> > return TestSession(req,sid)
> >
> > -------------------------------------------------------------------------------------------------
> >
> > src/requestobject.c
> > In the following code session is a PyObject* defined in the the
> > requestobject struct.
> >
> > static PyObject *req_get_session(requestobject *self, PyObject *args)
> > {
> > PyObject *m;
> > PyObject *sid;
> >
> > if (!self->session) {
> > m = PyImport_ImportModule("mod_python.TestSession");
> > /* is this redirected request with an existing session? */
> > sid = PyObject_CallMethod(self->subprocess_env, "get",
> > "(ss)","REDIRECT_PYSID", "");
> >
> > self->session = PyObject_CallMethod(m, "create_session", "(OO)",
> > self, sid);
> > Py_DECREF(m);
> > Py_DECREF(sid);
> > if (self->session == NULL)
> > return NULL;
> > }
>
> // increase ref count to fix segfault problem
> Py_INCREF(self->session);
>
> > return self->session;
> > }
> >
> > --------------------------------------------------------------------------------------------------------
> >
> > /var/log/apache2/error.log
> >
> > The full error log for approx 500000 requests (generated using grep -v
> > "TestSession.__init" error.log).
> >
> > [Tue Jun 07 16:35:32 2005] [notice] mod_python: Creating 32 session
> > mutexes based on 6 max processes and 25 max threads.
> > [Tue Jun 07 16:35:32 2005] [notice] Apache/2.0.54 (Debian GNU/Linux)
> > mod_python/3.2.0-dev-20050519 Python/2.3.5 configured -- resuming normal
> > operations
> > [Tue Jun 07 16:35:49 2005] [notice] child pid 11540 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 16:35:49 2005] [notice] child pid 11545 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 16:36:05 2005] [notice] child pid 11601 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 16:46:21 2005] [notice] child pid 11629 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 16:46:35 2005] [notice] child pid 11949 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:04:07 2005] [notice] child pid 12454 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:11:47 2005] [notice] child pid 12351 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:18:08 2005] [notice] child pid 12342 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:18:20 2005] [notice] child pid 13057 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:34:48 2005] [notice] child pid 12859 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:36:35 2005] [notice] child pid 13599 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:38:10 2005] [notice] child pid 13698 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:39:02 2005] [notice] child pid 13764 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:39:13 2005] [notice] child pid 13819 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:39:15 2005] [notice] child pid 13810 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:40:02 2005] [notice] child pid 13917 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:41:03 2005] [notice] child pid 13048 exit signal
> > Segmentation fault (11)
> > Fatal Python error: GC object already tracked
> > [Tue Jun 07 17:42:41 2005] [notice] child pid 14016 exit signal Aborted (6)
> > Fatal Python error: GC object already tracked
> > [Tue Jun 07 17:45:10 2005] [notice] child pid 14321 exit signal Aborted (6)
> > [Tue Jun 07 17:48:01 2005] [notice] child pid 14439 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:48:49 2005] [notice] child pid 14508 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:50:03 2005] [notice] child pid 14615 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:50:03 2005] [notice] child pid 14672 exit signal
> > Segmentation fault (11)
> > Fatal Python error: GC object already tracked
> > [Tue Jun 07 17:50:53 2005] [notice] child pid 14665 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:50:53 2005] [notice] child pid 14758 exit signal Aborted (6)
> > [Tue Jun 07 17:51:18 2005] [notice] child pid 14800 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:51:18 2005] [notice] child pid 14832 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:51:18 2005] [notice] child pid 14864 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:51:20 2005] [notice] child pid 14893 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:51:21 2005] [notice] child pid 14921 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:51:21 2005] [notice] child pid 14928 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:51:23 2005] [notice] child pid 14977 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:51:24 2005] [notice] child pid 15006 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:52:02 2005] [notice] child pid 15066 exit signal
> > Segmentation fault (11)
> > Fatal Python error: deletion of interned string failed
> > [Tue Jun 07 17:53:06 2005] [notice] child pid 15109 exit signal Aborted (6)
> > Fatal Python error: deletion of interned string failed
> > [Tue Jun 07 17:53:55 2005] [notice] child pid 15210 exit signal Aborted (6)
> > [Tue Jun 07 17:54:03 2005] [notice] child pid 15104 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:54:05 2005] [notice] child pid 15312 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:54:17 2005] [notice] child pid 15405 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:55:04 2005] [notice] child pid 15437 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:55:04 2005] [notice] child pid 15483 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:55:17 2005] [notice] child pid 15563 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:55:54 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: Traceback (most recent call last):
> > [Tue Jun 07 17:55:54 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: File "/usr/lib/python2.3/site-packages/mod_python/apache.py",
> > line 287, in HandlerDispatch\n log=debug)
> > [Tue Jun 07 17:55:54 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: File "/usr/lib/python2.3/site-packages/mod_python/apache.py",
> > line 422, in import_module\n module = sys.modules[module_name]
> > [Tue Jun 07 17:55:54 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: KeyError: 'mptest'
> > [Tue Jun 07 17:55:58 2005] [notice] child pid 15474 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:56:08 2005] [notice] child pid 15600 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:56:08 2005] [notice] child pid 15644 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:57:08 2005] [notice] child pid 15730 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:57:10 2005] [notice] child pid 15778 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:57:44 2005] [notice] child pid 15772 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:58:11 2005] [notice] child pid 15902 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:58:21 2005] [notice] child pid 15945 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: Traceback (most recent call last):
> > [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: File "/usr/lib/python2.3/site-packages/mod_python/apache.py",
> > line 299, in HandlerDispatch\n result = object(req)
> > [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: File "/var/www/mp/mptest.py", line 19, in handler\n sess =
> > req.get_session()
> > [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: File
> > "/usr/lib/python2.3/site-packages/mod_python/TestSession.py", line 46,
> > in create_session\n return TestSession(req,sid)
> > [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: File
> > "/usr/lib/python2.3/site-packages/mod_python/TestSession.py", line 40,
> > in unlock\n if self._lock and self._locked:
> > [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> > mptest: AttributeError: 'TestSession' object has no attribute '_lock'
> > [Tue Jun 07 17:59:38 2005] [notice] child pid 15939 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 18:02:59 2005] [notice] child pid 15865 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 18:02:59 2005] [notice] child pid 16051 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 18:03:02 2005] [notice] child pid 16142 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 18:04:03 2005] [notice] child pid 16170 exit signal
> > Segmentation fault (11)
> > Fatal Python error: GC object already tracked
> > [Tue Jun 07 18:04:16 2005] [notice] child pid 16248 exit signal Aborted (6)
> > [Tue Jun 07 18:04:40 2005] [notice] child pid 16285 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 18:05:15 2005] [notice] child pid 16318 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 18:16:49 2005] [notice] child pid 16573 exit signal
> > Segmentation fault (11)
> > [Tue Jun 07 18:17:50 2005] [notice] child pid 16924 exit signal
> > Segmentation fault (11)
> >
>
>
Re: session handling - the next generation
Posted by Jim Gallacher <jg...@sympatico.ca>.
Replying to my own email,
It looks like there was problem with a circular reference in
req->session->req. This resulted in a memory leak which I had
mis-interpreted as a reference count problem in req_get_session.
As a result, I was not doing a Py_INCREF(self->session) in
req_get_session(). This resulted in the intermittent segfault. (I just
*knew* I was going to screw up the reference counting. :) )
Refactoring TestSession to remove saved reference to the request object
fixed the memory leak problem. Adding Py_INCREF(self->session) in
req_get_session() fixed the segfault problem.
I still don't understand why the memory lead only occurs when the
session object is created in requestobject.c and not when created
directly in a python script, but there you go.
Jim
Jim Gallacher wrote:
> Hi All,
>
> Sorry for the long post, but I think I have a nice contribution to the
> session handling mechanism, but I've got a bug I just can't track down.
> Make yourself a cup of coffee, sit down and have a read.
>
> In an effort to make session handling more transparent and less error
> prone, I've added a new method called get_session to the request object.
>
> (As an aside, I've also created a new apache config directive
> PythonSessionOptions, a req.get_session_options() to read the directive,
> and a small internal change to req.internal_redirect() to pass a session
> to a redirected request).
>
> req.get_session() returns the session instance if it exists or creates a
> new one. Internally it calls some python code,
> mod_python.Session.create_session(), which does the actual work. The
> user will only use get_session() to access their session, and so all our
> deadlocking issues are gone forever.
>
> It mostly works, but I've found that approx 1 in 10000 requests will
> segfault. Calling mod_python.Session.create_session directly does not
> segfault (tested for 2000000 requests) so that is not the issue.
>
> I really hope someone can spot where the problem is. I've reduced
> everything to the simplest possible case. The unlock method in
> TestSession very rarely segfault or raise an exception and then
> segfault, but very rarely. It only fails 0.004% of the time based on
> 500000 requests. I suspect some kind of garbage collection issue, maybe
> a circular reference caused by req->session->_req, but I don't
> understand why it works 99.996 % of the time.
>
> I've included the full error.log for 5000000 requests at the end of this
> message, but the ones that caught my eye as unusual are:
>
> [Tue Jun 07 17:52:02 2005] [notice] child pid 15066 exit signal
> Segmentation fault (11)
> Fatal Python error: deletion of interned string failed
>
> [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: Traceback (most recent call last):
> [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: File "/usr/lib/python2.3/site-packages/mod_python/apache.py",
> line 299, in HandlerDispatch\n result = object(req)
> [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: File "/var/www/mp/mptest.py", line 19, in handler\n sess =
> req.get_session()
> [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: File
> "/usr/lib/python2.3/site-packages/mod_python/TestSession.py", line 46,
> in create_session\n return TestSession(req,sid)
> [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: File
> "/usr/lib/python2.3/site-packages/mod_python/TestSession.py", line 40,
> in unlock\n if self._lock and self._locked:
> [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: AttributeError: 'TestSession' object has no attribute '_lock'
>
> [Tue Jun 07 18:04:03 2005] [notice] child pid 16170 exit signal
> Segmentation fault (11)
> Fatal Python error: GC object already tracked
>
>
> I hope someone can give me some insight to the problem. Note that I'm
> using mpm-worker.
>
> Regards,
> Jim
>
> ------------------------------------------------------------------------
> mptest.py
> # the test handler
>
> from mod_python import apache
> from mod_python import TestSession
>
> def handler(req):
> req.content_type = 'text/plain'
> req.write('mptest.py')
>
> if req.path_info:
> test_case = req.path_info[1:]
> else:
> test_case = '1'
>
> if test_case == '1':
> # the good case - always works
> sess = TestSession.create_session(req,None)
> elif test_case == '2':
> # the bad case - sometimes segfaults
> sess = req.get_session()
> else:
> req.write('no test specified')
>
> return apache.OK
>
> ---------------------------------------------------------------------------------------------------
>
> mod_python/TestSession.py
> # emulate a simple test session
> import _apache
> from Session import _new_sid
>
> class TestSession(object):
>
> def __init__(self, req, sid=0, secret=None, timeout=0, lock=1):
> req.log_error("TestSession.__init__")
### CIRCULAR REFERENCE self._req = req
### causes memory leak
> self._req = req
###
> self._lock = 1
> self._locked = 0
> self._sid = _new_sid(req)
> self.lock()
> self.unlock()
>
> def lock(self):
> if self._lock:
> _apache._global_lock(self._req.server, self._sid)
> self._locked = 1
>
> def unlock(self):
> # unlock will ocassionally segfault
> if self._lock and self._locked:
> _apache._global_unlock(self._req.server, self._sid)
> self._locked = 0
>
> def create_session(req,sid):
> return TestSession(req,sid)
>
> -------------------------------------------------------------------------------------------------
>
> src/requestobject.c
> In the following code session is a PyObject* defined in the the
> requestobject struct.
>
> static PyObject *req_get_session(requestobject *self, PyObject *args)
> {
> PyObject *m;
> PyObject *sid;
>
> if (!self->session) {
> m = PyImport_ImportModule("mod_python.TestSession");
> /* is this redirected request with an existing session? */
> sid = PyObject_CallMethod(self->subprocess_env, "get",
> "(ss)","REDIRECT_PYSID", "");
>
> self->session = PyObject_CallMethod(m, "create_session", "(OO)",
> self, sid);
> Py_DECREF(m);
> Py_DECREF(sid);
> if (self->session == NULL)
> return NULL;
> }
// increase ref count to fix segfault problem
Py_INCREF(self->session);
> return self->session;
> }
>
> --------------------------------------------------------------------------------------------------------
>
> /var/log/apache2/error.log
>
> The full error log for approx 500000 requests (generated using grep -v
> "TestSession.__init" error.log).
>
> [Tue Jun 07 16:35:32 2005] [notice] mod_python: Creating 32 session
> mutexes based on 6 max processes and 25 max threads.
> [Tue Jun 07 16:35:32 2005] [notice] Apache/2.0.54 (Debian GNU/Linux)
> mod_python/3.2.0-dev-20050519 Python/2.3.5 configured -- resuming normal
> operations
> [Tue Jun 07 16:35:49 2005] [notice] child pid 11540 exit signal
> Segmentation fault (11)
> [Tue Jun 07 16:35:49 2005] [notice] child pid 11545 exit signal
> Segmentation fault (11)
> [Tue Jun 07 16:36:05 2005] [notice] child pid 11601 exit signal
> Segmentation fault (11)
> [Tue Jun 07 16:46:21 2005] [notice] child pid 11629 exit signal
> Segmentation fault (11)
> [Tue Jun 07 16:46:35 2005] [notice] child pid 11949 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:04:07 2005] [notice] child pid 12454 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:11:47 2005] [notice] child pid 12351 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:18:08 2005] [notice] child pid 12342 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:18:20 2005] [notice] child pid 13057 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:34:48 2005] [notice] child pid 12859 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:36:35 2005] [notice] child pid 13599 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:38:10 2005] [notice] child pid 13698 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:39:02 2005] [notice] child pid 13764 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:39:13 2005] [notice] child pid 13819 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:39:15 2005] [notice] child pid 13810 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:40:02 2005] [notice] child pid 13917 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:41:03 2005] [notice] child pid 13048 exit signal
> Segmentation fault (11)
> Fatal Python error: GC object already tracked
> [Tue Jun 07 17:42:41 2005] [notice] child pid 14016 exit signal Aborted (6)
> Fatal Python error: GC object already tracked
> [Tue Jun 07 17:45:10 2005] [notice] child pid 14321 exit signal Aborted (6)
> [Tue Jun 07 17:48:01 2005] [notice] child pid 14439 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:48:49 2005] [notice] child pid 14508 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:50:03 2005] [notice] child pid 14615 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:50:03 2005] [notice] child pid 14672 exit signal
> Segmentation fault (11)
> Fatal Python error: GC object already tracked
> [Tue Jun 07 17:50:53 2005] [notice] child pid 14665 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:50:53 2005] [notice] child pid 14758 exit signal Aborted (6)
> [Tue Jun 07 17:51:18 2005] [notice] child pid 14800 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:51:18 2005] [notice] child pid 14832 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:51:18 2005] [notice] child pid 14864 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:51:20 2005] [notice] child pid 14893 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:51:21 2005] [notice] child pid 14921 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:51:21 2005] [notice] child pid 14928 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:51:23 2005] [notice] child pid 14977 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:51:24 2005] [notice] child pid 15006 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:52:02 2005] [notice] child pid 15066 exit signal
> Segmentation fault (11)
> Fatal Python error: deletion of interned string failed
> [Tue Jun 07 17:53:06 2005] [notice] child pid 15109 exit signal Aborted (6)
> Fatal Python error: deletion of interned string failed
> [Tue Jun 07 17:53:55 2005] [notice] child pid 15210 exit signal Aborted (6)
> [Tue Jun 07 17:54:03 2005] [notice] child pid 15104 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:54:05 2005] [notice] child pid 15312 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:54:17 2005] [notice] child pid 15405 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:55:04 2005] [notice] child pid 15437 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:55:04 2005] [notice] child pid 15483 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:55:17 2005] [notice] child pid 15563 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:55:54 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: Traceback (most recent call last):
> [Tue Jun 07 17:55:54 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: File "/usr/lib/python2.3/site-packages/mod_python/apache.py",
> line 287, in HandlerDispatch\n log=debug)
> [Tue Jun 07 17:55:54 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: File "/usr/lib/python2.3/site-packages/mod_python/apache.py",
> line 422, in import_module\n module = sys.modules[module_name]
> [Tue Jun 07 17:55:54 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: KeyError: 'mptest'
> [Tue Jun 07 17:55:58 2005] [notice] child pid 15474 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:56:08 2005] [notice] child pid 15600 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:56:08 2005] [notice] child pid 15644 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:57:08 2005] [notice] child pid 15730 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:57:10 2005] [notice] child pid 15778 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:57:44 2005] [notice] child pid 15772 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:58:11 2005] [notice] child pid 15902 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:58:21 2005] [notice] child pid 15945 exit signal
> Segmentation fault (11)
> [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: Traceback (most recent call last):
> [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: File "/usr/lib/python2.3/site-packages/mod_python/apache.py",
> line 299, in HandlerDispatch\n result = object(req)
> [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: File "/var/www/mp/mptest.py", line 19, in handler\n sess =
> req.get_session()
> [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: File
> "/usr/lib/python2.3/site-packages/mod_python/TestSession.py", line 46,
> in create_session\n return TestSession(req,sid)
> [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: File
> "/usr/lib/python2.3/site-packages/mod_python/TestSession.py", line 40,
> in unlock\n if self._lock and self._locked:
> [Tue Jun 07 17:58:57 2005] [error] [client 192.168.1.12] PythonHandler
> mptest: AttributeError: 'TestSession' object has no attribute '_lock'
> [Tue Jun 07 17:59:38 2005] [notice] child pid 15939 exit signal
> Segmentation fault (11)
> [Tue Jun 07 18:02:59 2005] [notice] child pid 15865 exit signal
> Segmentation fault (11)
> [Tue Jun 07 18:02:59 2005] [notice] child pid 16051 exit signal
> Segmentation fault (11)
> [Tue Jun 07 18:03:02 2005] [notice] child pid 16142 exit signal
> Segmentation fault (11)
> [Tue Jun 07 18:04:03 2005] [notice] child pid 16170 exit signal
> Segmentation fault (11)
> Fatal Python error: GC object already tracked
> [Tue Jun 07 18:04:16 2005] [notice] child pid 16248 exit signal Aborted (6)
> [Tue Jun 07 18:04:40 2005] [notice] child pid 16285 exit signal
> Segmentation fault (11)
> [Tue Jun 07 18:05:15 2005] [notice] child pid 16318 exit signal
> Segmentation fault (11)
> [Tue Jun 07 18:16:49 2005] [notice] child pid 16573 exit signal
> Segmentation fault (11)
> [Tue Jun 07 18:17:50 2005] [notice] child pid 16924 exit signal
> Segmentation fault (11)
>