You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by David James <ja...@gmail.com> on 2005/07/20 02:07:13 UTC
[PROPOSAL] Safe Memory Management for the Python/SWIG bindings
Currently, if you make a memory management mistake in your Python (or
Perl or Java or Ruby) code that uses the Subversion SWIG bindings,
your program will crash with a segfault. Segfaults are difficult to
debug and can also be a source of security holes.
Here's a two-pronged proposal that aims to stop SWIG segfaults:
1. Track reference counts to each pool.
- When a variable is allocated from a pool, increment the pool reference count.
- When a variable is destroyed, decrement the pool reference count.
2. Validate pool memory (as a runtime configurable debugging option,
enabled by default for safety purposes)
- Before allocating memory (inside the SWIG/C bindings)
- Before accessing memory (using a __getattribute__ handler)
- Before writing to memory (using a __setattr__ handler)
As of r15370, we already verify that pools are correct before
allocating memory from them (in the python-bindings-improvements
branch). So we're already part-way there. I've described the
implementation for the rest of the safety features below.
IMPLEMENTATION
Here's a quick and dirty prototype for a "CheckedObject" class which
maintains a reference to a pool and validates that the pool is correct
before allowing it to be accessed. This object accomplishes tasks 1
and 2. When variables inside the CheckedObject class are accessed or
modified, we check the validity of the CObject before allowing access
to the underlying variable. We could modify the SWIG proxy wrappers
for CObjects to contain these functions.
class CheckedObject(object):
def __getattribute__(self, name):
if name != "assert_valid":
self.assert_valid()
return object.__getattribute__(self, name)
def __setattr__(self, name, value):
self.assert_valid()
return object.__setattr__(self, name, value)
def assert_valid(self):
object.__getattribute__(self,"pool").assert_valid()
We could write a script to modify the generated Python versions of the
SWIG wrappers to contain the above functions. Furthermore, whenever
swig objects are created, we would call object.__setattr__(obj,
"pool", pool), to add a reference to the pool that created the SWIG
object.
In order to detect when pools are cleared in pure C code, we will need
to add an APR cleanup function for each pool which is managed by C
code. This cleanup function would notify the Python bindings that the
C code has deleted the pool and would therefore mark the Python
version of the pool as invalid. If we did this, then users who attempt
to access a python variable after its memory has been deallocated,
would get a Pythonic error message instead of a segfault. (It would
solve issue #2172)
Whenever a checked object is passed into a SWIG/C function, the SWIG/C
function would call the "assert_valid" function to verify that the
object is valid.
CHALLENGES
In order for the above implementation to work, we will need to have a
Python proxy object for every C object that is accessible from Python.
Currently, SWIG only creates Python proxy objects if we provide it
with the details of the internals of the C struct.
Currently, we do not provide SWIG with all the details of how the
internals of Subversion are implemented, so SWIG treats some
Subversion datastructures as opaque pointers. We can fix this problem
by using the %include directive -- just %include the internal .h or .c
files which describe the structure. If we do this, then SWIG will
generate real Python objects for our "opaque pointers" -- not just C
Objects.
Subversion developers: What do you think of this idea?
Cheers,
David
--
David James -- http://www.cs.toronto.edu/~james
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Safe Memory Management for the Python/SWIG bindings
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
David James wrote:
> On 7/19/05, David James <ja...@gmail.com> wrote:
>
>>CHALLENGES
>>
>>In order for the above implementation to work, we will need to have a
>>Python proxy object for every C object that is accessible from Python.
>>Currently, SWIG only creates Python proxy objects if we provide it
>>with the details of the internals of the C struct.
>>
>>Currently, we do not provide SWIG with all the details of how the
>>internals of Subversion are implemented, so SWIG treats some
>>Subversion datastructures as opaque pointers.
>
> OK, I've identified the problem. So far, so good.
>
>
>>We can fix this problem by using the %include directive -- just
>>%include the internal .h or .c files which describe the structure.
>>If we do this, then SWIG will generate real Python objects for
>>our "opaque pointers" -- not just C Objects.
>
> I've looked into this issue further. There is an easier way to ask
> SWIG to setup proxy objects: just setup a dummy declaration, and tell
> SWIG which functions to call in order to create and destroy objects.
> Kouhei has demonstrated this technique inside core.i:
>
> /* Dummy declaration */
> struct apr_pool_t
> {
> };
>
> /* Leave memory administration to ruby's GC */
> %extend apr_pool_t
> {
> apr_pool_t(apr_pool_t *parent=NULL, apr_allocator_t *allocator=NULL) {
> /* Disable parent pool. */
> return svn_pool_create_ex(NULL, NULL);
> }
>
> ~apr_pool_t() {
> apr_pool_destroy(self);
> }
> };
>
>
> In order for my proposal to work, we will need to use Kouhei's
> technique to create Pythonic proxies for all of the SVN datatypes. If
> we define a single proxy for each object inside the SWIG interface
> file, SWIG will generate code in all three languages (Python, Perl,
> Ruby) automatically.
That seems reasonable to me, although as usual in this area, I have only
the bare minimum experience with SWIG and our language bindings.
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Safe Memory Management for the Python/SWIG bindings
Posted by David James <ja...@gmail.com>.
On 7/22/05, Mark Benedetto King <mb...@lowlatency.com> wrote:
> I know nothing of SWIG, but am a little concerned by this sort
> of GC interaction. What happens when a pool goes out-of-scope
> from Python's point of view, but references to it remain in
> datastructures in other pools?
>
> Phrased differently, how does python know whether the pool is
> still reachable?
Currently? It doesn't. It's your responsibility to keep the pool alive
by keeping it in scope. If you don't do this, Python will segfault.
I'd like to improve this situation, by keeping track of all of the
objects that were created from a pool. If we do this, the pool will
stay in scope as long as its child Python objects are still in scope.
After the pool and all of the Python-visible child objects have gone
out of scope, Python will delete the pool. Does this seem safe?
Python will only delete pools that were originally created in Python
-- we don't try to manage the C library subpools. In some cases (e.g.
callback functions), the Subversion library hands the Python
interpreter values which were created by an internal subpool. When the
internal subpool is deleted, your variables will be invalid -- but
Python won't know this. If you try to access a variable after its pool
has been deleted, Python will segfault.
I don't like segfaults. I'd like to see Python raise an exception instead.
Cheers,
David
--
David James -- http://www.cs.toronto.edu/~james
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Safe Memory Management for the Python/SWIG bindings
Posted by Mark Benedetto King <mb...@lowlatency.com>.
On Wed, Jul 20, 2005 at 04:23:40PM -0400, David James wrote:
> On 7/19/05, David James <ja...@gmail.com> wrote:
> > CHALLENGES
> >
> > In order for the above implementation to work, we will need to have a
> > Python proxy object for every C object that is accessible from Python.
> > Currently, SWIG only creates Python proxy objects if we provide it
> > with the details of the internals of the C struct.
> >
> > Currently, we do not provide SWIG with all the details of how the
> > internals of Subversion are implemented, so SWIG treats some
> > Subversion datastructures as opaque pointers.
> OK, I've identified the problem. So far, so good.
>
> > We can fix this problem by using the %include directive -- just
> > %include the internal .h or .c files which describe the structure.
> > If we do this, then SWIG will generate real Python objects for
> > our "opaque pointers" -- not just C Objects.
> I've looked into this issue further. There is an easier way to ask
> SWIG to setup proxy objects: just setup a dummy declaration, and tell
> SWIG which functions to call in order to create and destroy objects.
> Kouhei has demonstrated this technique inside core.i:
>
> /* Dummy declaration */
> struct apr_pool_t
> {
> };
>
> /* Leave memory administration to ruby's GC */
> %extend apr_pool_t
> {
> apr_pool_t(apr_pool_t *parent=NULL, apr_allocator_t *allocator=NULL) {
> /* Disable parent pool. */
> return svn_pool_create_ex(NULL, NULL);
> }
>
> ~apr_pool_t() {
> apr_pool_destroy(self);
> }
> };
>
>
> In order for my proposal to work, we will need to use Kouhei's
> technique to create Pythonic proxies for all of the SVN datatypes. If
> we define a single proxy for each object inside the SWIG interface
> file, SWIG will generate code in all three languages (Python, Perl,
> Ruby) automatically.
>
I know nothing of SWIG, but am a little concerned by this sort
of GC interaction. What happens when a pool goes out-of-scope
from Python's point of view, but references to it remain in
datastructures in other pools?
Phrased differently, how does python know whether the pool is
still reachable?
--ben
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Safe Memory Management for the Python/SWIG bindings
Posted by David James <ja...@gmail.com>.
On 7/19/05, David James <ja...@gmail.com> wrote:
> CHALLENGES
>
> In order for the above implementation to work, we will need to have a
> Python proxy object for every C object that is accessible from Python.
> Currently, SWIG only creates Python proxy objects if we provide it
> with the details of the internals of the C struct.
>
> Currently, we do not provide SWIG with all the details of how the
> internals of Subversion are implemented, so SWIG treats some
> Subversion datastructures as opaque pointers.
OK, I've identified the problem. So far, so good.
> We can fix this problem by using the %include directive -- just
> %include the internal .h or .c files which describe the structure.
> If we do this, then SWIG will generate real Python objects for
> our "opaque pointers" -- not just C Objects.
I've looked into this issue further. There is an easier way to ask
SWIG to setup proxy objects: just setup a dummy declaration, and tell
SWIG which functions to call in order to create and destroy objects.
Kouhei has demonstrated this technique inside core.i:
/* Dummy declaration */
struct apr_pool_t
{
};
/* Leave memory administration to ruby's GC */
%extend apr_pool_t
{
apr_pool_t(apr_pool_t *parent=NULL, apr_allocator_t *allocator=NULL) {
/* Disable parent pool. */
return svn_pool_create_ex(NULL, NULL);
}
~apr_pool_t() {
apr_pool_destroy(self);
}
};
In order for my proposal to work, we will need to use Kouhei's
technique to create Pythonic proxies for all of the SVN datatypes. If
we define a single proxy for each object inside the SWIG interface
file, SWIG will generate code in all three languages (Python, Perl,
Ruby) automatically.
Cheers,
David
--
David James -- http://www.cs.toronto.edu/~james
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org