You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mod_python-dev@quetz.apache.org by Graham Dumpleton <gr...@dscpl.com.au> on 2006/10/27 08:54:43 UTC
Python 2.5 nested auth functions in publisher.
Jim sent this to me when the python-dev list was down for maintenance.
Can anyone with Python 2.5 who knows something about function internals
enlighten us about what may have changed here. Previously the names
of nested functions appeared in func_code.co_names but that doesn't
appear to be the case in Python 2.5.
The code to support this is actually quite dodgy and you can see my
comments about it previously at:
http://issues.apache.org/jira/browse/MODPYTHON-43#action_62042
Personally I think it would be good if this feature of publisher went away,
unfortunately, some amount of code is guaranteed to rely on it.
So anyone got any ideas of how we can do the same thing in Python 2.5?
Thanks.
Graham
Jim Gallacher wrote ..
> Jim Gallacher wrote:
> > Fresh Debian unstable installation seems to have done the trick. My test
> > results:
> >
> > +1 Apache 2.3.3 (mpm-worker), Python 2.3.5
> > +1 Apache 2.3.3 (mpm-worker), Python 2.4.4c1
> > -1 Apache 2.3.3 (mpm-worker), Python 2.5.0
> >
> > With python 2.5 I get 2 failures:
> >
> > test_publisher_auth_nested
> > test_publisher_auth_method_nested
> >
> > which seems odd given that 2.4.4 passes, but I'll dig into it to isolate
> > the problem.
>
> It looks like something has changed in python 2.5 introspection that is
> messing up publisher.
>
> Test script testme.py
> ---------------------
>
> def testfunc():
> print 'stuff'
> def __auth__():
> print '__auth__ called'
> def __access__():
> print '__access__ called'
>
> def main():
> func_obj = testfunc
>
> func_code = func_obj.func_code
> print func_code.co_names
>
> if __name__ == '__main__':
> main()
>
> Results
> -------
>
> $ python2.3 testme.py
> ('__auth__', '__access__')
> $ python2.4 testme.py
> ('__auth__', '__access__')
> $ python2.5 testme.py
> ()
Re: Python 2.5 nested auth functions in publisher.
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
Got access to Python 2.5 finally. My test script works on it so they
have
fixed the ordering issue.
2.5 (r25:51908, Oct 29 2006, 01:52:52)
[GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)]
()
('req', '__auth__', '__access__', '__auth_realm__')
1
__auth__ (1, <code object __auth__ at 0xb7dbdbf0, file "hack.py",
line 7>)
__access__ (1, <code object __access__ at 0xb7dbd7b8, file "hack.py",
line 9>)
__auth_realm__ (1, 'REALM')
Have thus committed some changes back into mod_python for it now,
so should pass tests okay on Python 2.5.
Graham
On 29/10/2006, at 4:18 PM, Graham Dumpleton wrote:
>
> On 29/10/2006, at 12:05 PM, Graham Dumpleton wrote:
>
>>
>> On 28/10/2006, at 10:01 PM, Dan Eloff wrote:
>>
>>> On 10/28/06, Graham Dumpleton <gr...@dscpl.com.au> wrote:
>>>> Dan, the code that needs to be updated is:
>>>>
>>>> if "__auth__" in func_code.co_names:
>>>> i = list(func_code.co_names).index("__auth__")
>>>> __auth__ = func_code.co_consts[i+1]
>>>> if hasattr(__auth__, "co_name"):
>>>> __auth__ = new.function(__auth__, func_globals)
>>>> found_auth = 1
>>>>
>>>> Note how it accesses code objects for functions from co_consts. Do
>>>> they still appear
>>>> to be there in Python 2.5? Are you able to work out some code that
>>>> does the same
>>>> thing as this?
>>>>
>>>
>>> Using the test function:
>>>
>>>>>> def foo(a,b):
>>> d = 5
>>> def __auth__(req):
>>> return True
>>> e = d + 5
>>>
>>>>>> fc = foo.func_code
>>>>>> import new
>>>>>> func_globals = globals()
>>>>>> for i, var_name in enumerate(fc.co_varnames):
>>> if var_name == '__auth__':
>>> __auth__ = fc.co_consts[i-fc.co_argcount+1]
>>> if hasattr(__auth__, 'co_name'):
>>> __auth__ = new.function(__auth__, func_globals)
>>> found_auth = 1
>>> break
>>>
>>>>>> __auth__
>>> <function __auth__ at 0x01159830>
>>>
>>> I am curious as to the hasattr(__auth__, 'co_name') section. Is
>>> there
>>> any case where this is not true? (and does it make sense to say
>>> found_auth = 1 if it isn't?)
>>
>> The co_name check is making sure it is a code object as opposed to a
>> dictionary or some other constant.
>>
>> See:
>>
>> http://www.modpython.org/live/current/doc-html/hand-pub-alg-
>> auth.html
>>
>> for what __auth__ can be.
>
> Actually, I am partly wrong about that, when nesting these inside of a
> function they must be functions or a constant, they can't be a
> dictionary.
> The documentation even states this:
>
> Note that this technique will also work if __auth__ or __access__
> is a
> constant, but will not work if they are a dictionary or a list.
>
> What I have found though is that even in Python 2.3.5, the names
> can be
> found in co_varnames. The problem is that in Python 2.3.5, where the
> names appear in co_varnames, they aren't in the same order as they
> appear in co_names or as required for indexing into co_consts which
> looks like a bug to me.
>
> The question now is whether in Python 2.5 the names appear in
> co_varnames
> in the correct order or not. If they aren't in the correct order
> and it is still
> broken, makes it impossible for it to work.
>
> Can you run the following test program and see if you get what
> would be
> expected.
>
>
>
> def handler(req):
> def __auth__(req, user, password):
> return 1
> def __access__(req, user):
> return 1
> __auth_realm__ = 'REALM'
>
> func_code = handler.func_code
>
> print func_code.co_names
> print func_code.co_varnames
> print func_code.co_argcount
> print
>
> def lookup(name):
> i = None
> if name in func_code.co_names:
> names = func_code.co_names
> i = list(names).index(name)
> elif func_code.co_argcount < len(func_code.co_varnames):
> names = func_code.co_varnames[func_code.co_argcount:]
> if name in names:
> i = list(names).index(name)
> if i is not None:
> return (1, func_code.co_consts[i+1])
> return (0, None)
>
> print '__auth__', lookup('__auth__')
> print '__access__', lookup('__access__')
> print '__auth_realm__', lookup('__auth_realm__')
>
>
>
> On Python 2.3.5 I get:
>
>
>
> ('__auth__', '__access__', '__auth_realm__')
> ('req', '__access__', '__auth_realm__', '__auth__')
> 1
>
> __auth__ (1, <code object __auth__ at 0x6a8a0, file "hack.py", line
> 2>)
> __access__ (1, <code object __access__ at 0x70660, file "hack.py",
> line 4>)
> __auth_realm__ (1, 'REALM')
>
>
> Thanks.
>
> Graham
Re: Python 2.5 nested auth functions in publisher.
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
On 29/10/2006, at 12:05 PM, Graham Dumpleton wrote:
>
> On 28/10/2006, at 10:01 PM, Dan Eloff wrote:
>
>> On 10/28/06, Graham Dumpleton <gr...@dscpl.com.au> wrote:
>>> Dan, the code that needs to be updated is:
>>>
>>> if "__auth__" in func_code.co_names:
>>> i = list(func_code.co_names).index("__auth__")
>>> __auth__ = func_code.co_consts[i+1]
>>> if hasattr(__auth__, "co_name"):
>>> __auth__ = new.function(__auth__, func_globals)
>>> found_auth = 1
>>>
>>> Note how it accesses code objects for functions from co_consts. Do
>>> they still appear
>>> to be there in Python 2.5? Are you able to work out some code that
>>> does the same
>>> thing as this?
>>>
>>
>> Using the test function:
>>
>>>>> def foo(a,b):
>> d = 5
>> def __auth__(req):
>> return True
>> e = d + 5
>>
>>>>> fc = foo.func_code
>>>>> import new
>>>>> func_globals = globals()
>>>>> for i, var_name in enumerate(fc.co_varnames):
>> if var_name == '__auth__':
>> __auth__ = fc.co_consts[i-fc.co_argcount+1]
>> if hasattr(__auth__, 'co_name'):
>> __auth__ = new.function(__auth__, func_globals)
>> found_auth = 1
>> break
>>
>>>>> __auth__
>> <function __auth__ at 0x01159830>
>>
>> I am curious as to the hasattr(__auth__, 'co_name') section. Is there
>> any case where this is not true? (and does it make sense to say
>> found_auth = 1 if it isn't?)
>
> The co_name check is making sure it is a code object as opposed to a
> dictionary or some other constant.
>
> See:
>
> http://www.modpython.org/live/current/doc-html/hand-pub-alg-
> auth.html
>
> for what __auth__ can be.
Actually, I am partly wrong about that, when nesting these inside of a
function they must be functions or a constant, they can't be a
dictionary.
The documentation even states this:
Note that this technique will also work if __auth__ or __access__
is a
constant, but will not work if they are a dictionary or a list.
What I have found though is that even in Python 2.3.5, the names can be
found in co_varnames. The problem is that in Python 2.3.5, where the
names appear in co_varnames, they aren't in the same order as they
appear in co_names or as required for indexing into co_consts which
looks like a bug to me.
The question now is whether in Python 2.5 the names appear in
co_varnames
in the correct order or not. If they aren't in the correct order and
it is still
broken, makes it impossible for it to work.
Can you run the following test program and see if you get what would be
expected.
def handler(req):
def __auth__(req, user, password):
return 1
def __access__(req, user):
return 1
__auth_realm__ = 'REALM'
func_code = handler.func_code
print func_code.co_names
print func_code.co_varnames
print func_code.co_argcount
print
def lookup(name):
i = None
if name in func_code.co_names:
names = func_code.co_names
i = list(names).index(name)
elif func_code.co_argcount < len(func_code.co_varnames):
names = func_code.co_varnames[func_code.co_argcount:]
if name in names:
i = list(names).index(name)
if i is not None:
return (1, func_code.co_consts[i+1])
return (0, None)
print '__auth__', lookup('__auth__')
print '__access__', lookup('__access__')
print '__auth_realm__', lookup('__auth_realm__')
On Python 2.3.5 I get:
('__auth__', '__access__', '__auth_realm__')
('req', '__access__', '__auth_realm__', '__auth__')
1
__auth__ (1, <code object __auth__ at 0x6a8a0, file "hack.py", line 2>)
__access__ (1, <code object __access__ at 0x70660, file "hack.py",
line 4>)
__auth_realm__ (1, 'REALM')
Thanks.
Graham
Re: Python 2.5 nested auth functions in publisher.
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
On 28/10/2006, at 10:01 PM, Dan Eloff wrote:
> On 10/28/06, Graham Dumpleton <gr...@dscpl.com.au> wrote:
>> Dan, the code that needs to be updated is:
>>
>> if "__auth__" in func_code.co_names:
>> i = list(func_code.co_names).index("__auth__")
>> __auth__ = func_code.co_consts[i+1]
>> if hasattr(__auth__, "co_name"):
>> __auth__ = new.function(__auth__, func_globals)
>> found_auth = 1
>>
>> Note how it accesses code objects for functions from co_consts. Do
>> they still appear
>> to be there in Python 2.5? Are you able to work out some code that
>> does the same
>> thing as this?
>>
>
> Using the test function:
>
>>>> def foo(a,b):
> d = 5
> def __auth__(req):
> return True
> e = d + 5
>
>>>> fc = foo.func_code
>>>> import new
>>>> func_globals = globals()
>>>> for i, var_name in enumerate(fc.co_varnames):
> if var_name == '__auth__':
> __auth__ = fc.co_consts[i-fc.co_argcount+1]
> if hasattr(__auth__, 'co_name'):
> __auth__ = new.function(__auth__, func_globals)
> found_auth = 1
> break
>
>>>> __auth__
> <function __auth__ at 0x01159830>
>
> I am curious as to the hasattr(__auth__, 'co_name') section. Is there
> any case where this is not true? (and does it make sense to say
> found_auth = 1 if it isn't?)
The co_name check is making sure it is a code object as opposed to a
dictionary or some other constant.
See:
http://www.modpython.org/live/current/doc-html/hand-pub-alg-auth.html
for what __auth__ can be.
Graham
Re: Python 2.5 nested auth functions in publisher.
Posted by Dan Eloff <da...@gmail.com>.
On 10/28/06, Graham Dumpleton <gr...@dscpl.com.au> wrote:
> Dan, the code that needs to be updated is:
>
> if "__auth__" in func_code.co_names:
> i = list(func_code.co_names).index("__auth__")
> __auth__ = func_code.co_consts[i+1]
> if hasattr(__auth__, "co_name"):
> __auth__ = new.function(__auth__, func_globals)
> found_auth = 1
>
> Note how it accesses code objects for functions from co_consts. Do
> they still appear
> to be there in Python 2.5? Are you able to work out some code that
> does the same
> thing as this?
>
Using the test function:
>>> def foo(a,b):
d = 5
def __auth__(req):
return True
e = d + 5
>>> fc = foo.func_code
>>> import new
>>> func_globals = globals()
>>> for i, var_name in enumerate(fc.co_varnames):
if var_name == '__auth__':
__auth__ = fc.co_consts[i-fc.co_argcount+1]
if hasattr(__auth__, 'co_name'):
__auth__ = new.function(__auth__, func_globals)
found_auth = 1
break
>>> __auth__
<function __auth__ at 0x01159830>
I am curious as to the hasattr(__auth__, 'co_name') section. Is there
any case where this is not true? (and does it make sense to say
found_auth = 1 if it isn't?)
-Dan
Re: Python 2.5 nested auth functions in publisher.
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
On 27/10/2006, at 11:03 PM, Dan Eloff wrote:
> On 10/27/06, Graham Dumpleton <gr...@dscpl.com.au> wrote:
>> Unless they have really screwed things around, co_varnames is
>> specifically
>> for function argument names and is unlikely to contained nested
>> constant
>> names. If it did, then I would expect a lot of the publisher code to
>> break in
>> other ways as it uses co_varnames for the very specific purpose of
>> matching
>> form parameters against function arguments.
>
> I'd look into that code then.
>
>>>> fc.co_names
> ()
>>>> fc.co_varnames
> ('__auth__', '__access__')
>
>>>> def foo(a,b):
> d = 5
> def bar(c):
> return c
>
>>>> fc.co_names
> ()
>>>> fc.co_varnames
> ('a', 'b', 'd', 'bar')
>
> To get just args, try:
>
>>>> fc.co_varnames[:fc.co_argcount]
> ('a', 'b')
>
> And for just local vars:
>
>>>> fc.co_varnames[fc.co_argcount:]
> ('d', 'bar')
Dan, the code that needs to be updated is:
if "__auth__" in func_code.co_names:
i = list(func_code.co_names).index("__auth__")
__auth__ = func_code.co_consts[i+1]
if hasattr(__auth__, "co_name"):
__auth__ = new.function(__auth__, func_globals)
found_auth = 1
Note how it accesses code objects for functions from co_consts. Do
they still appear
to be there in Python 2.5? Are you able to work out some code that
does the same
thing as this?
Graham
Re: Python 2.5 nested auth functions in publisher.
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
Thanks, confirms my thinking when I was trying to sleep. That is,
that the
only way they could do it if using co_varnames was to rely on
co_argcount
to denote which are the function arguments and which aren't.
Strange, wander why they made this change?
Anyway, will log a JIRA issue and see if I can come up with alternate
code to cope with this. I wander if I am safe in assuming that < 2.5,
that
len(co_varnames) is always the same as co_argcount.
Graham
On 27/10/2006, at 11:03 PM, Dan Eloff wrote:
> On 10/27/06, Graham Dumpleton <gr...@dscpl.com.au> wrote:
>> Unless they have really screwed things around, co_varnames is
>> specifically
>> for function argument names and is unlikely to contained nested
>> constant
>> names. If it did, then I would expect a lot of the publisher code to
>> break in
>> other ways as it uses co_varnames for the very specific purpose of
>> matching
>> form parameters against function arguments.
>
> I'd look into that code then.
>
>>>> fc.co_names
> ()
>>>> fc.co_varnames
> ('__auth__', '__access__')
>
>>>> def foo(a,b):
> d = 5
> def bar(c):
> return c
>
>>>> fc.co_names
> ()
>>>> fc.co_varnames
> ('a', 'b', 'd', 'bar')
>
> To get just args, try:
>
>>>> fc.co_varnames[:fc.co_argcount]
> ('a', 'b')
>
> And for just local vars:
>
>>>> fc.co_varnames[fc.co_argcount:]
> ('d', 'bar')
>
> -Dan
Re: Python 2.5 nested auth functions in publisher.
Posted by Dan Eloff <da...@gmail.com>.
On 10/27/06, Graham Dumpleton <gr...@dscpl.com.au> wrote:
> Unless they have really screwed things around, co_varnames is
> specifically
> for function argument names and is unlikely to contained nested constant
> names. If it did, then I would expect a lot of the publisher code to
> break in
> other ways as it uses co_varnames for the very specific purpose of
> matching
> form parameters against function arguments.
I'd look into that code then.
>>> fc.co_names
()
>>> fc.co_varnames
('__auth__', '__access__')
>>> def foo(a,b):
d = 5
def bar(c):
return c
>>> fc.co_names
()
>>> fc.co_varnames
('a', 'b', 'd', 'bar')
To get just args, try:
>>> fc.co_varnames[:fc.co_argcount]
('a', 'b')
And for just local vars:
>>> fc.co_varnames[fc.co_argcount:]
('d', 'bar')
-Dan
Re: Python 2.5 nested auth functions in publisher.
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
On 27/10/2006, at 9:31 PM, Dan Eloff wrote:
> On 10/27/06, Graham Dumpleton <gr...@dscpl.com.au> wrote:
>> Jim sent this to me when the python-dev list was down for
>> maintenance.
>> Can anyone with Python 2.5 who knows something about function
>> internals
>> enlighten us about what may have changed here. Previously the names
>> of nested functions appeared in func_code.co_names but that doesn't
>> appear to be the case in Python 2.5.
>
> Confirmed. Try func_code.co_varnames instead. I don't know why the
> change, and a quick search wasn't very enlightening, other than they
> python guys mentioned something about co_names and co_varnames
> changed, and some people fixing this problem by using co_varnames in
> python 2.5.
Unless they have really screwed things around, co_varnames is
specifically
for function argument names and is unlikely to contained nested constant
names. If it did, then I would expect a lot of the publisher code to
break in
other ways as it uses co_varnames for the very specific purpose of
matching
form parameters against function arguments.
I guess time will tell.
Anyway, time for me to sleep. :-)
Graham
Re: Python 2.5 nested auth functions in publisher.
Posted by Dan Eloff <da...@gmail.com>.
On 10/27/06, Graham Dumpleton <gr...@dscpl.com.au> wrote:
> Jim sent this to me when the python-dev list was down for maintenance.
> Can anyone with Python 2.5 who knows something about function internals
> enlighten us about what may have changed here. Previously the names
> of nested functions appeared in func_code.co_names but that doesn't
> appear to be the case in Python 2.5.
Confirmed. Try func_code.co_varnames instead. I don't know why the
change, and a quick search wasn't very enlightening, other than they
python guys mentioned something about co_names and co_varnames
changed, and some people fixing this problem by using co_varnames in
python 2.5.
-Dan