You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mod_python-dev@quetz.apache.org by Sébastien Arnaud <ar...@emedialibrary.org> on 2006/09/07 06:59:03 UTC
Regex based publisher proposal
Hi,
I have been following with passion mod_python development for quite a
while now, and in the light of a few emails over the past few months
discussing web frameworks in mod_python, I decided I would attempt to
contribute to the project in order to move towards a fast, flexible
MVC mod_python only based web framework.
I have written 2 or 3 different ones along the past couple of years,
but nothing worthy of sharing by any mean. They have helped me
however to define what would be the "dream" web framework for
mod_python, but more importantly to identify the needed plumbing
improvements to mod_python.
One of the first needed improvements, in my opinion, is the capacity
to route web requests in a more flexible manner than via the current
publisher module. So, I would like to propose the following module
(pubre.py). It is basically a copy of the mod_python.publisher module
to the exception that a lot of the core handler code has been
modified to use regex in order to route a web request to the
appropriate module/function.
I have been developing against mod_python/trunk and I attached the
file for whoever wants to review it and give it a try. Keep in mind
though it is still probably rough around the edges and not any solid
testing has been performed yet. I only performed some trivial
benchmarking/stress testing to make sure that performance wise it was
on par with the current mod_python.publisher.
The default behavior is suppose to be 100% compatible with the way
mod_python.publisher behaves. Eventually though you would be able to
pass as a PythonOption the grammar of the urls in your web
application, by simply declaring something like:
<Directory "/mypath/mydir/">
AddHandler mod_python .py .html
PythonHandler mod_python.pubre
PythonOption "pubregex" "(?P<controller>[\w]+)?(\.(?P<extension>[\w]
+))?(/(?P<action>[^/]+))?(\?$)?"
</Directory>
I know that not all grammars will work with the current version
attached (due to some code being still dependent on the conservative
url structure /path/file.ext), eventually though, I hope I can get
this solved and allow any regex grammar to work.
Anyway, please share your comments and feedback to make sure I am
headed in the right direction by keeping in mind that my first goal
is to be able to publish using a defined regex url grammar a callable
class within a module. I believe that once this first step is
accomplished the real design of the web framework can begin.
Cheers!
S�bastien
Re: Regex based publisher proposal
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
On 07/09/2006, at 2:59 PM, Sébastien Arnaud wrote:
> Hi,
>
> I have been following with passion mod_python development for quite
> a while now, and in the light of a few emails over the past few
> months discussing web frameworks in mod_python, I decided I would
> attempt to contribute to the project in order to move towards a
> fast, flexible MVC mod_python only based web framework.
>
> I have written 2 or 3 different ones along the past couple of
> years, but nothing worthy of sharing by any mean. They have helped
> me however to define what would be the "dream" web framework for
> mod_python, but more importantly to identify the needed plumbing
> improvements to mod_python.
>
> One of the first needed improvements, in my opinion, is the
> capacity to route web requests in a more flexible manner than via
> the current publisher module. So, I would like to propose the
> following module (pubre.py). It is basically a copy of the
> mod_python.publisher module to the exception that a lot of the core
> handler code has been modified to use regex in order to route a web
> request to the appropriate module/function.
>
> I have been developing against mod_python/trunk and I attached the
> file for whoever wants to review it and give it a try. Keep in mind
> though it is still probably rough around the edges and not any
> solid testing has been performed yet. I only performed some trivial
> benchmarking/stress testing to make sure that performance wise it
> was on par with the current mod_python.publisher.
>
> The default behavior is suppose to be 100% compatible with the way
> mod_python.publisher behaves. Eventually though you would be able
> to pass as a PythonOption the grammar of the urls in your web
> application, by simply declaring something like:
>
> <Directory "/mypath/mydir/">
> AddHandler mod_python .py .html
> PythonHandler mod_python.pubre
> PythonOption "pubregex" "(?P<controller>[\w]+)?(\.(?P<extension>
> [\w]+))?(/(?P<action>[^/]+))?(\?$)?"
> </Directory>
>
> I know that not all grammars will work with the current version
> attached (due to some code being still dependent on the
> conservative url structure /path/file.ext), eventually though, I
> hope I can get this solved and allow any regex grammar to work.
>
> Anyway, please share your comments and feedback to make sure I am
> headed in the right direction by keeping in mind that my first goal
> is to be able to publish using a defined regex url grammar a
> callable class within a module. I believe that once this first step
> is accomplished the real design of the web framework can begin.
Can you provide an English description with examples of how your
extension is
used in practice so we don't have to reverse engineer it to work out
what it does?
Thanks.
Graham
Re: Regex based publisher proposal
Posted by Nicolas Lehuen <ni...@lehuen.com>.
Note that Routes can be integrated to mod_python, and provides a very useful
feature (that can be find in Rails as well) : URL generation, which is the
inverse of dispatching. Given parameter names, the URL generator build a
full URL that can be inserted into an HTML page, for example.
I'm dreaming almost daily of a similar feature for the publisher (the best
would simply to have a modpython.publisher.generate_url(req,any_callable)
method that would automagically build the full or relative url to the
callable given in parameter (given a few requirements on the callable) and
based on the current request (you remember that discussion we had about
knowing what was the full URL that was used to reach any given handler ?
that was the idea behind my question).
Regards,
Nicolas
2006/9/7, Nicolas Lehuen <ni...@lehuen.com>:
>
> What Sébastien proposes is the kind of dispatching mechanism used in
> Django, and (without regular expressions) in Rails and Routes (
> http://routes.groovie.org/). This tends to prove that some people find it
> useful.
>
> The proposed implementation, however, is another story. There are the
> problems that Graham noticed, plus the fact that the current publishing
> system is richer than the simple controller + action couple (you can buidl
> hierarchies of objects, have callable object anywhere in the hierarchy,
> etc.).
>
> Also, the publisher supports skipping the extension part of the module, so
> you CAN have /controller/action URIs that already behave quite like what
> Sébastien suggests.
>
> Regards,
> Nicolas
>
> 2006/9/7, Jorey Bump <li...@joreybump.com>:
>
> > Sébastien Arnaud wrote:
> >
> > > <Directory "/mypath/mydir/">
> > > AddHandler mod_python .py .html
> > > PythonHandler mod_python.pubre
> > > PythonOption "pubregex"
> > >
> > "(?P<controller>[\w]+)?(\.(?P<extension>[\w]+))?(/(?P<action>[^/]+))?(\?$)?"
> >
> > >
> > > </Directory>
> > >
> > > I know that not all grammars will work with the current version
> > attached
> > > (due to some code being still dependent on the conservative url
> > > structure /path/file.ext), eventually though, I hope I can get this
> > > solved and allow any regex grammar to work.
> > >
> > > Anyway, please share your comments and feedback to make sure I am
> > headed
> > > in the right direction by keeping in mind that my first goal is to be
> > > able to publish using a defined regex url grammar a callable class
> > > within a module. I believe that once this first step is accomplished
> > the
> > > real design of the web framework can begin.
> >
> > Is regex essential to what you're trying to accomplish? I do something
> > similar in a minimal handler I've written that simply makes a list of
> > the URL parts:
> >
> > resource = req.path_info.strip('/').split('/')
> >
> > Then I walk the list in order to determine the action that needs to be
> > taken. A very primitive example would be:
> >
> > if resource[0] == 'time':
> > # use first item only, returns formatted local date & time
> > # i.e. http://host/path/to/handler/time
> > import time
> > now = time.strftime('%Y-%m-%d %H:%M:%S')
> > req.content_type = "text/plain"
> > req.write(now)
> > elif resource[0] == 'color':
> > # use second URL part as a key in a data structure
> > # i.e. http://host/path/to/handler/color/Lance
> > data = {'Lance': 'blue',
> > 'Robin': 'idunno',
> > 'Galahad': 'yellow'}
> > req.content_type = "text/plain"
> > try:
> > req.write("%s's favorite color is %s." % (resource[1],
> > data[resource[1]]))
> > except KeyError:
> > req.write("%s's favorite color is unknown." % (resource[1],))
> > except IndexError:
> > req.write('Specify a name (/color/Someone).')
> >
> > Then http://host/handler/time returns:
> >
> > 2006-09-07 11:17:19
> >
> > and http://host/handler/color/Lance returns:
> >
> > Lance's favorite color is blue.
> >
> > My motivation was to support friendly dynamic URLs
> > (http://host/handler/foo/bar/baz ) in the simplest way possible. Since
> > the action taken is handled in logic (instead of a strict mapping to
> > either functions, modules, or objects), it's extremely flexible. The
> > trick is in determining if an argument should be part of the abs_path or
> >
> > handled as a query (http://host/handler/foo?bar=baz).
> >
> > Of course, there's nothing stopping you from using the remainder of the
> > list as values for your app:
> >
> > if resource[0] == 'order':
> > # use remaining resources as multiple values for single argument
> > # http://host/path/to/handler/order/item/item/item
> > if len(resource) > 2:
> > # there was a list of items
> > order = '%s and %s' % (', '.join(resource[1:-1]), resource[-1])
> > elif len(resource) == 2:
> > # there was a single item
> > order = resource[1]
> > else:
> > # no items in URL
> > order = 'nothing'
> > req.content_type = "text/plain"
> > req.write('You have ordered %s.' % (order))
> >
> > Then this URL:
> >
> > http://host/handler/order/spam/spam/spam/eggs/spam
> >
> > would return:
> >
> > You have ordered spam, spam, spam, eggs and spam.
> >
> > This isn't always appropriate, and one needs to be careful (too much
> > rope, as they say). It's normally better to handle arbitrarily long
> > lists of values with query arguments or POST variables.
> >
> > Anyway, I don't mean to hijack your thread, Sébastien. I'm just
> > wondering if you are trying to accomplish the same thing, or have a
> > specific reason for using regex, which really can slow an application
> > down. You could easily adapt this simpler approach to enforce a stricter
> > mapping of your own convention.
> >
> >
> >
> >
> >
>
Re: Regex based publisher proposal
Posted by Nicolas Lehuen <ni...@lehuen.com>.
What Sébastien proposes is the kind of dispatching mechanism used in Django,
and (without regular expressions) in Rails and Routes (
http://routes.groovie.org/). This tends to prove that some people find it
useful.
The proposed implementation, however, is another story. There are the
problems that Graham noticed, plus the fact that the current publishing
system is richer than the simple controller + action couple (you can buidl
hierarchies of objects, have callable object anywhere in the hierarchy,
etc.).
Also, the publisher supports skipping the extension part of the module, so
you CAN have /controller/action URIs that already behave quite like what
Sébastien suggests.
Regards,
Nicolas
2006/9/7, Jorey Bump <li...@joreybump.com>:
>
> Sébastien Arnaud wrote:
>
> > <Directory "/mypath/mydir/">
> > AddHandler mod_python .py .html
> > PythonHandler mod_python.pubre
> > PythonOption "pubregex"
> >
> "(?P<controller>[\w]+)?(\.(?P<extension>[\w]+))?(/(?P<action>[^/]+))?(\?$)?"
> >
> > </Directory>
> >
> > I know that not all grammars will work with the current version attached
> > (due to some code being still dependent on the conservative url
> > structure /path/file.ext), eventually though, I hope I can get this
> > solved and allow any regex grammar to work.
> >
> > Anyway, please share your comments and feedback to make sure I am headed
> > in the right direction by keeping in mind that my first goal is to be
> > able to publish using a defined regex url grammar a callable class
> > within a module. I believe that once this first step is accomplished the
> > real design of the web framework can begin.
>
> Is regex essential to what you're trying to accomplish? I do something
> similar in a minimal handler I've written that simply makes a list of
> the URL parts:
>
> resource = req.path_info.strip('/').split('/')
>
> Then I walk the list in order to determine the action that needs to be
> taken. A very primitive example would be:
>
> if resource[0] == 'time':
> # use first item only, returns formatted local date & time
> # i.e. http://host/path/to/handler/time
> import time
> now = time.strftime('%Y-%m-%d %H:%M:%S')
> req.content_type = "text/plain"
> req.write(now)
> elif resource[0] == 'color':
> # use second URL part as a key in a data structure
> # i.e. http://host/path/to/handler/color/Lance
> data = {'Lance': 'blue',
> 'Robin': 'idunno',
> 'Galahad': 'yellow'}
> req.content_type = "text/plain"
> try:
> req.write("%s's favorite color is %s." % (resource[1],
> data[resource[1]]))
> except KeyError:
> req.write("%s's favorite color is unknown." % (resource[1],))
> except IndexError:
> req.write('Specify a name (/color/Someone).')
>
> Then http://host/handler/time returns:
>
> 2006-09-07 11:17:19
>
> and http://host/handler/color/Lance returns:
>
> Lance's favorite color is blue.
>
> My motivation was to support friendly dynamic URLs
> (http://host/handler/foo/bar/baz) in the simplest way possible. Since
> the action taken is handled in logic (instead of a strict mapping to
> either functions, modules, or objects), it's extremely flexible. The
> trick is in determining if an argument should be part of the abs_path or
> handled as a query (http://host/handler/foo?bar=baz).
>
> Of course, there's nothing stopping you from using the remainder of the
> list as values for your app:
>
> if resource[0] == 'order':
> # use remaining resources as multiple values for single argument
> # http://host/path/to/handler/order/item/item/item
> if len(resource) > 2:
> # there was a list of items
> order = '%s and %s' % (', '.join(resource[1:-1]), resource[-1])
> elif len(resource) == 2:
> # there was a single item
> order = resource[1]
> else:
> # no items in URL
> order = 'nothing'
> req.content_type = "text/plain"
> req.write('You have ordered %s.' % (order))
>
> Then this URL:
>
> http://host/handler/order/spam/spam/spam/eggs/spam
>
> would return:
>
> You have ordered spam, spam, spam, eggs and spam.
>
> This isn't always appropriate, and one needs to be careful (too much
> rope, as they say). It's normally better to handle arbitrarily long
> lists of values with query arguments or POST variables.
>
> Anyway, I don't mean to hijack your thread, Sébastien. I'm just
> wondering if you are trying to accomplish the same thing, or have a
> specific reason for using regex, which really can slow an application
> down. You could easily adapt this simpler approach to enforce a stricter
> mapping of your own convention.
>
>
>
>
>
Re: Regex based publisher proposal
Posted by Jorey Bump <li...@joreybump.com>.
Sébastien Arnaud wrote:
> <Directory "/mypath/mydir/">
> AddHandler mod_python .py .html
> PythonHandler mod_python.pubre
> PythonOption "pubregex"
> "(?P<controller>[\w]+)?(\.(?P<extension>[\w]+))?(/(?P<action>[^/]+))?(\?$)?"
>
> </Directory>
>
> I know that not all grammars will work with the current version attached
> (due to some code being still dependent on the conservative url
> structure /path/file.ext), eventually though, I hope I can get this
> solved and allow any regex grammar to work.
>
> Anyway, please share your comments and feedback to make sure I am headed
> in the right direction by keeping in mind that my first goal is to be
> able to publish using a defined regex url grammar a callable class
> within a module. I believe that once this first step is accomplished the
> real design of the web framework can begin.
Is regex essential to what you're trying to accomplish? I do something
similar in a minimal handler I've written that simply makes a list of
the URL parts:
resource = req.path_info.strip('/').split('/')
Then I walk the list in order to determine the action that needs to be
taken. A very primitive example would be:
if resource[0] == 'time':
# use first item only, returns formatted local date & time
# i.e. http://host/path/to/handler/time
import time
now = time.strftime('%Y-%m-%d %H:%M:%S')
req.content_type = "text/plain"
req.write(now)
elif resource[0] == 'color':
# use second URL part as a key in a data structure
# i.e. http://host/path/to/handler/color/Lance
data = {'Lance': 'blue',
'Robin': 'idunno',
'Galahad': 'yellow'}
req.content_type = "text/plain"
try:
req.write("%s's favorite color is %s." % (resource[1],
data[resource[1]]))
except KeyError:
req.write("%s's favorite color is unknown." % (resource[1],))
except IndexError:
req.write('Specify a name (/color/Someone).')
Then http://host/handler/time returns:
2006-09-07 11:17:19
and http://host/handler/color/Lance returns:
Lance's favorite color is blue.
My motivation was to support friendly dynamic URLs
(http://host/handler/foo/bar/baz) in the simplest way possible. Since
the action taken is handled in logic (instead of a strict mapping to
either functions, modules, or objects), it's extremely flexible. The
trick is in determining if an argument should be part of the abs_path or
handled as a query (http://host/handler/foo?bar=baz).
Of course, there's nothing stopping you from using the remainder of the
list as values for your app:
if resource[0] == 'order':
# use remaining resources as multiple values for single argument
# http://host/path/to/handler/order/item/item/item
if len(resource) > 2:
# there was a list of items
order = '%s and %s' % (', '.join(resource[1:-1]), resource[-1])
elif len(resource) == 2:
# there was a single item
order = resource[1]
else:
# no items in URL
order = 'nothing'
req.content_type = "text/plain"
req.write('You have ordered %s.' % (order))
Then this URL:
http://host/handler/order/spam/spam/spam/eggs/spam
would return:
You have ordered spam, spam, spam, eggs and spam.
This isn't always appropriate, and one needs to be careful (too much
rope, as they say). It's normally better to handle arbitrarily long
lists of values with query arguments or POST variables.
Anyway, I don't mean to hijack your thread, Sébastien. I'm just
wondering if you are trying to accomplish the same thing, or have a
specific reason for using regex, which really can slow an application
down. You could easily adapt this simpler approach to enforce a stricter
mapping of your own convention.
Re: Regex based publisher proposal
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
On 08/09/2006, at 5:15 PM, Graham Dumpleton wrote:
> Now I am well and truly confused and not sure if I have broken
> mod_python 3.3
> or not. Short of it is, that I cannot get something like:
>
> <Location /location>
> AddHandler mod_python .xxx
> PythonHandler /Users/grahamd/Projects/testing/xxx.py
> </Location>
FWIW, you can only use AddHandler with Location directives if you also
set ModMimeUsePathInfo directive.
<Location /location>
AddHandler mod_python .xxx
PythonHandler /Users/grahamd/Projects/testing/xxx.py
ModMimeUsePathInfo On
</Location>
This way the mod_mime module which processes AddHandler directives,
will take into consideration the req.path_info value which is where the
extension will lie in this case.
The only caveat to this is that the extension must be at the end of
the URL
and cannot be embedded within the path. Ie., a URL of:
/location/cat/dog.xxx
will match and handler called, but not if it is:
/location/cat.xxx/dog
I'll add notes about this too that new article of mine when I get a
chance.
Graham
Re: Regex based publisher proposal
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
On 08/09/2006, at 3:42 PM, Graham Dumpleton wrote:
>> In regards to:
>>
>> path,module_name = os.path.split(req.filename)
>>
>> # Trimming the front part of req.uri
>> if module_name=='':
>> req_url = ''
>> else:
>> req_url = req.uri[req.uri.index(module_name):]
>>
>> It is bringing a question I have had for a long time in
>> mod_python. Is there a way to identify accurately the relative
>> virtual path of a request other than comparing to the physical
>> path, especially when using apache definition such as VirtualHost
>> & Location? For example let's say you define something like:
>> <Location /myapp/admin/>
>> AddHandler mod_python .py .html
>> PythonHandler mod_python.publisher
>> </Location>
>>
>> And there comes a request such as http://myserver/myapp/admin/
>> login.html
>>
>> How do you determine accurately that the part of the path /myapp/
>> admin/ is in fact the virtual root of your application ?
>
> If a Directory directive is used, or the handler directives appear
> in a .htaccess file,
> the you can use req.hlist.directory.
>
> Caveats to this are that prior to mod_python 3.3, if the handler
> directive appears
> inside of a Files/FileMatch directive within those two contexts,
> the value was wrong
> and could not be used. Further, prior to mod_python 3.3, the result
> would be wrong
> if use wildcards in the Directory directive, or the DirectoryMatch
> variant was used.
>
> When the Location directive is used, there is no physical directory
> which corresponds
> to it and so req.hlist.directory wouldn't be valid as its purpose
> is to denote the physical
> directory a directive was associated with. A caveat to this though,
> is that prior to
> mod_python 3.3, req.hlist.directory would actually list the path
> specified in the
> Location directive. This isn't the case though in mod_python 3.3
> and it will instead
> be None when used in the Location or LocationMatch directives as it
> causes problems
> otherwise with a non existent path being used as a location to
> search for Python
> modules.
>
> With my brain being clouded at the moment, I cant remember off the
> top of my head
> what the correct way is and I would have to check things, but when
> you use the
> Location directive, isn't req.path_info the part of the URL which
> lies below where the
> Location directive was specified for. If my memory is correct, this
> doesn't mean you
> can simply take the length of req.path_info and drop off that much
> off req.uri though.
> The problem is that req.uri isn't normalised in the way that
> req.path_info is, so it may
> contain repeating slashes and other crud that can stuff things up.
> You need to address
Now I am well and truly confused and not sure if I have broken
mod_python 3.3
or not. Short of it is, that I cannot get something like:
<Location /location>
AddHandler mod_python .xxx
PythonHandler /Users/grahamd/Projects/testing/xxx.py
</Location>
to work. If I use SetHandler instead of AddHandler inside of the
Location directive,
ie.:
<Location /location>
SetHandler mod_python
PythonHandler /Users/grahamd/Projects/testing/xxx.py
</Location>
then it works as I expected, with handler containing:
from mod_python import apache
def handler(req):
req.content_type = 'text/plain'
req.write('req.filename = %s\n' % req.filename)
req.write('req.uri = %s\n' % req.uri)
req.write('req.path_info = %s\n' % req.path_info)
return apache.OK
producing:
req.filename = /usr/local/apache-2.2/htdocs/location
req.uri = /location/cat/dog.xxx
req.path_info = /cat/dog.xxx
Because Location directive is used, then req.filename is sort of
meaningless, although
one could do:
req.write('req.document_root() = %s\n' % req.document_root())
req.write('location = %s\n' % req.filename[len
(req.document_root()):])
and get:
req.document_root() = /usr/local/apache-2.2/htdocs
location = /location
Now the question is whether AddHandler should work within a Location
directive. Since
it doesn't, my first guess is that it can't because no matching
against actual files is being
performed and so no extension for a target could be derived.
I note that using an extension qualifier to the PythonHandler
directive doesn't work
either. Ie., following doesn't call the handler if request is /
dog.xxx or /dog/cat.xxx.
<Location /location>
SetHandler mod_python
PythonHandler /Users/grahamd/Projects/testing/xxx.py | .xxx
</Location>
All probably quite logical and the only way to do something different
based on extension
in the URL would be to specify a fixup handler and not a response
handler, and for
the fixup handler to know it is being used in a location directive,
map the URL itself
somehow and then set req.handler to 'mod_python' and call
req.add_handler() to give
the handler to be called.
Ie., Apache configuration of:
<Location /location>
PythonFixupHandler /Users/grahamd/Projects/testing/xxx.py
</Location>
and handler file containing:
from mod_python import apache
import posixpath
def handler(req):
req.content_type = 'text/plain'
req.write('req.filename = %s\n' % req.filename)
req.write('req.uri = %s\n' % req.uri)
req.write('req.path_info = %s\n' % req.path_info)
req.write('req.document_root() = %s\n' % req.document_root())
req.write('location = %s\n' % req.filename[len
(req.document_root()):])
return apache.OK
# This doesn't copy with concept of path info after extension.
def fixuphandler(req):
extension = posixpath.splitext(req.path_info)[1]
if extension == '.xxx':
req.handler = 'mod_python'
req.add_handler('PythonHandler', handler)
return apache.OK
return apache.DECLINED
BTW, using an absolute path for a handler is new in mod_python 3.3.
Assigning to
req.handler also only works from 3.3 onwards, so don't expect this to
work on older
versions of mod_python.
Having said all that, when you said:
>> <Location /myapp/admin/>
>> AddHandler mod_python .py .html
>> PythonHandler mod_python.publisher
>> </Location>
is that what you are using and I should actually expect AddHandler to
work inside
of a Location directive?
Graham
Re: Regex based publisher proposal
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
On 10/09/2006, at 12:28 PM, Sébastien Arnaud wrote:
> I love the wiki you put together, lots of great stuff in there! I
> am going to go back to the drawing board here due to all the new
> things you presented available in mod_python 3.3 which I did not
> know about. It looks like a fixuphandler would fit the bill for
> what I am attempting to do. Once I get something working, I will
> make sure to give more detailed info and a clear example of how the
> framework/url matching works so that you can better advise on which
> mod_python 3.3 features can be better taken advantage of.
>
> In regards to mod_python.publisher, it looks like indeed it is
> better to divorce myself from it after the points you brought up.
> On that topic though, should I reuse at all the ModuleCache object
> to load & cache modules? Or is there something more efficient/
> better in mod_python 3.3?
Ignore the ModuleCache class which appears to be used by
mod_python.publisher
as when the new importer gets used in mod_python 3.3 it doesn't
actually get used.
In mod_python 3.3, I would suggest you simply assume/require that the
new module
importer is used and use apache.import_module() instead. The new
module importer
is the default in 3.3, but it still patches itself in in place of the
old module importers
and so just reading the code everything still looks to use the old.
The most important things you will want to know about
apache.import_module() in
mod_python 3.3 are:
1. The first argument 'module' can actually now also be an absolute
path to a module
code file, including '.py' extension. By specifying the actual
location, you skip any
searching through directories trying to find an appropriate match.
2. You don't need to ever supply 'log' or 'autoreload' arguments as
the function is now
able to always get them directly from Apache configuration
information appropriate to
the request.
3. Because of (1), when using handler directives in Apache
configuration, you can
now supply an absolute path for a handler module file as well. Where
module file is
in non standard location, this avoids need to setup PythonPath. The
need to set up
PythonPath explicitly should be much reduced and using it couldn't
indicate a wrong
setup and you might even find mod_python complaining about it.
You can check out some of the behaviour of the new module importer by
reading
the old mailing list thread at:
http://www.modpython.org/pipermail/mod_python/2006-April/020936.html
http://www.modpython.org/pipermail/mod_python/2006-May/021097.html
http://www.mail-archive.com/python-dev@httpd.apache.org/msg02185.html
There are lots of details though and not sure that these capture it
all. I still need to
turn this all into a wiki page as well and use some of it for
official mod_python docs. :-)
Graham
Re: Regex based publisher proposal
Posted by Sébastien Arnaud <ar...@emedialibrary.org>.
Hi Graham,
Thank you for all the detailed info! Hope you are feeling better. I
did not mean to get you all worked up with a foggy brain ;)
I love the wiki you put together, lots of great stuff in there! I am
going to go back to the drawing board here due to all the new things
you presented available in mod_python 3.3 which I did not know about.
It looks like a fixuphandler would fit the bill for what I am
attempting to do. Once I get something working, I will make sure to
give more detailed info and a clear example of how the framework/url
matching works so that you can better advise on which mod_python 3.3
features can be better taken advantage of.
In regards to mod_python.publisher, it looks like indeed it is better
to divorce myself from it after the points you brought up. On that
topic though, should I reuse at all the ModuleCache object to load &
cache modules? Or is there something more efficient/better in
mod_python 3.3?
Finally, in relation with the PythonAddHandler within a Location
directive, actually I did not have it working as is - I was just
trying to cover all bases on how to determine best the virtual path
root. Thanks for digging this throughly. Let me try to digest all the
info you gave me and infuse it back into my code. I'll post back on
the list when I have something else worthy to look at.
Thanks!
Sébastien
Re: Regex based publisher proposal
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
On 08/09/2006, at 2:53 PM, Sébastien Arnaud wrote:
> In short, Nicolas did read my mind correctly in regards to what I
> am attempting to do here. I have searched and searched like many
> python developers for a proper web framework and I have settled for
> mod_python about 2 years ago. I tried afterwards the new "popular"
> ones such as Ruby on Rails, TurboGears, Django, but not really
> liked those because mostly with their integration with SQL Objects
> type DB abstraction, which I never bought into, and their
> dependence on specific templating systems. Nevertheless, they did
> introduce some interesting concepts (such as regex url mapping),
> which I am attempting here (poorly it seems) to implement directly
> in mod_python.
I'd still appreciate an English description with examples as I
haven't delved too
much into some of these other packages. Ie., what does the URL look
like for
a concrete example, and how does that relate to what files Apache may
match
in the file system and subsequently what may be interpreted within
the handler
from the path info.
I know I could probably sit down and work it out, but my excuse is I
am not well,
even had to take a couple days off work, so not thinking too
straight. ;-)
> To put things back into context, the code I submitted was a simple
> draft from a passion impulse over the labor day week-end and was
> never (in my mind) a solid implementation but more something to
> illustrate what could maybe done and open a discussion.
My feedback was intended to be constructive, not critical. :-)
> In regards to:
>
> path,module_name = os.path.split(req.filename)
>
> # Trimming the front part of req.uri
> if module_name=='':
> req_url = ''
> else:
> req_url = req.uri[req.uri.index(module_name):]
>
> It is bringing a question I have had for a long time in mod_python.
> Is there a way to identify accurately the relative virtual path of
> a request other than comparing to the physical path, especially
> when using apache definition such as VirtualHost & Location? For
> example let's say you define something like:
> <Location /myapp/admin/>
> AddHandler mod_python .py .html
> PythonHandler mod_python.publisher
> </Location>
>
> And there comes a request such as http://myserver/myapp/admin/
> login.html
>
> How do you determine accurately that the part of the path /myapp/
> admin/ is in fact the virtual root of your application ?
If a Directory directive is used, or the handler directives appear in
a .htaccess file,
the you can use req.hlist.directory.
Caveats to this are that prior to mod_python 3.3, if the handler
directive appears
inside of a Files/FileMatch directive within those two contexts, the
value was wrong
and could not be used. Further, prior to mod_python 3.3, the result
would be wrong
if use wildcards in the Directory directive, or the DirectoryMatch
variant was used.
When the Location directive is used, there is no physical directory
which corresponds
to it and so req.hlist.directory wouldn't be valid as its purpose is
to denote the physical
directory a directive was associated with. A caveat to this though,
is that prior to
mod_python 3.3, req.hlist.directory would actually list the path
specified in the
Location directive. This isn't the case though in mod_python 3.3 and
it will instead
be None when used in the Location or LocationMatch directives as it
causes problems
otherwise with a non existent path being used as a location to search
for Python
modules.
With my brain being clouded at the moment, I cant remember off the
top of my head
what the correct way is and I would have to check things, but when
you use the
Location directive, isn't req.path_info the part of the URL which
lies below where the
Location directive was specified for. If my memory is correct, this
doesn't mean you
can simply take the length of req.path_info and drop off that much
off req.uri though.
The problem is that req.uri isn't normalised in the way that
req.path_info is, so it may
contain repeating slashes and other crud that can stuff things up.
You need to address
that before you can use it.
> Also, a little bit trickier when you get something like: http://
> myserver/myapp/admin/display/asc/xxx
>
> 1) Is the only way to pass along all requests (not filtered by
> extensions) to your mod_python handler is to use something like:
> SetHandler python-program
> PythonHandler myhandler
In mod_python 3.3, you will be able to specify a fixup handler which
did:
def fixuphandler(req):
req.handler = 'mod_python'
req.add_handler('PythonHandler', 'myhandler')
return apache.OK
The bit that doesn't work before mod_python 3.3 is the assignment to
the req.handler
attribute, with it being read only in older versions.
In mod_python 3.3, when using req.add_handler(), the handler need not
even be a
string, but could be an actual callable object.
I still haven't quite finished the article, nor have I announced the
new site yet, although
was close to doing so, but you might want to read:
https://www.dscpl.com.au/wiki/ModPython/Articles/
SetHandlerVersusAddHandler
This document hasn't been thoroughly checked or reviewed yet, so
feedback and/or
corrections are most appreciated.
Some of the stuff in mod_python 3.3 should make it much easier to
create more
flexible systems. This is in part why I wanted you to describe in
English what you were
trying to achieve rather how you thought you needed to do it. That
way I can more
easily relate it to how it could be done in mod_python 3.3 where new
features may be
able to be used.
> 2) In this situation determining the virtual root of your
> application is crucial to be able to separate the regular path from
> the potential parameters/actions so how would you go about it? I
> have in the past used the cheap trick of passing along in a
> PythonOption the virtual path of my app, but there must be a better
> way. I have still failed though to find in the mod_python docs a
> way to read the properties of an apache <location> or <virtual
> host> directive in order to determine automatically the proper
> virtual root of my handler
Provided things are done inside a Directory/DirectoryMatch
or .htaccess file, then the
req.hlist.directory attribute can be used. As to the Location/
LocationMatch case, I'll
try and figure that out later.
> Finally, all I am really trying to build here is a drop-in
> replacement for mod_python.publisher by providing by default the
> same functionality, but to offer the flexibility found in newer web
> frameworks via regex mapping and eventually implement a reverse url
> mapping method, just like Nicolas described it.
I would really suggest you not base in on mod_python.publisher code
as it is actually
flawed in some important ways that make working out relative URLs to
resources
a problem. See:
http://www.modpython.org/pipermail/mod_python/2006-March/020501.html
for a description of some of the problems, but in short,
mod_python.publisher
doesn't deal with trailing slash redirection in the way it probably
should. The result
is that the same resource can be accessed with different URLs at
different
levels in the URL namespace. Depending on through which URL it is
accessed
a relative URL to something in the same directory has to be different.
Anyway, time for me to go out and get some fresh air. Then I better
try and finish up
some stuff on my new site so can make it officially public.
Graham
Re: Regex based publisher proposal
Posted by Sébastien Arnaud <ar...@emedialibrary.org>.
Hi Dan,
I use the Queue class to perform DB pooling, and use 3 different
types of templating system in various mod_python applications. So
yes, mod_python makes all those customization very easy and it is why
I love it as well. The only downside I have found is that after
developing several applications, maintaining all those custom
handlers can be daunting, and I just wished that some of this basic
additional "plumbing" was avail to some extent in mod_python.
There are also some clear disadvantages of having all mod_python user
to write their "own" handler, it is not always that trivial to write
a robust basic framework that is secure, and thread safe. It also
seems like an awful waste of time and duplication of efforts, even if
the learning experience opportunity is great.
Finally, in my current work environment where I advocated the use of
mod_python, I have found myself in a position where bringing in new
fresh developers can be a little bit painful, when you have to
basically start supporting your own custom framework, write
documentation on its behavior, etc... Anyway, that's partly where my
motivation came from and hope that others will support the idea.
Thanks,
Sébastien
P.S. I will try WingIDE, I believe I tried it out a while back, but
under OS X the interface via X11 did not feel quite right and was
pretty slow if I remember correctly. I'll give it another chance
though ;)
On Sep 8, 2006, at 9:38 AM, Dan Eloff wrote:
>> * DB connection pooling
>
> Got that. It's actually very simple. Just derive from the Queue class,
> it's already synchronized. Override pop with the logic for handing out
> connections. Simply use put to return it when you're done. I recommend
> putting the db information in a config file like rails does, nobody
> wants to have to put that all over the place.
>
>> * Generic rendering template system (empowering the developer to use
>> virtually ANY templating system avail in python)
>
> That's the way mod_python is as far as I know, it pretty much lets you
> do your own thing for templates. Since everyone seems to have
> different ideas about what templates they like, I think that's great.
>
>> * Remote DB Sessions (for scalability)
>
> Got that for mysql, but is based on a different session model than
> mod_python uses. Someone else posted a mysql session here recently, I
> do not know what became of it, I would recommend you follow up before
> it goes cold.
>
>> * Advanced Debugging
>
> Get wingide, I've never seen a better debugger for python. It handles
> remote debugging on mod_python flawlessly. When PythonDebug is on, I
> automaticaly connect to the debugger from python. I tried Komodo first
> for the same, but it is worse than useless. If you want help getting
> that working let me know.
>
> -Dan
Re: Regex based publisher proposal
Posted by Jorey Bump <li...@joreybump.com>.
Dan Eloff wrote:
>> * DB connection pooling
>
> Got that. It's actually very simple. Just derive from the Queue class,
> it's already synchronized. Override pop with the logic for handing out
> connections. Simply use put to return it when you're done. I recommend
> putting the db information in a config file like rails does, nobody
> wants to have to put that all over the place.
Do you have any example code to share? I just moved an application to a
shiny new server with new (stable) versions of everything, and my
persistent connections to MySQL stopped working.
Re: Regex based publisher proposal
Posted by Dan Eloff <da...@gmail.com>.
> * DB connection pooling
Got that. It's actually very simple. Just derive from the Queue class,
it's already synchronized. Override pop with the logic for handing out
connections. Simply use put to return it when you're done. I recommend
putting the db information in a config file like rails does, nobody
wants to have to put that all over the place.
> * Generic rendering template system (empowering the developer to use
> virtually ANY templating system avail in python)
That's the way mod_python is as far as I know, it pretty much lets you
do your own thing for templates. Since everyone seems to have
different ideas about what templates they like, I think that's great.
> * Remote DB Sessions (for scalability)
Got that for mysql, but is based on a different session model than
mod_python uses. Someone else posted a mysql session here recently, I
do not know what became of it, I would recommend you follow up before
it goes cold.
> * Advanced Debugging
Get wingide, I've never seen a better debugger for python. It handles
remote debugging on mod_python flawlessly. When PythonDebug is on, I
automaticaly connect to the debugger from python. I tried Komodo first
for the same, but it is worse than useless. If you want help getting
that working let me know.
-Dan
Re: Regex based publisher proposal
Posted by Sébastien Arnaud <ar...@emedialibrary.org>.
Thank you all for taking the time to read my code, I apologize for
not including a better description of what I intended to do, but I
was reluctant to post a very very long message to the list to start
with, so I am going to blame my poor editing skills and the late hour
I sent this email at ;)
In short, Nicolas did read my mind correctly in regards to what I am
attempting to do here. I have searched and searched like many python
developers for a proper web framework and I have settled for
mod_python about 2 years ago. I tried afterwards the new "popular"
ones such as Ruby on Rails, TurboGears, Django, but not really liked
those because mostly with their integration with SQL Objects type DB
abstraction, which I never bought into, and their dependence on
specific templating systems. Nevertheless, they did introduce some
interesting concepts (such as regex url mapping), which I am
attempting here (poorly it seems) to implement directly in mod_python.
To put things back into context, the code I submitted was a simple
draft from a passion impulse over the labor day week-end and was
never (in my mind) a solid implementation but more something to
illustrate what could maybe done and open a discussion.
To review quickly the code comments though, the mapper_cache object
is not thread safe yet, and I added only later the ability to define
your regex as a PythonOption argument and definetly forgot to
consider what could happen when two different parts of the URL
namespace use different regex's and don't execute in different Python
interpreter instances.
In regards to:
path,module_name = os.path.split(req.filename)
# Trimming the front part of req.uri
if module_name=='':
req_url = ''
else:
req_url = req.uri[req.uri.index(module_name):]
It is bringing a question I have had for a long time in mod_python.
Is there a way to identify accurately the relative virtual path of a
request other than comparing to the physical path, especially when
using apache definition such as VirtualHost & Location? For example
let's say you define something like:
<Location /myapp/admin/>
AddHandler mod_python .py .html
PythonHandler mod_python.publisher
</Location>
And there comes a request such as http://myserver/myapp/admin/login.html
How do you determine accurately that the part of the path /myapp/
admin/ is in fact the virtual root of your application ?
Also, a little bit trickier when you get something like: http://
myserver/myapp/admin/display/asc/xxx
1) Is the only way to pass along all requests (not filtered by
extensions) to your mod_python handler is to use something like:
SetHandler python-program
PythonHandler myhandler
2) In this situation determining the virtual root of your application
is crucial to be able to separate the regular path from the potential
parameters/actions so how would you go about it? I have in the past
used the cheap trick of passing along in a PythonOption the virtual
path of my app, but there must be a better way. I have still failed
though to find in the mod_python docs a way to read the properties of
an apache <location> or <virtual host> directive in order to
determine automatically the proper virtual root of my handler.
Graham, I chose to re-embedded the code of mod_python publisher,
mainly because I figured I would have to modify some other parts, if
I want to not only publish a function, but in fact publish a callable
class (which would derive from a future mywebframework.action class).
Correct me if I am wrong, but right now it is not possible to publish
a callable class in a module with mod_python publisher.
Jorey, I dig your implementation without using regex, but here I am
really trying to use regex. I also thought that performance would be
an issue, but on some of my early benchmarks I did not get any worse
performance than the current mod_python.publisher method. So, I think
that if implemented properly performance might not even be drawback
of using regex.
Finally, all I am really trying to build here is a drop-in
replacement for mod_python.publisher by providing by default the same
functionality, but to offer the flexibility found in newer web
frameworks via regex mapping and eventually implement a reverse url
mapping method, just like Nicolas described it.
I have looked closely at routes, but I did not want all the features
they implemented and did not like the idea of adding an external
dependency just for the routing/mapping process.
I strongly believe in a simple mod_python web framework that would
arm developers with essential tools such as:
* regex based routing with a reverve mapping method
* web object oriented (a request maps to a class instanced for the call)
* regex based parameter validation (to insure the good form of data
passed along )
* DB connection pooling
* Generic rendering template system (empowering the developer to use
virtually ANY templating system avail in python)
* Remote DB Sessions (for scalability)
* Advanced Debugging
* Close integration of apache features exposed by mod_python
Regards,
Sébastien
On Sep 7, 2006, at 1:17 AM, Graham Dumpleton wrote:
>
> On 07/09/2006, at 2:59 PM, Sébastien Arnaud wrote:
>> Anyway, please share your comments and feedback to make sure I am
>> headed in the right direction by keeping in mind that my first
>> goal is to be able to publish using a defined regex url grammar a
>> callable class within a module. I believe that once this first
>> step is accomplished the real design of the web framework can begin.
>
> A few comments while I work out what your code actually does.
>
> class Mapper:
> """ This is the object to cache the regex engine """
> regex = "(?P<controller>[\w]+)?(\.(?P<extension>[\w]+))?(/(?
> P<action>[^/]+))?(\?$)?"
> regex_compared = 0
>
> def __init__(self):
> self.reobj = re.compile(self.regex)
>
> def __call__(self, uri, cre):
> if(cre!=None and not self.regex_compared and cre!=self.regex):
> self.regex = cre
> self.reobj = re.compile(self.regex)
> self.regex_compared = 1
> m = self.reobj.match(uri)
> if m:
> return (m.group('controller'), m.group('extension'), m.group
> ('action'))
> else:
> return (None, None, None)
>
> mapper_cache = Mapper()
>
> This is not thread safe and use in a multithreaded MPM, ie., winnt
> and worker, may
> result in failure. I also suspect if you would have problems where
> two different parts
> of the URL namespace use different regex's and they aren't
> executing in different
> Python interpreter instances.
>
> path,module_name = os.path.split(req.filename)
>
> # Trimming the front part of req.uri
> if module_name=='':
> req_url = ''
> else:
> req_url = req.uri[req.uri.index(module_name):]
>
> This is not a very robust way of doing this can technically could
> fail in certain cases.
>
> # Now determine the actual Python module code file
> # to load. This will first try looking for the file
> # '/path/<module_name>.py'.
> req.filename = path + '/' + controller + '.py'
> if not exists(req.filename):
> raise apache.SERVER_RETURN, apache.HTTP_NOT_FOUND
>
> I am not really sure why you go to all this trouble. For the way
> the default regex
> is written, this could possibly just as easily be achieved using
> standard mod_python.publisher,
> using subdirectories in document tree and use of MultiViews
> matching in an appropriate
> way.
>
> In other words, am not convinced that your code is required at all
> and you may be
> able to achieve the same thing as default regex using standard
> mod_python.publisher.
> At worst case, you might need to use Apache RewriteRule. In
> mod_python 3.3, you
> could probably do all this with a very simple fixup handler as well.
>
> Thus, as already requested, can you actually supply some examples
> of how this is
> used in practice.
>
> BTW, you could also have done this by using a wrapper handler
> around the
> existing mod_python.publisher handler as well, thereby avoiding
> having to cut and
> paste all the code.
>
> Graham
Re: Regex based publisher proposal
Posted by Graham Dumpleton <gr...@dscpl.com.au>.
On 07/09/2006, at 2:59 PM, Sébastien Arnaud wrote:
> Anyway, please share your comments and feedback to make sure I am
> headed in the right direction by keeping in mind that my first goal
> is to be able to publish using a defined regex url grammar a
> callable class within a module. I believe that once this first step
> is accomplished the real design of the web framework can begin.
A few comments while I work out what your code actually does.
class Mapper:
""" This is the object to cache the regex engine """
regex = "(?P<controller>[\w]+)?(\.(?P<extension>[\w]+))?(/(?P<action>
[^/]+))?(\?$)?"
regex_compared = 0
def __init__(self):
self.reobj = re.compile(self.regex)
def __call__(self, uri, cre):
if(cre!=None and not self.regex_compared and cre!=self.regex):
self.regex = cre
self.reobj = re.compile(self.regex)
self.regex_compared = 1
m = self.reobj.match(uri)
if m:
return (m.group('controller'), m.group('extension'), m.group
('action'))
else:
return (None, None, None)
mapper_cache = Mapper()
This is not thread safe and use in a multithreaded MPM, ie., winnt
and worker, may
result in failure. I also suspect if you would have problems where
two different parts
of the URL namespace use different regex's and they aren't executing
in different
Python interpreter instances.
path,module_name = os.path.split(req.filename)
# Trimming the front part of req.uri
if module_name=='':
req_url = ''
else:
req_url = req.uri[req.uri.index(module_name):]
This is not a very robust way of doing this can technically could
fail in certain cases.
# Now determine the actual Python module code file
# to load. This will first try looking for the file
# '/path/<module_name>.py'.
req.filename = path + '/' + controller + '.py'
if not exists(req.filename):
raise apache.SERVER_RETURN, apache.HTTP_NOT_FOUND
I am not really sure why you go to all this trouble. For the way the
default regex
is written, this could possibly just as easily be achieved using
standard mod_python.publisher,
using subdirectories in document tree and use of MultiViews matching
in an appropriate
way.
In other words, am not convinced that your code is required at all
and you may be
able to achieve the same thing as default regex using standard
mod_python.publisher.
At worst case, you might need to use Apache RewriteRule. In
mod_python 3.3, you
could probably do all this with a very simple fixup handler as well.
Thus, as already requested, can you actually supply some examples of
how this is
used in practice.
BTW, you could also have done this by using a wrapper handler around the
existing mod_python.publisher handler as well, thereby avoiding
having to cut and
paste all the code.
Graham