You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mod_python-dev@quetz.apache.org by "Gregory (Grisha) Trubetskoy" <gr...@apache.org> on 2005/08/18 15:38:24 UTC
Re: flex [was mod_python 3.2.0-BETA available for testing]
OK, here is the flex scoop - as the the docs point out, anything before
2.5.31 is not reentrant and I think even uses a slightly different
interface so older flex won't even process the psp_parser.l file
correctly.
Looking at Fedora Core 4, it still has flex 2.5.4a. (Note that 2.5.31 >
2.5.4 because 31 > 4 - I for a while had trouble seeing that for some
reason), so the new flex is still not commonplace.
So until reentrant flex becomes commonplace, the solution was to include
a pre-parsed psp_parser.c so that you woudln't need flex at all to compile
mod_python. Looks like this still should be the case.
The ./configure should just print a warning that if flex is not found or
too old, should you need to rebuild psp_parser.c you will need to get the
right version of flex.
Grisha
On Thu, 18 Aug 2005, Jim Gallacher wrote:
> Graham Dumpleton wrote:
>> A few comments:
>>
>> 1. If you have an older version of flex than that expected, it gives
>> message:
>>
>> checking flex version... configure: WARNING: Flex version 2.5.31 or
>> greater is required. The one you have seems to be 2.5.4. Use --with-flex
>> to specify another.
>>
>> There is nothing in the README describing why such a requirement exists.
>> The only veiled comment is in the "configure" script itself where it
>> says:
>>
>> # check for correct flex version
>> # requires flex 2.5.31 for reentrant support
>>
>> A more prominent explanation is required in the README with "configure"
>> outputing with the message above a pointer to consult the README for
>> further details.
>>
>> What also isn't at all clear, is whether the flex warning is relevant
>> anyway if pre-generated code for the lexical analyser is provided in the
>> tar ball and that is used.
>
> This is explained in the html docs, but didn't make it into the README. I
> wasn't sure if the WARNING was the right idea,
>
>
>
> See doc-html/inst-configure.html:
>
> """ Attempts to locate flex and determine its version. If flex cannot be
> found in your PATH configure will fail. If the wrong version is found
> configure will generate a warning. You can generally ignore this warning
> unless you need to re-create src/psp_parser.c.
>
> The parser used by psp (See 4.9) is written in C generated using flex. This
> requires a reentrant version of flex which at this time is 2.5.31. Most
> platforms however ship with version 2.5.4 which is not suitable, so a
> pre-generated copy of psp_parser.c is included with the source. If you do
> need to compile src/psp_parser.c you must get the correct flex version.
>
> If the first flex binary in the path is not suitable or not the one desired
> you can specify an alternative location with the --with-flex option, e.g:
>
> $ ./configure --with-flex=/usr/local/bin/flex
>
> """
>
> I'll copy this into the README as well.
>
> In 3.1.x the path to flex was hard coded to /usr/local/bin, which caused
> compilation to fail and required the user to edit the Makefile manually.
> Other than the detection of the flex path, the behaviour is the same as
> 3.1.x. The previous warning about needing the correct flex version was just a
> comment in the Makefile. Any suggestions on how to make this clearer are
> welcome.
>
>> Thus, what is the real story, am I going to have a problem or not if
>> I have an older version of flex?
>
> No, see html docs snippet above.
>
> No time to look at the rest. Will comment tonight.
>
> Jim
>
>>
>> 2. Publisher still doesn't check special extensions supplied by the
>> AddHandler or PythonHandler directives. Ie.,
>>
>> http://issues.apache.org/jira/browse/MODPYTHON-72
>>
>> This may mean that code that previously worked no longer works. Ie., this
>> feature was in there before. Only caveat is that the feature may not have
>> worked properly before anyway, so existing code may not be affected after
>> all. See:
>>
>> http://issues.apache.org/jira/browse/MODPYTHON-22
>>
>> Bit unsure on this one.
>>
>> 3. Only other issue at the moment I can think of is whether or not
>> anything
>> prominent has been put in main documentation about changed behaviour of
>> module
>> loading in mod_python.publisher. In the main the changes are for the
>> better
>> and things will actually work better.
>>
>> There is one case though where existing code can stop working. This is
>> where
>> code makes an assumption that global data within a publisher module is
>> preserved
>> across module reloads.
>>
>> I am not saying this change is wrong, but it perhaps may need to be
>> clearly
>> explained in documentation else we will get lots of questions about it
>> no doubt, with the common answer probably being that they shouldn't be
>> storing
>> such persistent module data in a module that can be reloaded anyway.
>>
>> Thats all for now until I can get some real time to look at it. :-)
>>
>> Graham
>>
>>
>>
>> On 18/08/2005, at 4:03 AM, Jim Gallacher wrote:
>>
>>> Here are the rules:
>>>
>>> In order for a file to be officially announced, it has to be tested by
>>> developers on the dev list. Anyone subscribed to this list can (and
>>> should feel obligated to :-) ) test it, and provide feedback *to _this_
>>> list*! (Not the mod_python@modpython.org list, and preferably not me
>>> personally).
>>>
>>> The files are (temporarily) available here:
>>>
>>> http://www.modpython.org/dist/
>>>
>>> Please download it, then do the usual
>>>
>>> $ ./configure --with-apxs=/wherever/it/is
>>> $ make
>>> $ (su)
>>> # make install
>>>
>>> Then (as non-root user!)
>>>
>>> $ cd test
>>> $ python test.py
>>>
>>> And see if any tests fail. If they pass, send a +1 to the list, if they
>>> fail, send the details (the versions of OS, Python and Apache, the test
>>> output, and suggestions, if any).
>>>
>>> Thank you,
>>>
>>> Jim
>>
>>
>
Re: flex [was mod_python 3.2.0-BETA available for testing]
Posted by Jim Gallacher <jg...@sympatico.ca>.
Nick wrote:
> Jim,
>
> I don't think it's too verbose, but maybe you could delay it to the end
> of the configure script so you don't have to either interrupt with
> control-C or scroll back to see what went "wrong."
Good point. I'll move it to the end. Since most systems still ship with
the older flex, most people that compile mod_python will see this message.
> Here's another idea: Fail the flex test fairly silently (e.g. just
> "no"), but fall back to a script that generates a nice, verbose error
> message explaining the situation. That way, when the user tries to call
> "make" after modifying the .l file, the fake flex alternative script
> gets called, displays the message, and exits with status 1.
My personal preference is to keep this sort of thing in configure. I
hate it when configure succeeds but then make fails. I'm more inclined
to leave the message in configure, but with the additional test you
suggest added just in case the .l file has changed.
Jim
> Nick
>
> Jim Gallacher wrote:
>
>> Gregory (Grisha) Trubetskoy wrote:
>>
>>>
>>> OK, here is the flex scoop - as the the docs point out, anything
>>> before 2.5.31 is not reentrant and I think even uses a slightly
>>> different interface so older flex won't even process the psp_parser.l
>>> file correctly.
>>>
>>> Looking at Fedora Core 4, it still has flex 2.5.4a. (Note that 2.5.31
>>> > 2.5.4 because 31 > 4 - I for a while had trouble seeing that for
>>> some reason), so the new flex is still not commonplace.
>>>
>>> So until reentrant flex becomes commonplace, the solution was to include
>>> a pre-parsed psp_parser.c so that you woudln't need flex at all to
>>> compile mod_python. Looks like this still should be the case.
>>>
>>> The ./configure should just print a warning that if flex is not found
>>> or too old, should you need to rebuild psp_parser.c you will need to
>>> get the right version of flex.
>>>
>>
>> I've made the changes to configure.in, but before I commit I wanted to
>> get some feedback. Are the following configure messages unclear or too
>> verbose?
>>
>> For the case of the missing flex, ./configure will generate:
>> ...
>> checking for --with-flex... no
>> checking for flex... no
>> configure: WARNING: flex not found
>> You can generally ignore this warning unless you need to regenerate
>> psp_parser.c from psp_parse.l. If you do need regenerate psp_parser.c,
>> use --with-flex to specify the location of flex.
>> See the README for more information.
>> ...
>>
>> For the case of the wrong flex version, ./configure will generate:
>> ...
>> checking for --with-flex... no
>> checking for flex... /usr/local/sbin/flex
>> found /usr/local/sbin/flex, we'll use this. Use --with-flex to specify
>> another.
>> checking flex version... configure: WARNING: Flex version 2.5.4 found.
>> Version 2.5.31 or greater is required.
>> You can generally ignore this warning unless you need to regenerate
>> psp_parser.c from psp_parse.l. If you do need regenerate psp_parser.c,
>> use --with-flex to specify the location of the correct flex version.
>> See the README for more information.
>> ...
>>
>> Any comments?
>>
>> Jim
>
>
Re: flex [was mod_python 3.2.0-BETA available for testing]
Posted by "Gregory (Grisha) Trubetskoy" <gr...@apache.org>.
On Fri, 26 Aug 2005, Nick wrote:
> Here's another idea: Fail the flex test fairly silently (e.g. just "no"), but
> fall back to a script that generates a nice, verbose error message explaining
> the situation. That way, when the user tries to call "make" after modifying
> the .l file, the fake flex alternative script gets called, displays the
> message, and exits with status 1.
Yep, you probably don't need a separate script - it can all fit in the
Makefile somewhere. That is how it worked originally, sort of, 'cept there
was no message - it just failed if flex wasn't there ;-)
But I'm also perfectly fine with what Jim already has in configure.in, if
we don't have time for these refinements, I'd leave it as it is, no
biggie.
Grisha
Re: flex [was mod_python 3.2.0-BETA available for testing]
Posted by Nick <ni...@dd.revealed.net>.
Jim,
I don't think it's too verbose, but maybe you could delay it to the end of
the configure script so you don't have to either interrupt with control-C or
scroll back to see what went "wrong."
Here's another idea: Fail the flex test fairly silently (e.g. just "no"),
but fall back to a script that generates a nice, verbose error message
explaining the situation. That way, when the user tries to call "make"
after modifying the .l file, the fake flex alternative script gets called,
displays the message, and exits with status 1.
Nick
Jim Gallacher wrote:
> Gregory (Grisha) Trubetskoy wrote:
>
>>
>> OK, here is the flex scoop - as the the docs point out, anything
>> before 2.5.31 is not reentrant and I think even uses a slightly
>> different interface so older flex won't even process the psp_parser.l
>> file correctly.
>>
>> Looking at Fedora Core 4, it still has flex 2.5.4a. (Note that 2.5.31
>> > 2.5.4 because 31 > 4 - I for a while had trouble seeing that for
>> some reason), so the new flex is still not commonplace.
>>
>> So until reentrant flex becomes commonplace, the solution was to include
>> a pre-parsed psp_parser.c so that you woudln't need flex at all to
>> compile mod_python. Looks like this still should be the case.
>>
>> The ./configure should just print a warning that if flex is not found
>> or too old, should you need to rebuild psp_parser.c you will need to
>> get the right version of flex.
>>
>
> I've made the changes to configure.in, but before I commit I wanted to
> get some feedback. Are the following configure messages unclear or too
> verbose?
>
> For the case of the missing flex, ./configure will generate:
> ...
> checking for --with-flex... no
> checking for flex... no
> configure: WARNING: flex not found
> You can generally ignore this warning unless you need to regenerate
> psp_parser.c from psp_parse.l. If you do need regenerate psp_parser.c,
> use --with-flex to specify the location of flex.
> See the README for more information.
> ...
>
> For the case of the wrong flex version, ./configure will generate:
> ...
> checking for --with-flex... no
> checking for flex... /usr/local/sbin/flex
> found /usr/local/sbin/flex, we'll use this. Use --with-flex to specify
> another.
> checking flex version... configure: WARNING: Flex version 2.5.4 found.
> Version 2.5.31 or greater is required.
> You can generally ignore this warning unless you need to regenerate
> psp_parser.c from psp_parse.l. If you do need regenerate psp_parser.c,
> use --with-flex to specify the location of the correct flex version.
> See the README for more information.
> ...
>
> Any comments?
>
> Jim
Re: flex [was mod_python 3.2.0-BETA available for testing]
Posted by Jim Gallacher <jg...@sympatico.ca>.
Gregory (Grisha) Trubetskoy wrote:
>
> OK, here is the flex scoop - as the the docs point out, anything before
> 2.5.31 is not reentrant and I think even uses a slightly different
> interface so older flex won't even process the psp_parser.l file correctly.
>
> Looking at Fedora Core 4, it still has flex 2.5.4a. (Note that 2.5.31 >
> 2.5.4 because 31 > 4 - I for a while had trouble seeing that for some
> reason), so the new flex is still not commonplace.
>
> So until reentrant flex becomes commonplace, the solution was to include
> a pre-parsed psp_parser.c so that you woudln't need flex at all to
> compile mod_python. Looks like this still should be the case.
>
> The ./configure should just print a warning that if flex is not found or
> too old, should you need to rebuild psp_parser.c you will need to get
> the right version of flex.
>
I've made the changes to configure.in, but before I commit I wanted to
get some feedback. Are the following configure messages unclear or too
verbose?
For the case of the missing flex, ./configure will generate:
...
checking for --with-flex... no
checking for flex... no
configure: WARNING: flex not found
You can generally ignore this warning unless you need to regenerate
psp_parser.c from psp_parse.l. If you do need regenerate psp_parser.c,
use --with-flex to specify the location of flex.
See the README for more information.
...
For the case of the wrong flex version, ./configure will generate:
...
checking for --with-flex... no
checking for flex... /usr/local/sbin/flex
found /usr/local/sbin/flex, we'll use this. Use --with-flex to specify
another.
checking flex version... configure: WARNING: Flex version 2.5.4 found.
Version 2.5.31 or greater is required.
You can generally ignore this warning unless you need to regenerate
psp_parser.c from psp_parse.l. If you do need regenerate psp_parser.c,
use --with-flex to specify the location of the correct flex version.
See the README for more information.
...
Any comments?
Jim
Re: flex [was mod_python 3.2.0-BETA available for testing]
Posted by Jim Gallacher <jg...@sympatico.ca>.
Gregory (Grisha) Trubetskoy wrote:
>
> OK, here is the flex scoop - as the the docs point out, anything before
> 2.5.31 is not reentrant and I think even uses a slightly different
> interface so older flex won't even process the psp_parser.l file correctly.
>
> Looking at Fedora Core 4, it still has flex 2.5.4a. (Note that 2.5.31 >
> 2.5.4 because 31 > 4 - I for a while had trouble seeing that for some
> reason), so the new flex is still not commonplace.
This is true. I took a quick survey when I was adding the --with-flex
option, and 2.5.4 was the most common. Oh, and the version numbering
confused me for a while as well, but the configure script does take it
into account when deciding if it has the correct flex.
> So until reentrant flex becomes commonplace, the solution was to include
> a pre-parsed psp_parser.c so that you woudln't need flex at all to
> compile mod_python. Looks like this still should be the case.
>
> The ./configure should just print a warning that if flex is not found or
> too old, should you need to rebuild psp_parser.c you will need to get
> the right version of flex.
When I added the flex detection I had not considered the case where
there was *no* flex on the system... just tested for the correct
version. :-(
Also, I wasn't sure how verbose to make the configure message. I just
figured that people would see the WARNING and then go to the README to
see what the deal was, which would have been the perfect plan... if I
had actually updated the README accordingly.
I'll make the fixes to configure and the README. Is the explanation in
doc-html/inst-configure.html clear or does it need to be revised as well?
Jim
>> See doc-html/inst-configure.html:
>>
>> """ Attempts to locate flex and determine its version. If flex
>> cannot be found in your PATH configure will fail. If the wrong
>> version is found configure will generate a warning. You can generally
>> ignore this warning unless you need to re-create src/psp_parser.c.
>>
>> The parser used by psp (See 4.9) is written in C generated using flex.
>> This requires a reentrant version of flex which at this time is
>> 2.5.31. Most platforms however ship with version 2.5.4 which is not
>> suitable, so a pre-generated copy of psp_parser.c is included with the
>> source. If you do need to compile src/psp_parser.c you must get the
>> correct flex version.
>>
>> If the first flex binary in the path is not suitable or not the one
>> desired you can specify an alternative location with the --with-flex
>> option, e.g:
>>
>> $ ./configure --with-flex=/usr/local/bin/flex
>>
>> """
>>
>> I'll copy this into the README as well.
>>
>> In 3.1.x the path to flex was hard coded to /usr/local/bin, which
>> caused compilation to fail and required the user to edit the Makefile
>> manually. Other than the detection of the flex path, the behaviour is
>> the same as 3.1.x. The previous warning about needing the correct flex
>> version was just a comment in the Makefile. Any suggestions on how to
>> make this clearer are welcome.
>>