You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pylucene-dev@lucene.apache.org by Bill Janssen <ja...@parc.com> on 2010/03/03 01:56:41 UTC

Arghh! Makefile mysteries...

I'm updating my build scripts for Linux.  It's a bit tricky, because I'm
installing all the Python modules in a separate spot,
/opt/uplib-1.7.9/lib[64]/python[2.5,2.6,2.7]/[site,dist]-packages/,
depending on the Linux distro and machine architecture.  Call it SITEDIR.
I'm trying not to affect the user's installed non-UpLib Python modules.

To build PyLucene properly, I install a patched version of setuptools in
that location, as well, and use that version of setuptools to build
PyLucene.  It's tricky, because setuptools out of the box doesn't seem
to grok the lib/lib64 and site/dist differences, and usually tries to
install in the wrong place.  So I patch the setuptools sources, then
build an egg (python setup.py bdist_egg), then run "sh" on the egg,
specifying the --install-dir switch, to get setuptools into the correct
spot.

Then I build JCC, and install it into SITEDIR.

Then I try to build PyLucene.  That's where I run into difficulty.  I
setenv PYTHONPATH to SITEDIR, and invoke "make" like this (in this case):

export python=/usr/bin/python
export PYTHONPATH=${SITEDIR}
export JCCCOMMAND="${python} -m jcc"
make PREFIX_PYTHON=/usr ANT=ant PYTHON=${python} JCC="${JCCCOMMAND}" NUM_FILES=1

which gives me this:

/usr/bin/python -m jcc --shared --jar lucene-java-2.9.1/build/lucene-core-2.9.1.jar --jar lucene-java-2.9.1/build/contrib/snowball/lucene-snowball-2.9.1.jar --jar lucene-java-2.9.1/build/contrib/analyzers/common/lucene-analyzers-2.9.1.jar --jar lucene-java-2.9.1/build/contrib/regex/lucene-regex-2.9.1.jar --jar lucene-java-2.9.1/build/contrib/memory/lucene-memory-2.9.1.jar --jar lucene-java-2.9.1/build/contrib/highlighter/lucene-highlighter-2.9.1.jar --jar lucene-java-2.9.1/build/contrib/queries/lucene-queries-2.9.1.jar --jar lucene-java-2.9.1/build/contrib/instantiated/lucene-instantiated-2.9.1.jar --jar build/jar/extensions.jar --package java.lang java.lang.System java.lang.Runtime --package java.util java.util.Arrays java.text.SimpleDateFormat --package java.io java.io.StringReader java.io.InputStreamReader java.io.FileInputStream --exclude org.apache.lucene.queryParser.Token --exclude org.apache.lucene.queryParser.TokenMgrError --exclude org.apache.lucene.queryParser.Query
 ParserTokenManager --exclude org.apache.lucene.queryParser.ParseException --exclude org.apache.lucene.search.regex.JakartaRegexpCapabilities --exclude org.apache.regexp.RegexpTunnel --python lucene --mapping org.apache.lucene.document.Document 'get:(Ljava/lang/String;)Ljava/lang/String;' --mapping java.util.Properties 'getProperty:(Ljava/lang/String;)Ljava/lang/String;' --sequence org.apache.lucene.search.Hits 'length:()I' 'doc:(I)Lorg/apache/lucene/document/Document;' --rename org.apache.lucene.search.highlight.SpanScorer=HighlighterSpanScorer --version 2.9.1 --module python/collections.py --files 1 --build 
Traceback (most recent call last):
  File "/usr/lib/python2.5/runpy.py", line 95, in run_module
    filename, loader, alter_sys)
  File "/usr/lib/python2.5/runpy.py", line 52, in _run_module_code
    mod_name, mod_fname, mod_loader)
  File "/usr/lib/python2.5/runpy.py", line 32, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.5/site-packages/JCC-1.9-py2.5-linux-i686.egg/jcc/__init__.py", line 28, in <module>
    cpp.jcc(sys.argv)
  File "/usr/lib/python2.5/site-packages/JCC-1.9-py2.5-linux-i686.egg/jcc/cpp.py", line 340, in jcc
    raise ValueError, "Invalid argument: %s" %(arg)
ValueError: Invalid argument: --rename
make: *** [compile] Error 255

You'll notice that what's happening is that I'm not loading the JCC (2.4.1) from
SITEDIR; instead, I'm loading an old one (1.9) from the system location.  Why
is this?  I thought it might be due to setuptools, but I can't reproduce this

Re: Arghh! Makefile mysteries...

Posted by Bill Janssen <ja...@parc.com>.
Andi Vajda <va...@apache.org> wrote:

> Maybe it is because Python picks stuff from site-packages first ? (not sure)

Maybe so.  Anytime setuptools is involved, I suspect just that.  That
bug is still in the setuptools tracker.

But I created a Makefile with just this in it:

all:
        python -c "import os; print os.environ.get('PYTHONPATH')"
        python -c "import sys; print sys.path"
        python -m jcc --version

And then I do a "make all" on it.  The last line errs out, but does so
with the right stack trace, if PYTHONPATH is set.  What I don't understand
is what the difference is between this and the PyLucene Makefile.

Bill

/tmp/x 91 % make all
python -c "import os; print os.environ.get('PYTHONPATH')"
None
python -c "import sys; print sys.path"
['', '/usr/lib/python2.5/site-packages/JCC-1.9-py2.5-linux-i686.egg', '/usr/lib/python2.5/site-packages/lucene-2.4.0_649619-py2.5-linux-i686.egg', '/usr/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg', '/usr/lib/python2.5/site-packages/vobject-0.8.1c-py2.5.egg', '/usr/lib/python2.5/site-packages/python_dateutil-1.4.1-py2.5.egg', '/usr/lib/python25.zip', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload', '/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages/Numeric', '/usr/lib/python2.5/site-packages/PIL', '/usr/lib/python2.5/site-packages/gst-0.10', '/usr/lib/python2.5/site-packages/gtk-2.0', '/opt/uplib-1.7.5/lib/python2.5/site-packages', '/opt/uplib-1.7.5/lib/python2.5/site-packages/PIL', '/opt/uplib-1.7.9/lib/python2.5/site-packages', '/opt/uplib-1.7.9/lib/python2.5/site-packages/PIL']
python -m jcc --version
Traceback (most recent call last):
  File "/usr/lib/python2.5/runpy.py", line 95, in run_module
    filename, loader, alter_sys)
  File "/usr/lib/python2.5/runpy.py", line 52, in _run_module_code
    mod_name, mod_fname, mod_loader)
  File "/usr/lib/python2.5/runpy.py", line 32, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.5/site-packages/JCC-1.9-py2.5-linux-i686.egg/jcc/__init__.py", line 28, in <module>
    cpp.jcc(sys.argv)
  File "/usr/lib/python2.5/site-packages/JCC-1.9-py2.5-linux-i686.egg/jcc/cpp.py", line 308, in jcc
    version = args[i]
IndexError: list index out of range
make: *** [all] Error 255
greta : /tmp/x 89 % setenv PYTHONPATH /opt/uplib-1.7.9/lib/python2.5/site-packages
greta : /tmp/x 90 % make all
python -c "import os; print os.environ.get('PYTHONPATH')"
/opt/uplib-1.7.9/lib/python2.5/site-packages
python -c "import sys; print sys.path"
['', '/opt/uplib-1.7.9/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg', '/opt/uplib-1.7.9/lib/python2.5/site-packages/vobject-0.8.1c-py2.5.egg', '/opt/uplib-1.7.9/lib/python2.5/site-packages/JCC-2.4.1-py2.5-linux-i686.egg', '/opt/uplib-1.7.9/lib/python2.5/site-packages/lucene-2.9.1-py2.5-linux-i686.egg', '/usr/lib/python2.5/site-packages/JCC-1.9-py2.5-linux-i686.egg', '/usr/lib/python2.5/site-packages/lucene-2.4.0_649619-py2.5-linux-i686.egg', '/usr/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg', '/usr/lib/python2.5/site-packages/vobject-0.8.1c-py2.5.egg', '/usr/lib/python2.5/site-packages/python_dateutil-1.4.1-py2.5.egg', '/opt/uplib-1.7.9/lib/python2.5/site-packages', '/usr/lib/python25.zip', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk', '/usr/lib/python2.5/lib-dynload', '/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages/Numeric', '/usr/lib/python2.5/site-packages/PIL', '/usr/lib/python2.5/site-packa
 ges/gst-0.10', '/usr/lib/python2.5/site-packages/gtk-2.0', '/opt/uplib-1.7.5/lib/python2.5/site-packages', '/opt/uplib-1.7.5/lib/python2.5/site-packages/PIL', '/opt/uplib-1.7.9/lib/python2.5/site-packages/PIL']
python -m jcc --version
Traceback (most recent call last):
  File "/usr/lib/python2.5/runpy.py", line 95, in run_module
    filename, loader, alter_sys)
  File "/usr/lib/python2.5/runpy.py", line 52, in _run_module_code
    mod_name, mod_fname, mod_loader)
  File "/usr/lib/python2.5/runpy.py", line 32, in _run_code
    exec code in run_globals
  File "/opt/uplib-1.7.9/lib/python2.5/site-packages/JCC-2.4.1-py2.5-linux-i686.egg/jcc/__init__.py", line 27, in <module>
    import jcc.__main__
  File "/opt/uplib-1.7.9/lib/python2.5/site-packages/JCC-2.4.1-py2.5-linux-i686.egg/jcc/__main__.py", line 88, in <module>
    cpp.jcc(sys.argv)
  File "/opt/uplib-1.7.9/lib/python2.5/site-packages/JCC-2.4.1-py2.5-linux-i686.egg/jcc/cpp.py", line 337, in jcc
    version = args[i]
IndexError: list index out of range
make: *** [all] Error 255
/tmp/x 91 % 

> I think this whole mess would be a lot simpler if you used virtualenv.py.
> 
>   http://pypi.python.org/pypi/virtualenv
> 
> It lets you do what you're trying to do - install stuff without
> affecting a user's install, yet override some of their libs in a
> special place inheriting from their setup otherwise if you so choose.

No thanks.  Just more complexity to manage.  I'd like to get setuptools
out of the mix.  It's easy_install.pth that's screwing up the sys.path
order.

Bill

Re: Arghh! Makefile mysteries...

Posted by Andi Vajda <va...@apache.org>.
On Tue, 2 Mar 2010, Bill Janssen wrote:

> I'm updating my build scripts for Linux.  It's a bit tricky, because I'm
> installing all the Python modules in a separate spot,
> /opt/uplib-1.7.9/lib[64]/python[2.5,2.6,2.7]/[site,dist]-packages/,
> depending on the Linux distro and machine architecture.  Call it SITEDIR.
> I'm trying not to affect the user's installed non-UpLib Python modules.
>
> To build PyLucene properly, I install a patched version of setuptools in
> that location, as well, and use that version of setuptools to build
> PyLucene.  It's tricky, because setuptools out of the box doesn't seem
> to grok the lib/lib64 and site/dist differences, and usually tries to
> install in the wrong place.  So I patch the setuptools sources, then
> build an egg (python setup.py bdist_egg), then run "sh" on the egg,
> specifying the --install-dir switch, to get setuptools into the correct
> spot.
>
> Then I build JCC, and install it into SITEDIR.
>
> Then I try to build PyLucene.  That's where I run into difficulty.  I
> setenv PYTHONPATH to SITEDIR, and invoke "make" like this (in this case):
>
> export python=/usr/bin/python
> export PYTHONPATH=${SITEDIR}
> export JCCCOMMAND="${python} -m jcc"
> make PREFIX_PYTHON=/usr ANT=ant PYTHON=${python} JCC="${JCCCOMMAND}" NUM_FILES=1
>
> which gives me this:
>
> /usr/bin/python -m jcc --shared --jar lucene-java-2.9.1/build/lucene-core-2.9.1.jar --jar lucene-java-2.9.1/build/contrib/snowball/lucene-snowball-2.9.1.jar --jar lucene-java-2.9.1/build/contrib/analyzers/common/lucene-analyzers-2.9.1.jar --jar lucene-java-2.9.1/build/contrib/regex/lucene-regex-2.9.1.jar --jar lucene-java-2.9.1/build/contrib/memory/lucene-memory-2.9.1.jar --jar lucene-java-2.9.1/build/contrib/highlighter/lucene-highlighter-2.9.1.jar --jar lucene-java-2.9.1/build/contrib/queries/lucene-queries-2.9.1.jar --jar lucene-java-2.9.1/build/contrib/instantiated/lucene-instantiated-2.9.1.jar --jar build/jar/extensions.jar --package java.lang java.lang.System java.lang.Runtime --package java.util java.util.Arrays java.text.SimpleDateFormat --package java.io java.io.StringReader java.io.InputStreamReader java.io.FileInputStream --exclude org.apache.lucene.queryParser.Token --exclude org.apache.lucene.queryParser.TokenMgrError --exclude org.apache.lucene.queryParser.Que
 ry
> ParserTokenManager --exclude org.apache.lucene.queryParser.ParseException --exclude org.apache.lucene.search.regex.JakartaRegexpCapabilities --exclude org.apache.regexp.RegexpTunnel --python lucene --mapping org.apache.lucene.document.Document 'get:(Ljava/lang/String;)Ljava/lang/String;' --mapping java.util.Properties 'getProperty:(Ljava/lang/String;)Ljava/lang/String;' --sequence org.apache.lucene.search.Hits 'length:()I' 'doc:(I)Lorg/apache/lucene/document/Document;' --rename org.apache.lucene.search.highlight.SpanScorer=HighlighterSpanScorer --version 2.9.1 --module python/collections.py --files 1 --build
> Traceback (most recent call last):
>  File "/usr/lib/python2.5/runpy.py", line 95, in run_module
>    filename, loader, alter_sys)
>  File "/usr/lib/python2.5/runpy.py", line 52, in _run_module_code
>    mod_name, mod_fname, mod_loader)
>  File "/usr/lib/python2.5/runpy.py", line 32, in _run_code
>    exec code in run_globals
>  File "/usr/lib/python2.5/site-packages/JCC-1.9-py2.5-linux-i686.egg/jcc/__init__.py", line 28, in <module>
>    cpp.jcc(sys.argv)
>  File "/usr/lib/python2.5/site-packages/JCC-1.9-py2.5-linux-i686.egg/jcc/cpp.py", line 340, in jcc
>    raise ValueError, "Invalid argument: %s" %(arg)
> ValueError: Invalid argument: --rename
> make: *** [compile] Error 255
>
> You'll notice that what's happening is that I'm not loading the JCC (2.4.1) from
> SITEDIR; instead, I'm loading an old one (1.9) from the system location.  Why
> is this?  I thought it might be due to setuptools, but I can't reproduce this

Maybe it is because Python picks stuff from site-packages first ? (not sure)

I think this whole mess would be a lot simpler if you used virtualenv.py.

   http://pypi.python.org/pypi/virtualenv

It lets you do what you're trying to do - install stuff without affecting a 
user's install, yet override some of their libs in a special place 
inheriting from their setup otherwise if you so choose.

Once you've created a virtual env, what is picked up depends on the python 
executable invoked - the one in the virtual env or the system one - and 
involves no messing with env vars at all.

Andi..