You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pylucene-dev@lucene.apache.org by Julian Maibaum <j....@semantics.de> on 2010/05/12 18:30:39 UTC

Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Hi,

Christian Heimes, Dirk Rothe, and I have jcc-wrapped bobo-browse (http://code.google.com/p/bobo-browse/) in order to add faceted search capabilities to PyLucene. However, the two modules don't play well together, as wrappers from PyLucene cannot be used in a bobo-browse context and vice versa.
What is the best way to get classes from two different jcc python extension modules to interact?

Setup:
X64 Ubuntu 9.10 Python 2.6 Tried both versions:
   JCC-2.4.1-py2.6-linux-x86_64.egg compiled with --shared
   JCC-2.5.1-py2.6-linux-x86_64.egg compiled with --shared
Tried both versions:
   lucene-2.9.1-py2.6-linux-x86_64.egg
   lucene-2.9.2-py2.6-linux-x86_64.egg
bobobrowse-2.5.0_rc1-py2.6-linux-x86_64.egg
Both the bobo-browse python extension module and PyLucene are attached to the main python thread and have been initialized using their respective initVM methods.

I have attached the makefile (modified from PyLucene) that was used to build bobo-browse.

#---- START CODE ----
import bobobrowse as bb
import lucene
import os

CLASSPATH = os.pathsep.join((lucene.CLASSPATH, bb.CLASSPATH)
lucene.initVM(CLASSPATH)
bb.initVM(CLASSPATH)

typeHandler = bb.MultiValueFacetHandler("type")
placeHandler = bb.MultiValueFacetHandler("id")

facetHandlers = lucene.JArray('object')(2, bb.FacetHandler)
facetHandlers[0] = typeHandler
facetHandlers[1] = placeHandler

reader = bb.IndexReader.open('/path/to/my/lucene/index', True)
facetHandlers = bb.Arrays.asList(facetHandlers)
indexReader = bb.BoboIndexReader.getInstance(reader, facetHandlers)

#---- END CODE ----

Expected:
The example code above works - java objects from classes wrapped as part of the PyLucene module can transparently be used by other python extension modules such as bobo-browse.

Experienced:
   Throws Errors:
Traceback (most recent call last):
  File "./dynamic/vls/search/facets.py", line 115, in <module>
    test()
  File "./dynamic/smc/jcc/__init__.py", line 52, in wrapper
    return func(*args, **kwargs)
  File "./dynamic/vls/search/facets.py", line 57, in test
    facetHandlers = lucene.JArray('object')(2, bb.FacetHandler)
ValueError: <type 'FacetHandler'>

   and

bobobrowse.InvalidArgsError: (<type 'BoboIndexReader'>, 'getInstance', (<IndexReader: org.apache.lucene.index.DirectoryOwningReader@30419d05>, <List: [com.browseengine.bobo.facets.impl.MultiValueFacetHandler@3f829e6f, com.browseengine.bobo.facets.impl.MultiValueFacetHandler@2980f96c]>))

   respectively.

The Errors can be avoided by changing the corresponding lines in the example above to:
facetHandlers = bb.JArray('object')(2, bb.FacetHandler)
reader = bb.IndexReader.open('/path/to/my/lucene/index', True)

However, this only works because lucene's IndexReader is referenced by bobo-browse and has therefore been wrapped at build-time. At later stages other lucene classes are needed to work with bobobrowse that no wrappers have been built for (e.g. QueryParser). I understand that jcc can be forced to built and include wrappers for e.g. org.apache.lucene.queryParser.QueryParser that are then accessible for use with classes from the bobo-browse module as e.g 'bobobrowse.QueryParser'. The most simple solution is calling jcc with an additional --jar lucene.jar parameter - in effect leading to lucene being installed twice: First self-contained PyLucene module and, second, as part of bobo-browse in the flat bobo-browse module namespace. This is not quite desirable:
Since
bb.IndexReader.class_ == lucene.IndexReader.class_
--> class org.apache.lucene.index.IndexReader, anyways, it would arguably be more intuitive to be able to use the available wrappers in the context of other jcc modules even if they have not specifically been forced into that other module. Is there any way to obtain the desired behavior?


Thank you very much and all the best,

Julian Maibaum


j.maibaum(at)semantics.de
s<e>mantics GmbH
http://www.semantics.de

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Tue, 18 May 2010, Christian Heimes wrote:

>> The rather strange bug is that the bobobrowse package doesn't contain
>> any of the bobobrowse classes when I compile JCC, PyLucene and
>> Bobo-Browse inside our environment. Let's see how it's behaving with
>> your latest changes.
>
> Looks like this bug is gone for good. I suspect it was an incompatibility 
> between my rpath and your dlopen() hack.

Good, I'm glad to see my dlopen() hack go too. I knew there had to be a 
better way. Thanks for suggesting it.

Andi..

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Christian Heimes <li...@cheimes.de>.
> The rather strange bug is that the bobobrowse package doesn't contain
> any of the bobobrowse classes when I compile JCC, PyLucene and
> Bobo-Browse inside our environment. Let's see how it's behaving with
> your latest changes.

Looks like this bug is gone for good. I suspect it was an 
incompatibility between my rpath and your dlopen() hack.

Thanks!

Christian

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Thu, 20 May 2010, Julian Maibaum wrote:

>>>> Thanks Andi! I'll give it a try tomorrow. I'll also ask Julian to provide 
>>>> some test and sample code for you.
>
> Thank you very much for the great work and effort you're putting into jcc, 
> Andi. As far as I can tell, the linking between classes from PyLucene and 
> bobo-browse works flawlessly now - cool :)

This is most excellent news. This new --import feature was the missing piece 
to make --shared really useful. Thank you for pointing it out.

> I have attached the latest version of the Makefile that Christian has 
> modified from PyLucene as well as a simple sample test-case based on 
> http://snaprojects.jira.com/wiki/display/BOBO/Getting+Started#GettingStarted-Example%3A

Andi..

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Julian Maibaum <j....@semantics.de>.
>>> Thanks Andi! I'll give it a try tomorrow. I'll also ask Julian to provide some test and sample code for you.

Thank you very much for the great work and effort you're putting into jcc, Andi. As far as I can tell, the linking between classes from PyLucene and bobo-browse works flawlessly now - cool :)

I have attached the latest version of the Makefile that Christian has modified from PyLucene as well as a simple sample test-case based on http://snaprojects.jira.com/wiki/display/BOBO/Getting+Started#GettingStarted-Example%3A

Julian

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Christian Heimes <ch...@cheimes.de>.
Am 18.05.2010 23:19, schrieb Andi Vajda:
> The definition of 'sensible' then becomes a shouting match. This is why
> there is a dozen of so-called sensible defaults for each and every platform
> claimed to be supported. Between python 2.5, 2.6 and 2.7 the way to invoke a
> module from the command line changed three times. What's a sensible default
> to that ??

Good point ... At least "python -m jcc.__main__" should work across 
Python 2.5 to 2.7. The regression of "python2.6 -m package" in Python 
2.6 was a real bummer.

> No, -l doesn't work, but adding the full path of the .so to the link command
> line works fine. I just add lucene._lucene.__file__ to setup()'s
> extra_link_args.

Oh, it's so easy? I was looking for a special argument similar to -l.

>> I'll send you a new Makefile and some example code tomorrow. In the mean time
>> I'm happy to write that your changes have fixed the issue with class_ and the
>> missing OrderHitsDesc enum.
>
> Excellent, thanks for the good news !

Sorry, the it's going to take me a little longer until I have something 
useful for you.

Christian

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Tue, 18 May 2010, Christian Heimes wrote:

>> The Makefile has about a dozen sets of defaults for a variety of platforms.
>> Pick the one closest to your setup, uncomment it (and change it if needed).
>> Then you don't have to enter these on the command line.
>
> My point is that you may want to remove most examples and replace them with 
> sensible defaults that work for all Un*x platforms and Python 2.5+.

The definition of 'sensible' then becomes a shouting match. This is why 
there is a dozen of so-called sensible defaults for each and every platform 
claimed to be supported. Between python 2.5, 2.6 and 2.7 the way to invoke a 
module from the command line changed three times. What's a sensible default 
to that ??

>> This is how I solved this on Windows. I'll try to see if the same trick
>> works on Linux. If it does, then the export=True parameter would become
>> unnecessary. Still, unless the dynamic loader can find _lucene.so on its 
>> own
>> (and it should be via some -rpath tricks), loading the --import'ed module
>> first might still be a good idea.
>
> I tried to implement the very same feature. You beat me to it. :) It's 
> working very well on Linux. How did you trick the linker to include 
> _lucene.so as a dependency? The -l option obviously doesn't work since the 
> shared library has no "lib" prefix.

No, -l doesn't work, but adding the full path of the .so to the link command 
line works fine. I just add lucene._lucene.__file__ to setup()'s 
extra_link_args.

>> I'd very much like to try to reproduce your problems with your software.
>> Is there any way I can get your stuff and build it here ?
>> It would be much easier to debug this if we worked with the same stuff.
>
> I'll send you a new Makefile and some example code tomorrow. In the mean time 
> I'm happy to write that your changes have fixed the issue with class_ and the 
> missing OrderHitsDesc enum.

Excellent, thanks for the good news !

Andi..

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Christian Heimes <li...@cheimes.de>.
> The Makefile has about a dozen sets of defaults for a variety of platforms.
> Pick the one closest to your setup, uncomment it (and change it if needed).
> Then you don't have to enter these on the command line.

My point is that you may want to remove most examples and replace them 
with sensible defaults that work for all Un*x platforms and Python 2.5+.

> This is how I solved this on Windows. I'll try to see if the same trick
> works on Linux. If it does, then the export=True parameter would become
> unnecessary. Still, unless the dynamic loader can find _lucene.so on its own
> (and it should be via some -rpath tricks), loading the --import'ed module
> first might still be a good idea.

I tried to implement the very same feature. You beat me to it. :) It's 
working very well on Linux. How did you trick the linker to include 
_lucene.so as a dependency? The -l option obviously doesn't work since 
the shared library has no "lib" prefix.

> I'd very much like to try to reproduce your problems with your software.
> Is there any way I can get your stuff and build it here ?
> It would be much easier to debug this if we worked with the same stuff.

I'll send you a new Makefile and some example code tomorrow. In the mean 
time I'm happy to write that your changes have fixed the issue with 
class_ and the missing OrderHitsDesc enum.

>>> bobobrowse.FacetSpec.FacetSortSpec.OrderHitsDesc
<FacetSpec$FacetSortSpec: OrderHitsDesc>

>> - I've run into a rather strange bug with our custom builds of Python 2.6. It
>> differs from the stock build of Ubuntu in two ways. The python2.6 binary is
>> linked against libpython2.6.so (Ubuntu's python2.6 binary has libpython2.6.a
>> statically linked in) and I've set the environment variable LD_RUN_PATH which
>> causes the linker to add a "library rpath" to all ELF binaries. I wasn't able
>> to find the reason for the issue so far. Any hints are appreciated.
>
> You didn't describe the "rather strange bug" or am I missing something ?

Uups, I hit the send button too early. Sorry

The rather strange bug is that the bobobrowse package doesn't contain 
any of the bobobrowse classes when I compile JCC, PyLucene and 
Bobo-Browse inside our environment. Let's see how it's behaving with 
your latest changes.

Christian

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Tue, 18 May 2010, Andi Vajda wrote:

> I changed the branch_3x pylucene branch to point at the trunk for jcc.
> You might need to remove the jcc directory from your branch_3x checkout 
> before updating so as to not conflict with the new svn:externals setting.

An svn >= 1.5 client is required.

Ando..

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Tue, 18 May 2010, Christian Heimes wrote:

>> Adding the --import'ed .so libraries to the link line on Linux solve 
>> several
>> of the problems you reported:
>>
>>     - import order doesn't matter anymore
>>     - initVM() order doesn't matter anymore
>>     - class_ properties are now correct
>> 
>> This is now checked into the branch_3x pylucene's branch.
>> Please give it another try with your software,
>
> Thanks Andi! I'll give it a try tomorrow. I'll also ask Julian to provide 
> some test and sample code for you.

I changed the branch_3x pylucene branch to point at the trunk for jcc.
You might need to remove the jcc directory from your branch_3x checkout 
before updating so as to not conflict with the new svn:externals setting.

Andi..

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Christian Heimes <li...@cheimes.de>.
> Adding the --import'ed .so libraries to the link line on Linux solve several
> of the problems you reported:
>
>     - import order doesn't matter anymore
>     - initVM() order doesn't matter anymore
>     - class_ properties are now correct
>
> This is now checked into the branch_3x pylucene's branch.
> Please give it another try with your software,

Thanks Andi! I'll give it a try tomorrow. I'll also ask Julian to 
provide some test and sample code for you.

Christian

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Tue, 18 May 2010, Andi Vajda wrote:

>
> On Tue, 18 May 2010, Andi Vajda wrote:
>
>> I don't have this problem with my simple program.
>> My SillyAnalyzer class extends Lucene's Analyzer.
>> foo.Analyzer is the one imported from lucene.
>>
>>  >>> import lucene, foo
>>  >>> lucene.initVM()
>>  <jcc.JCCEnv object at 0x1004100d8>
>>  >>> foo.initVM(".")
>>  <jcc.JCCEnv object at 0x100410180>
>>  >>> foo.SillyAnalyzer.helloWorld()
>>  Hello World
>>  >>> foo.SillyAnalyzer.class_
>>  <Class: class bar.SillyAnalyzer>
>>  >>> foo.Analyzer.class_
>>  <Class: class org.apache.lucene.analysis.Analyzer>
>
> I reproduced the problem on Linux but not on Mac OS X.
> I suspect some linker issue...

Adding the --import'ed .so libraries to the link line on Linux solve several 
of the problems you reported:

   - import order doesn't matter anymore
   - initVM() order doesn't matter anymore
   - class_ properties are now correct

This is now checked into the branch_3x pylucene's branch.
Please give it another try with your software,

Thanks !

Andi..

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Tue, 18 May 2010, Andi Vajda wrote:

> I don't have this problem with my simple program.
> My SillyAnalyzer class extends Lucene's Analyzer.
> foo.Analyzer is the one imported from lucene.
>
>  >>> import lucene, foo
>  >>> lucene.initVM()
>  <jcc.JCCEnv object at 0x1004100d8>
>  >>> foo.initVM(".")
>  <jcc.JCCEnv object at 0x100410180>
>  >>> foo.SillyAnalyzer.helloWorld()
>  Hello World
>  >>> foo.SillyAnalyzer.class_
>  <Class: class bar.SillyAnalyzer>
>  >>> foo.Analyzer.class_
>  <Class: class org.apache.lucene.analysis.Analyzer>

I reproduced the problem on Linux but not on Mac OS X.
I suspect some linker issue...

Andi..

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Tue, 18 May 2010, Christian Heimes wrote:

> Setup
> =====
> I'm running the tests on an Ubuntu 10.04 X86_64 box with Python 2.6.5, Sun 
> Java 1.6.0_20, GCC 4.4.3 and patched setuptools 0.6c11 (all 64bit).
>
> JCC:
> $ svn co 
> http://svn.apache.org/repos/asf/lucene/pylucene/branches/branch_3x/jcc jcc_3x
> $ cd jcc_3x
> $ JCC_JDK=/usr/lib/jvm/java-6-sun python2.6 setup.py build
> $ JCC_JDK=/usr/lib/jvm/java-6-sun python2.6 setup.py install
>
> Small issue: The install target always rewrites the jcc.config module. This 
> is a problem when I install JCC with "sudo python2.6 setup.py install" since 
> it's not trivial to keep the environment variable JCC_JDK across a sudo call. 
> How do you like the idea to bind the creation of jcc.config to the build or 
> build_ext target?

That makes sense. Consider it a bug that it doesn't.

> LUCENE:
> $ cd ../pylucene-2.9.2-1
> $ make ANT=ant PYTHON=python2.6 JCC="python2.6 -m jcc.__main__ --shared" 
> NUM_FILES=2
> $ make ANT=ant PYTHON=python2.6 JCC="python2.6 -m jcc.__main__ --shared" 
> NUM_FILES=2 install
>
> Suggestion: How about providing sensible defaults for ANT, PYTHON, NUM_FILES 
> and JCC? It's annoying that I have to type the arguments over and over again. 
> ;)

The Makefile has about a dozen sets of defaults for a variety of platforms. 
Pick the one closest to your setup, uncomment it (and change it if needed).
Then you don't have to enter these on the command line.


> Issues
> ======
>
> - I have to follow a strict order (first import lucene, than 
> lucene.initVM(export=True), finally import bobobrowse). When I try to import 
> bobobrowse before the call to initVM(export=True), the import fails with 
> unresolved symbols. Have you tried to add _lucene.so as a dependency to 
> _bobobrowse.so? I believe a dependency should make the dlopen() call 
> obsolete.

This is how I solved this on Windows. I'll try to see if the same trick 
works on Linux. If it does, then the export=True parameter would become 
unnecessary. Still, unless the dynamic loader can find _lucene.so on its own 
(and it should be via some -rpath tricks), loading the --import'ed module 
first might still be a good idea.

> - With "--import lucene" JCC gets the class_ attribute of all types wrong, 
> that are part of the bobobrowse package. It either shows java.lang.Object as 
> class_ or the first lucene class in the hierarchy. Without "--import lucene" 
> it works as expected.

I don't have this problem with my simple program.
My SillyAnalyzer class extends Lucene's Analyzer.
foo.Analyzer is the one imported from lucene.

   >>> import lucene, foo
   >>> lucene.initVM()
   <jcc.JCCEnv object at 0x1004100d8>
   >>> foo.initVM(".")
   <jcc.JCCEnv object at 0x100410180>
   >>> foo.SillyAnalyzer.helloWorld()
   Hello World
   >>> foo.SillyAnalyzer.class_
   <Class: class bar.SillyAnalyzer>
   >>> foo.Analyzer.class_
   <Class: class org.apache.lucene.analysis.Analyzer>

I'd very much like to try to reproduce your problems with your software.
Is there any way I can get your stuff and build it here ?
It would be much easier to debug this if we worked with the same stuff.

> - I've run into a rather strange bug with our custom builds of Python 2.6. It 
> differs from the stock build of Ubuntu in two ways. The python2.6 binary is 
> linked against libpython2.6.so (Ubuntu's python2.6 binary has libpython2.6.a 
> statically linked in) and I've set the environment variable LD_RUN_PATH which 
> causes the linker to add a "library rpath" to all ELF binaries. I wasn't able 
> to find the reason for the issue so far. Any hints are appreciated.

You didn't describe the "rather strange bug" or am I missing something ?

Andi..

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Tue, 18 May 2010, Christian Heimes wrote:

>> Can you suggest a patch for this ?
>> It is not immediately obvious to me how to do this.
>
> I've attached a quick patch. Sorry for all the noise, my editor is configured 
> to reformat Python code.
>
> The patch provides a custom build_py command class for distutils / 
> setuptools. It writes jcc.config during the build_py phase if the file 
> doesn't exist or a build command is provided on the command line.

I reworked your patch somewhat and integrated it.
It's checked in rev 945938.

I learned some distutils-fu along the way :)
Thank you for the contribution !

Andi..

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Christian Heimes <ch...@cheimes.de>.
> Can you suggest a patch for this ?
> It is not immediately obvious to me how to do this.

I've attached a quick patch. Sorry for all the noise, my editor is 
configured to reformat Python code.

The patch provides a custom build_py command class for distutils / 
setuptools. It writes jcc.config during the build_py phase if the file 
doesn't exist or a build command is provided on the command line.

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Tue, 18 May 2010, Christian Heimes wrote:

> Setup
> =====
> I'm running the tests on an Ubuntu 10.04 X86_64 box with Python 2.6.5, Sun 
> Java 1.6.0_20, GCC 4.4.3 and patched setuptools 0.6c11 (all 64bit).
>
> JCC:
> $ svn co 
> http://svn.apache.org/repos/asf/lucene/pylucene/branches/branch_3x/jcc jcc_3x
> $ cd jcc_3x
> $ JCC_JDK=/usr/lib/jvm/java-6-sun python2.6 setup.py build
> $ JCC_JDK=/usr/lib/jvm/java-6-sun python2.6 setup.py install
>
> Small issue: The install target always rewrites the jcc.config module. This 
> is a problem when I install JCC with "sudo python2.6 setup.py install" since 
> it's not trivial to keep the environment variable JCC_JDK across a sudo call. 
> How do you like the idea to bind the creation of jcc.config to the build or 
> build_ext target?

Can you suggest a patch for this ?
It is not immediately obvious to me how to do this.

Andi..

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Christian Heimes <li...@cheimes.de>.
Andi Vajda wrote:
>> On Linux, there's got to be a way to make this work but I haven't found how
>> yet. I seem to be having trouble with linking to the Python Type objects. On
>> Windows, I expect this to be even more difficult.
>
> Adding a call to dlopen("_lucene.so", RTLD_NOW | RTLD_GLOBAL) solves the
> problem on Linux. I'm thinking of adding this to the initVM() call with an
> optional 'export=true/false' parameter... unless there is a better way to go
> about this.

Dear Andi,

thanks for your hard work! So far we have have found multiple smaller 
and larger issues with your experimental feature.

Setup
=====
I'm running the tests on an Ubuntu 10.04 X86_64 box with Python 2.6.5, 
Sun Java 1.6.0_20, GCC 4.4.3 and patched setuptools 0.6c11 (all 64bit).

JCC:
$ svn co 
http://svn.apache.org/repos/asf/lucene/pylucene/branches/branch_3x/jcc 
jcc_3x
$ cd jcc_3x
$ JCC_JDK=/usr/lib/jvm/java-6-sun python2.6 setup.py build
$ JCC_JDK=/usr/lib/jvm/java-6-sun python2.6 setup.py install

Small issue: The install target always rewrites the jcc.config module. 
This is a problem when I install JCC with "sudo python2.6 setup.py 
install" since it's not trivial to keep the environment variable JCC_JDK 
across a sudo call. How do you like the idea to bind the creation of 
jcc.config to the build or build_ext target?

LUCENE:
$ cd ../pylucene-2.9.2-1
$ make ANT=ant PYTHON=python2.6 JCC="python2.6 -m jcc.__main__ --shared" 
NUM_FILES=2
$ make ANT=ant PYTHON=python2.6 JCC="python2.6 -m jcc.__main__ --shared" 
NUM_FILES=2 install

Suggestion: How about providing sensible defaults for ANT, PYTHON, 
NUM_FILES and JCC? It's annoying that I have to type the arguments over 
and over again. ;)

BOBO-BROWSE:
$ cd ../bobobrowse
$ make
python2.6 -m jcc.__main__ --shared --jar 
bobo-browse-2.5.0-rc1/dist/bobo-browse-2.5.0-rc1.jar --include 
bobo-browse-2.5.0-rc1/lib/master/servlet-api.jar --include 
bobo-browse-2.5.0-rc1/lib/master/spring-webmvc.jar --include 
bobo-browse-2.5.0-rc1/lib/master/spring.jar --include 
bobo-browse-2.5.0-rc1/lib/master/log4j.jar --include 
bobo-browse-2.5.0-rc1/lib/master/protobuf-java-2.2.0.jar --include 
bobo-browse-2.5.0-rc1/lib/master/xstream.jar --include 
bobo-browse-2.5.0-rc1/lib/master/fastutil.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/ant.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/xercesImpl.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/commons-collections.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/commons-cli.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/commons-configuration.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/commons-logging.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/commons-lang.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/commons-digester.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/commons-httpclient.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/zoie-2.0.0-rc1.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/json.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/dwr.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/xmlParserAPIs.jar --classpath 
bobo-browse-2.5.0-rc1/lib/master/kamikaze-2.0.0.jar --import lucene 
--package java.lang java.lang.System java.lang.Runtime --package 
java.util java.util.Arrays java.text.SimpleDateFormat --package java.io 
java.io.StringReader java.io.InputStreamReader java.io.FileInputStream 
--python bobobrowse --version 2.5.0-rc1 --files 2 --build
...
$ make install

As you can see I'm using "--import lucene" here.

$ python2.6
>>> import lucene
>>> lucene.initVM(export=True)
<jcc.JCCEnv object at 0x7f405de450d8>
>>> import bobobrowse
>>> bobobrowse.initVM()
<jcc.JCCEnv object at 0x7f405de451c8>


Issues
======

- I have to follow a strict order (first import lucene, than 
lucene.initVM(export=True), finally import bobobrowse). When I try to 
import bobobrowse before the call to initVM(export=True), the import 
fails with unresolved symbols. Have you tried to add _lucene.so as a 
dependency to _bobobrowse.so? I believe a dependency should make the 
dlopen() call obsolete.

$ ldd 
/usr/local/lib/python2.6/dist-packages/bobobrowse-2.5.0_rc1-py2.6-linux-x86_64.egg/bobobrowse/_bobobrowse.so
         linux-vdso.so.1 =>  (0x00007fffc37ff000)
         libjcc.so => 
/usr/local/lib/python2.6/dist-packages/JCC-2.6-py2.6-linux-x86_64.egg/libjcc.so 
(0x00007fbbdc9f8000)
         libjava.so => /usr/lib/jvm/java-6-sun/jre/lib/amd64/libjava.so 
(0x00007fbbdc8c8000)
         libjvm.so => 
/usr/lib/jvm/java-6-sun/jre/lib/amd64/server/libjvm.so (0x00007fbbdbe50000)
         libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fbbdbb35000)
         libm.so.6 => /lib/libm.so.6 (0x00007fbbdb8b1000)
         libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007fbbdb69a000)
         libpthread.so.0 => /lib/libpthread.so.0 (0x00007fbbdb47d000)
         libc.so.6 => /lib/libc.so.6 (0x00007fbbdb0fa000)
         libpython2.6.so.1.0 => /usr/lib/libpython2.6.so.1.0 
(0x00007fbbdac49000)
         libverify.so => 
/usr/lib/jvm/java-6-sun/jre/lib/amd64/libverify.so (0x00007fbbdab3a000)
         libnsl.so.1 => /lib/libnsl.so.1 (0x00007fbbda91f000)
         libdl.so.2 => /lib/libdl.so.2 (0x00007fbbda71b000)
         /lib64/ld-linux-x86-64.so.2 (0x00007fbbdd1af000)
         libssl.so.0.9.8 => /lib/libssl.so.0.9.8 (0x00007fbbda4cb000)
         libcrypto.so.0.9.8 => /lib/libcrypto.so.0.9.8 (0x00007fbbda13a000)
         libz.so.1 => /lib/libz.so.1 (0x00007fbbd9f23000)
         libutil.so.1 => /lib/libutil.so.1 (0x00007fbbd9d20000)


- With "--import lucene" JCC gets the class_ attribute of all types 
wrong, that are part of the bobobrowse package. It either shows 
java.lang.Object as class_ or the first lucene class in the hierarchy. 
Without "--import lucene" it works as expected.

>>> bobobrowse.FacetSpec.class_
<Class: class java.lang.Object>

BoboBrowser is a subclass of MultiBoboBrowser which extends lucene's 
MultiSearcher
>>> bobobrowse.MultiBoboBrowser.class_
<Class: class org.apache.lucene.search.MultiSearcher>

>>> bobobrowse.BoboBrowser.class_
<Class: class org.apache.lucene.search.MultiSearcher>

>>> bobobrowse.Tokenizer.class_
<Class: class org.apache.lucene.analysis.Tokenizer>


- With "--import lucene" JCC doesn't wrap certain classes and it's 
missing bobobrowse.FacetSpec.FacetSortSpec.OrderHitsDesc. Without 
"--import lucene" the enum is wrapped correctly.

>>> bobobrowse.FacetSpec.FacetSortSpec.OrderHitsDesc
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
AttributeError: type object 'FacetSpec$FacetSortSpec' has no attribute 
'OrderHitsDesc'


- I've run into a rather strange bug with our custom builds of Python 
2.6. It differs from the stock build of Ubuntu in two ways. The 
python2.6 binary is linked against libpython2.6.so (Ubuntu's python2.6 
binary has libpython2.6.a statically linked in) and I've set the 
environment variable LD_RUN_PATH which causes the linker to add a 
"library rpath" to all ELF binaries. I wasn't able to find the reason 
for the issue so far. Any hints are appreciated.

Christian

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Sun, 16 May 2010, Andi Vajda wrote:

>
> On Sun, 16 May 2010, Andi Vajda wrote:
>
>> 
>> On Sun, 16 May 2010, Andi Vajda wrote:
>> 
>>> On Linux, there's got to be a way to make this work but I haven't found 
>>> how yet. I seem to be having trouble with linking to the Python Type 
>>> objects. On Windows, I expect this to be even more difficult.
>> 
>> Adding a call to dlopen("_lucene.so", RTLD_NOW | RTLD_GLOBAL) solves the 
>> problem on Linux. I'm thinking of adding this to the initVM() call with an 
>> optional 'export=true/false' parameter... unless there is a better way to 
>> go about this.
>
> This is now implemented and checked into the branch_3x branch.

And it now also works on Windows...

Andi..

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Sun, 16 May 2010, Andi Vajda wrote:

>
> On Sun, 16 May 2010, Andi Vajda wrote:
>
>> On Linux, there's got to be a way to make this work but I haven't found how 
>> yet. I seem to be having trouble with linking to the Python Type objects. 
>> On Windows, I expect this to be even more difficult.
>
> Adding a call to dlopen("_lucene.so", RTLD_NOW | RTLD_GLOBAL) solves the 
> problem on Linux. I'm thinking of adding this to the initVM() call with an 
> optional 'export=true/false' parameter... unless there is a better way to go 
> about this.

This is now implemented and checked into the branch_3x branch.

Andi..

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Sun, 16 May 2010, Andi Vajda wrote:

> On Linux, there's got to be a way to make this work but I haven't found how 
> yet. I seem to be having trouble with linking to the Python Type objects. On 
> Windows, I expect this to be even more difficult.

Adding a call to dlopen("_lucene.so", RTLD_NOW | RTLD_GLOBAL) solves the 
problem on Linux. I'm thinking of adding this to the initVM() call with an 
optional 'export=true/false' parameter... unless there is a better way to go 
about this.

Andi..

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Thu, 13 May 2010, Andi Vajda wrote:

> On Fri, 14 May 2010, Christian Heimes wrote:
>
>> Thanks for the suggestion, Andi! I'll try your suggestion on Monday. I hope 
>> it's sufficient to create a working version of lucene + bobo-browse in 
>> order to provide my co-worker Julian a foundation for his work on 
>> bobo-browse. But I fear that it's not going to work. I carefully excluded 
>> any jar file from the bobobrowse egg that is already part of pylucene 
>> (--classpath instead of --include or --jar). The lucene class are probably 
>> wrapped because bobo-browse references the classes in its jar file.
>> 
>> By the way I've created the Makefile for bobo-browse based on your Makefile 
>> for pylucene. With your template and your good work it was trivial to 
>> create the Python bindings. Thanks ;)
>> 
>> I've an idea how you can solve the inter-package issue. Is it possible to 
>> create some kind of type interference for JCC types that point to the same 
>> Java class (e.g. lucene.Query.class_ == bobobrowse.Query.class_)?
>
> Ah, I see now the bigger problem. What you're really after in the case where 
> bobo would cause its own lucene wrappers to be generated is the ability to 
> import them from the lucene extension instead. For this, the symbols would 
> need to be exported and the C++ headers generated by jcc during the pylucene 
> build included in the lucene egg. I think this could be done. Give me a few 
> days to think about this... say via a new --import jcc commmand line option.

I added support for a new jcc command line option called --import which 
accepts the name of another jcc-built extension such as 'lucene'.

This is checked into the new branch_3x pylucene branch [1] which mirrors the 
new lucene branch_3x branch that reflects the recent changes in the lucene 
svn tree with regards to backwards compatibility changes.

The new JCC in that branch now installs all header files it generates for a 
--shared extension into the extension's egg under an 'include' directory.
Therefore you must rebuild pylucene with this latest JCC first to try this 
out.

When an extension is imported via jcc's --import command line flag, its 
header files are scanned so that the importing extension reuses the code 
from the imported extension instead of generating wrappers for these classes 
again.

This works quite well on Mac OS X where the dynamic linker finds the 
code objects (methods) for the imported classes for the imported extension 
at runtime without any problems provided the jcc-imported extensions are 
imported into python first.

On Linux, there's got to be a way to make this work but I haven't found how 
yet. I seem to be having trouble with linking to the Python Type objects. On 
Windows, I expect this to be even more difficult.

I don't know what operating system you're on. If you're on Mac OS X, please 
try this out and let me know if it solves your problem. I intend to work on 
the required Linux linker tricks next.

I created a simple test case that extends Lucene's Analyzer class but in 
another extension that is built by thus importing lucene. The command line 
to build it looks like this:
python -m jcc.__main__ --classpath `pwd` --python foo --import lucene bar.SillyAnalyzer --shared --build --install

Andi..

[1] http://svn.apache.org/repos/asf/lucene/pylucene/branches/branch_3x/

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Fri, 14 May 2010, Christian Heimes wrote:

> Am 13.05.2010 20:27, schrieb Andi Vajda:
>> Here is an idea that I think should work but I couldn't test it because 
>> bobo
>> is built from maven and pylucene isn't.
>> 
>> Basically, the idea is that you must be really careful in not having Lucene
>> classes come from two or more different places:
>>    1. build pylucene the usual way
>>    2. build bobo by giving it the lucene jar files that are inside 
>> pylucene's
>>       lucene egg by setting the classpath accordingly:
>>       $ CLASSPATH=`python -c 'import lucene; print lucene.CLASSPATH'`
>>    3. build the bobo extension with jcc using the same classpath and by 
>> _not_
>>       using --jar or --include with any of the lucene jar files already 
>> inside
>>       pylucene's lucene egg, relying solely on the classpath for finding
>>       lucene classes
>>    4. Once all is built, you don't need to set any classpath in addition to
>>       what the initVM() calls are already doing:
>>       >>>  import lucene; lucene.initVM()
>>       >>>  import bobo; bobo.initVM()
>>       The bobo extension adds its classpath to what lucene's already setup
>>       and finds the lucene classes there.
>> 
>> Again, I didn't test this because it's too much work to disentangle the 
>> bobo
>> build from maven so that I can control where things are coming from.
>> You understand the bobo build better and can probably try this quicker.
>
> Thanks for the suggestion, Andi! I'll try your suggestion on Monday. I hope 
> it's sufficient to create a working version of lucene + bobo-browse in order 
> to provide my co-worker Julian a foundation for his work on bobo-browse. But 
> I fear that it's not going to work. I carefully excluded any jar file from 
> the bobobrowse egg that is already part of pylucene (--classpath instead of 
> --include or --jar). The lucene class are probably wrapped because 
> bobo-browse references the classes in its jar file.
>
> By the way I've created the Makefile for bobo-browse based on your Makefile 
> for pylucene. With your template and your good work it was trivial to create 
> the Python bindings. Thanks ;)
>
> I've an idea how you can solve the inter-package issue. Is it possible to 
> create some kind of type interference for JCC types that point to the same 
> Java class (e.g. lucene.Query.class_ == bobobrowse.Query.class_)?

Ah, I see now the bigger problem. What you're really after in the case where 
bobo would cause its own lucene wrappers to be generated is the ability to 
import them from the lucene extension instead. For this, the symbols would 
need to be exported and the C++ headers generated by jcc during the pylucene 
build included in the lucene egg. I think this could be done. Give me a few 
days to think about this... say via a new --import jcc commmand line option.

Andi..

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Christian Heimes <li...@cheimes.de>.
Am 13.05.2010 20:27, schrieb Andi Vajda:
> Here is an idea that I think should work but I couldn't test it because bobo
> is built from maven and pylucene isn't.
>
> Basically, the idea is that you must be really careful in not having Lucene
> classes come from two or more different places:
>    1. build pylucene the usual way
>    2. build bobo by giving it the lucene jar files that are inside pylucene's
>       lucene egg by setting the classpath accordingly:
>       $ CLASSPATH=`python -c 'import lucene; print lucene.CLASSPATH'`
>    3. build the bobo extension with jcc using the same classpath and by _not_
>       using --jar or --include with any of the lucene jar files already inside
>       pylucene's lucene egg, relying solely on the classpath for finding
>       lucene classes
>    4. Once all is built, you don't need to set any classpath in addition to
>       what the initVM() calls are already doing:
>       >>>  import lucene; lucene.initVM()
>       >>>  import bobo; bobo.initVM()
>       The bobo extension adds its classpath to what lucene's already setup
>       and finds the lucene classes there.
>
> Again, I didn't test this because it's too much work to disentangle the bobo
> build from maven so that I can control where things are coming from.
> You understand the bobo build better and can probably try this quicker.

Thanks for the suggestion, Andi! I'll try your suggestion on Monday. I 
hope it's sufficient to create a working version of lucene + bobo-browse 
in order to provide my co-worker Julian a foundation for his work on 
bobo-browse. But I fear that it's not going to work. I carefully 
excluded any jar file from the bobobrowse egg that is already part of 
pylucene (--classpath instead of --include or --jar). The lucene class 
are probably wrapped because bobo-browse references the classes in its 
jar file.

By the way I've created the Makefile for bobo-browse based on your 
Makefile for pylucene. With your template and your good work it was 
trivial to create the Python bindings. Thanks ;)

I've an idea how you can solve the inter-package issue. Is it possible 
to create some kind of type interference for JCC types that point to the 
same Java class (e.g. lucene.Query.class_ == bobobrowse.Query.class_)?

Christian

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Wed, 12 May 2010, Andi Vajda wrote:

>> Christian Heimes, Dirk Rothe, and I have jcc-wrapped bobo-browse 
>> (http://code.google.com/p/bobo-browse/) in order to add faceted search 
>> capabilities to PyLucene. However, the two modules don't play well 
>> together, as wrappers from PyLucene cannot be used in a bobo-browse context 
>> and vice versa.
>> What is the best way to get classes from two different jcc python extension 
>> modules to interact?
>
> I've noticed that when there is overlap in Java classes between two JCC-built 
> extensions, these kinds of errors occur. One can imagine that if the classes 
> have the same name but are not from the same codebase that funny version 
> conflicts might arise, for example.
>
> But I haven't looked into exactly what the problem is but a simple workaround 
> is to just create one extension that combines both Lucene and Bobo-browse. 
> Take the JCC invocation for PyLucene from its Makefile and add the things 
> would want from Bobo-browse you want from it and you should have something 
> that works better.

Here is an idea that I think should work but I couldn't test it because bobo 
is built from maven and pylucene isn't.

Basically, the idea is that you must be really careful in not having Lucene 
classes come from two or more different places:
  1. build pylucene the usual way
  2. build bobo by giving it the lucene jar files that are inside pylucene's
     lucene egg by setting the classpath accordingly:
     $ CLASSPATH=`python -c 'import lucene; print lucene.CLASSPATH'`
  3. build the bobo extension with jcc using the same classpath and by _not_
     using --jar or --include with any of the lucene jar files already inside
     pylucene's lucene egg, relying solely on the classpath for finding
     lucene classes
  4. Once all is built, you don't need to set any classpath in addition to
     what the initVM() calls are already doing:
     >>> import lucene; lucene.initVM()
     >>> import bobo; bobo.initVM()
     The bobo extension adds its classpath to what lucene's already setup
     and finds the lucene classes there.

Again, I didn't test this because it's too much work to disentangle the bobo 
build from maven so that I can control where things are coming from.
You understand the bobo build better and can probably try this quicker.

Please let me know if this works for you.
Thanks !

Andi..

Re: Problems passing PyLucene objects to jcc-wrapped bobo-browse api

Posted by Andi Vajda <va...@apache.org>.
On Wed, 12 May 2010, Julian Maibaum wrote:

> Christian Heimes, Dirk Rothe, and I have jcc-wrapped bobo-browse 
> (http://code.google.com/p/bobo-browse/) in order to add faceted search 
> capabilities to PyLucene. However, the two modules don't play well together, 
> as wrappers from PyLucene cannot be used in a bobo-browse context and vice 
> versa.
> What is the best way to get classes from two different jcc python extension 
> modules to interact?

I've noticed that when there is overlap in Java classes between two 
JCC-built extensions, these kinds of errors occur. One can imagine that if 
the classes have the same name but are not from the same codebase that funny 
version conflicts might arise, for example.

But I haven't looked into exactly what the problem is but a simple 
workaround is to just create one extension that combines both Lucene and 
Bobo-browse. Take the JCC invocation for PyLucene from its Makefile and add 
the things would want from Bobo-browse you want from it and you should have 
something that works better.

Andi..

>
> Setup:
> X64 Ubuntu 9.10 Python 2.6 Tried both versions:
>  JCC-2.4.1-py2.6-linux-x86_64.egg compiled with --shared
>  JCC-2.5.1-py2.6-linux-x86_64.egg compiled with --shared
> Tried both versions:
>  lucene-2.9.1-py2.6-linux-x86_64.egg
>  lucene-2.9.2-py2.6-linux-x86_64.egg
> bobobrowse-2.5.0_rc1-py2.6-linux-x86_64.egg
> Both the bobo-browse python extension module and PyLucene are attached to the 
> main python thread and have been initialized using their respective initVM 
> methods.
>
> I have attached the makefile (modified from PyLucene) that was used to build 
> bobo-browse.
>
> #---- START CODE ----
> import bobobrowse as bb
> import lucene
> import os
>
> CLASSPATH = os.pathsep.join((lucene.CLASSPATH, bb.CLASSPATH)
> lucene.initVM(CLASSPATH)
> bb.initVM(CLASSPATH)
>
> typeHandler = bb.MultiValueFacetHandler("type")
> placeHandler = bb.MultiValueFacetHandler("id")
>
> facetHandlers = lucene.JArray('object')(2, bb.FacetHandler)
> facetHandlers[0] = typeHandler
> facetHandlers[1] = placeHandler
>
> reader = bb.IndexReader.open('/path/to/my/lucene/index', True)
> facetHandlers = bb.Arrays.asList(facetHandlers)
> indexReader = bb.BoboIndexReader.getInstance(reader, facetHandlers)
>
> #---- END CODE ----
>
> Expected:
> The example code above works - java objects from classes wrapped as part of 
> the PyLucene module can transparently be used by other python extension 
> modules such as bobo-browse.
>
> Experienced:
>  Throws Errors:
> Traceback (most recent call last):
> File "./dynamic/vls/search/facets.py", line 115, in <module>
>   test()
> File "./dynamic/smc/jcc/__init__.py", line 52, in wrapper
>   return func(*args, **kwargs)
> File "./dynamic/vls/search/facets.py", line 57, in test
>   facetHandlers = lucene.JArray('object')(2, bb.FacetHandler)
> ValueError: <type 'FacetHandler'>
>
>  and
>
> bobobrowse.InvalidArgsError: (<type 'BoboIndexReader'>, 'getInstance', 
> (<IndexReader: org.apache.lucene.index.DirectoryOwningReader@30419d05>, 
> <List: [com.browseengine.bobo.facets.impl.MultiValueFacetHandler@3f829e6f, 
> com.browseengine.bobo.facets.impl.MultiValueFacetHandler@2980f96c]>))
>
>  respectively.
>
> The Errors can be avoided by changing the corresponding lines in the example 
> above to:
> facetHandlers = bb.JArray('object')(2, bb.FacetHandler)
> reader = bb.IndexReader.open('/path/to/my/lucene/index', True)
>
> However, this only works because lucene's IndexReader is referenced by 
> bobo-browse and has therefore been wrapped at build-time. At later stages 
> other lucene classes are needed to work with bobobrowse that no wrappers have 
> been built for (e.g. QueryParser). I understand that jcc can be forced to 
> built and include wrappers for e.g. org.apache.lucene.queryParser.QueryParser 
> that are then accessible for use with classes from the bobo-browse module as 
> e.g 'bobobrowse.QueryParser'. The most simple solution is calling jcc with an 
> additional --jar lucene.jar parameter - in effect leading to lucene being 
> installed twice: First self-contained PyLucene module and, second, as part of 
> bobo-browse in the flat bobo-browse module namespace. This is not quite 
> desirable:
> Since
> bb.IndexReader.class_ == lucene.IndexReader.class_
> --> class org.apache.lucene.index.IndexReader, anyways, it would arguably be 
> more intuitive to be able to use the available wrappers in the context of 
> other jcc modules even if they have not specifically been forced into that 
> other module. Is there any way to obtain the desired behavior?
>
>
> Thank you very much and all the best,
>
> Julian Maibaum
>
>
> j.maibaum(at)semantics.de
> s<e>mantics GmbH
> http://www.semantics.de
>