You are viewing a plain text version of this content. The canonical link for it is here.
Posted to pylucene-dev@lucene.apache.org by Eduard Rozenberg <ed...@pobox.com> on 2014/05/27 23:35:51 UTC
PyLucene 4.8.0 - 'make install' with 'root=' missing a couple of files
Hello folks,
I’m working on packaging PyLucene to a Slackware
package by using a setup.cfg in the source directory
and redirecting the installation root to
/tmp/pylucene_installdir.
I noticed that a couple of files are missing when doing
this alternate root install compared to the regular install.
########## setup.cfg ###########
[easy_install]
[build]
[install]
root = /tmp/pylucene_installdir
compile = False
force = True
single-version-externally-managed = True
##############################
I noticed that two files are missing when I do this root=
install compared to the regular install to /usr/lib…/
Are these two files below not necessary when
packaging PyLucene for distribution?
Missing: native_libs.txt
------------------------------
Contains:
lucene/_lucene.so
Missing: _lucene.py
--------------------------
Contains:
def __bootstrap__():
global __bootstrap__, __loader__, __file__
import sys, pkg_resources, imp
__file__ = pkg_resources.resource_filename(__name__, '_lucene.so')
__loader__ = None; del __bootstrap__, __loader__
imp.load_dynamic(__name__,__file__)
__bootstrap__()
Thanks in advance!
Regards,
—Ed
Re: PyLucene 4.8.0 - samples/FacetExample.py appears broken
Posted by Andi Vajda <va...@apache.org>.
On Mon, 2 Jun 2014, Thomas Koch wrote:
> Am 30.05.2014 um 23:08 schrieb Andi Vajda <va...@apache.org>:
>
>> Thanks Thomas !
>> Once you do that maybe it's time to make a 4.8.1 release too.
>> Let me know when you're ready with a fix.
>
> TLDR; fix is available: I've now revised the FacetExample.py to use the new API - it's tested to work with pylucene-4.8.0-1
> The source can be found here: https://dl.dropboxusercontent.com/u/4384120/Python/FacetExample.py
> Andi, can you please update the file in the repo? thanks.
Done, it's in trunk revision 1599384.
Thank you !
ANdi..
>
> details: I should correct myself. The FacetExample was 'ported' to 4.x (with new fully qualified imports) already.
> However in Java Lucene Release 4.7.0 [2014-02-26] some API Changes did break the code:
> LUCENE-5339: The facet module was simplified/reworked to make the APIs more approachable to new users.
>
> Note: when migrating to the new API, you must pass the Document that is returned
> from FacetConfig.build() to IndexWriter.addDocument().
> (Shai Erera, Gilad Barkai, Rob Muir, Mike McCandless)
> see https://issues.apache.org/jira/browse/LUCENE-5339 for details!
> (This was really a good and needed simplification of the Facet API!)
>
> Some notes for those interested in facet search in Lucene:
>
> - there are some Java samples available with Lucene Java, e.g.
> http://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_4_8/lucene/demo/src/java/org/apache/lucene/demo/facet/SimpleFacetsExample.java
> (which I took as a reference)
>
> - back in 4.4 there was some good documentation (besides Javadocs) - the "Apache Lucene Faceted Search "User's Guide
> http://lucene.apache.org/core/4_4_0/facet/org/apache/lucene/facet/doc-files/userguide.html
>
> I can't find it for more recent versions though and according to Changelog it was removed with 4.5 because it's outdated:
> LUCENE-4894: remove facet userguide as it was outdated. Partially absorbed into
> package's documentation and classes javadocs.
> (Shai Erera)
> (still think it's partially useful introductory reading though...)
>
> - developer hint:
>
> whereas in Java you can write
> doc.add(new FacetField("Publish Date", "2010", "10", "20"));
> here "Publish Date" is the 'dimension'
> and "2010", "10", "20" is the 'path'
>
> in Python you need to declare the path as an array argument:
> doc.add(FacetField("Publish Date", ["2010", "10", "20"]))
>
>
> Java introduced 'varargs' feature to pass unspecified number of argument to a method in JDK 5.0.
> Python can do similar by using the *args which can be passed a list of arguments - as positional arguments.
> Seems however JCC does for some (probably good) reason not map Java varargs to python *args.
>
>
> regards,
> Thomas
>
Re: PyLucene 4.8.0 - samples/FacetExample.py appears broken
Posted by Andi Vajda <va...@apache.org>.
On Mon, 2 Jun 2014, Andi Vajda wrote:
>
>> On Jun 2, 2014, at 2:04, Thomas Koch <ko...@orbiteam.de> wrote:
>>
>>> Am 30.05.2014 um 23:08 schrieb Andi Vajda <va...@apache.org>:
>>>
>>> Thanks Thomas !
>>> Once you do that maybe it's time to make a 4.8.1 release too.
>>> Let me know when you're ready with a fix.
>>
>> TLDR; fix is available: I've now revised the FacetExample.py to use the new API - it's tested to work with pylucene-4.8.0-1
>> The source can be found here: https://dl.dropboxusercontent.com/u/4384120/Python/FacetExample.py
>> Andi, can you please update the file in the repo? thanks.
>>
>> details: I should correct myself. The FacetExample was 'ported' to 4.x (with new fully qualified imports) already.
>> However in Java Lucene Release 4.7.0 [2014-02-26] some API Changes did break the code:
>> LUCENE-5339: The facet module was simplified/reworked to make the APIs more approachable to new users.
>>
>> Note: when migrating to the new API, you must pass the Document that is returned
>> from FacetConfig.build() to IndexWriter.addDocument().
>> (Shai Erera, Gilad Barkai, Rob Muir, Mike McCandless)
>> see https://issues.apache.org/jira/browse/LUCENE-5339 for details!
>> (This was really a good and needed simplification of the Facet API!)
>>
>> Some notes for those interested in facet search in Lucene:
>>
>> - there are some Java samples available with Lucene Java, e.g.
>> http://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_4_8/lucene/demo/src/java/org/apache/lucene/demo/facet/SimpleFacetsExample.java
>> (which I took as a reference)
>>
>> - back in 4.4 there was some good documentation (besides Javadocs) - the "Apache Lucene Faceted Search "User's Guide
>> http://lucene.apache.org/core/4_4_0/facet/org/apache/lucene/facet/doc-files/userguide.html
>>
>> I can't find it for more recent versions though and according to Changelog it was removed with 4.5 because it's outdated:
>> LUCENE-4894: remove facet userguide as it was outdated. Partially absorbed into
>> package's documentation and classes javadocs.
>> (Shai Erera)
>> (still think it's partially useful introductory reading though...)
>>
>> - developer hint:
>>
>> whereas in Java you can write
>> doc.add(new FacetField("Publish Date", "2010", "10", "20"));
>> here "Publish Date" is the 'dimension'
>> and "2010", "10", "20" is the 'path'
>>
>> in Python you need to declare the path as an array argument:
>> doc.add(FacetField("Publish Date", ["2010", "10", "20"]))
>>
>>
>> Java introduced 'varargs' feature to pass unspecified number of argument to a method in JDK 5.0.
>> Python can do similar by using the *args which can be passed a list of arguments - as positional arguments.
>> Seems however JCC does for some (probably good) reason not map Java varargs to python *args.
>
> Thank you, Thomas, for the fix and explanations. There is no reason for
> JCC not to support Java's varargs feature other than it hasn't been
> implemented so far.
I've now done a first pass at adding support for Java varargs in JCC by
reconstructing the last-args array the Java method actually expects during
Python->Java argument parsing.
This is checked into svn trunk rev 1599407.
I tested this new feature with FacetExample.py. I removed all 'unnecessary'
arrays and the sample still works !
This could use a bit of testing...
Andi..
Re: PyLucene 4.8.0 - samples/FacetExample.py appears broken
Posted by Andi Vajda <va...@apache.org>.
> On Jun 2, 2014, at 2:04, Thomas Koch <ko...@orbiteam.de> wrote:
>
>> Am 30.05.2014 um 23:08 schrieb Andi Vajda <va...@apache.org>:
>>
>> Thanks Thomas !
>> Once you do that maybe it's time to make a 4.8.1 release too.
>> Let me know when you're ready with a fix.
>
> TLDR; fix is available: I've now revised the FacetExample.py to use the new API - it's tested to work with pylucene-4.8.0-1
> The source can be found here: https://dl.dropboxusercontent.com/u/4384120/Python/FacetExample.py
> Andi, can you please update the file in the repo? thanks.
>
> details: I should correct myself. The FacetExample was 'ported' to 4.x (with new fully qualified imports) already.
> However in Java Lucene Release 4.7.0 [2014-02-26] some API Changes did break the code:
> LUCENE-5339: The facet module was simplified/reworked to make the APIs more approachable to new users.
>
> Note: when migrating to the new API, you must pass the Document that is returned
> from FacetConfig.build() to IndexWriter.addDocument().
> (Shai Erera, Gilad Barkai, Rob Muir, Mike McCandless)
> see https://issues.apache.org/jira/browse/LUCENE-5339 for details!
> (This was really a good and needed simplification of the Facet API!)
>
> Some notes for those interested in facet search in Lucene:
>
> - there are some Java samples available with Lucene Java, e.g.
> http://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_4_8/lucene/demo/src/java/org/apache/lucene/demo/facet/SimpleFacetsExample.java
> (which I took as a reference)
>
> - back in 4.4 there was some good documentation (besides Javadocs) - the "Apache Lucene Faceted Search "User's Guide
> http://lucene.apache.org/core/4_4_0/facet/org/apache/lucene/facet/doc-files/userguide.html
>
> I can't find it for more recent versions though and according to Changelog it was removed with 4.5 because it's outdated:
> LUCENE-4894: remove facet userguide as it was outdated. Partially absorbed into
> package's documentation and classes javadocs.
> (Shai Erera)
> (still think it's partially useful introductory reading though...)
>
> - developer hint:
>
> whereas in Java you can write
> doc.add(new FacetField("Publish Date", "2010", "10", "20"));
> here "Publish Date" is the 'dimension'
> and "2010", "10", "20" is the 'path'
>
> in Python you need to declare the path as an array argument:
> doc.add(FacetField("Publish Date", ["2010", "10", "20"]))
>
>
> Java introduced 'varargs' feature to pass unspecified number of argument to a method in JDK 5.0.
> Python can do similar by using the *args which can be passed a list of arguments - as positional arguments.
> Seems however JCC does for some (probably good) reason not map Java varargs to python *args.
Thank you, Thomas, for the fix and explanations. There is no reason for JCC not to support Java's varargs feature other than it hasn't been implemented so far.
Andi..
>
>
> regards,
> Thomas
Re: PyLucene 4.8.0 - samples/FacetExample.py appears broken
Posted by Eduard Rozenberg <ed...@pobox.com>.
Hello Thomas,
Thanks for the update on the facets! I just tested this
with my PyLucene 4.8.0 and can confirm it works great:
Indexed 2 documents with facets.
search by term 'white' ...
Categories: 1
'root' (2)
Root-a-Categories: 3
'f1' (2)
'f2' (1)
'f3' (1)
Author: 2
'Bob' (1)
'Lisa' (1)
search by term 'car' ...
Categories: 1
'root' (1)
Root-a-Categories: 2
'f1' (1)
'f2' (1)
Author: 1
'Bob' (1)
search all documents ...
Categories: 1
'root' (2)
Root-a-Categories: 3
'f1' (2)
'f2' (1)
'f3' (1)
Author: 2
'Bob' (1)
'Lisa' (1)
Best regards,
—Ed
On Jun 2, 2014, at 8:04, Thomas Koch <ko...@orbiteam.de> wrote:
> Am 30.05.2014 um 23:08 schrieb Andi Vajda <va...@apache.org>:
>
>> Thanks Thomas !
>> Once you do that maybe it's time to make a 4.8.1 release too.
>> Let me know when you're ready with a fix.
>
> TLDR; fix is available: I've now revised the FacetExample.py to use the new API - it's tested to work with pylucene-4.8.0-1
> The source can be found here: https://dl.dropboxusercontent.com/u/4384120/Python/FacetExample.py
> Andi, can you please update the file in the repo? thanks.
>
> details: I should correct myself. The FacetExample was 'ported' to 4.x (with new fully qualified imports) already.
> However in Java Lucene Release 4.7.0 [2014-02-26] some API Changes did break the code:
> LUCENE-5339: The facet module was simplified/reworked to make the APIs more approachable to new users.
>
> Note: when migrating to the new API, you must pass the Document that is returned
> from FacetConfig.build() to IndexWriter.addDocument().
> (Shai Erera, Gilad Barkai, Rob Muir, Mike McCandless)
> see https://issues.apache.org/jira/browse/LUCENE-5339 for details!
> (This was really a good and needed simplification of the Facet API!)
>
> Some notes for those interested in facet search in Lucene:
>
> - there are some Java samples available with Lucene Java, e.g.
> http://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_4_8/lucene/demo/src/java/org/apache/lucene/demo/facet/SimpleFacetsExample.java
> (which I took as a reference)
>
> - back in 4.4 there was some good documentation (besides Javadocs) - the "Apache Lucene Faceted Search "User's Guide
> http://lucene.apache.org/core/4_4_0/facet/org/apache/lucene/facet/doc-files/userguide.html
>
> I can't find it for more recent versions though and according to Changelog it was removed with 4.5 because it's outdated:
> LUCENE-4894: remove facet userguide as it was outdated. Partially absorbed into
> package's documentation and classes javadocs.
> (Shai Erera)
> (still think it's partially useful introductory reading though...)
>
> - developer hint:
>
> whereas in Java you can write
> doc.add(new FacetField("Publish Date", "2010", "10", "20"));
> here "Publish Date" is the 'dimension'
> and "2010", "10", "20" is the 'path'
>
> in Python you need to declare the path as an array argument:
> doc.add(FacetField("Publish Date", ["2010", "10", "20"]))
>
>
> Java introduced 'varargs' feature to pass unspecified number of argument to a method in JDK 5.0.
> Python can do similar by using the *args which can be passed a list of arguments - as positional arguments.
> Seems however JCC does for some (probably good) reason not map Java varargs to python *args.
>
>
> regards,
> Thomas
Re: PyLucene 4.8.0 - samples/FacetExample.py appears broken
Posted by Thomas Koch <ko...@orbiteam.de>.
Am 30.05.2014 um 23:08 schrieb Andi Vajda <va...@apache.org>:
> Thanks Thomas !
> Once you do that maybe it's time to make a 4.8.1 release too.
> Let me know when you're ready with a fix.
TLDR; fix is available: I've now revised the FacetExample.py to use the new API - it's tested to work with pylucene-4.8.0-1
The source can be found here: https://dl.dropboxusercontent.com/u/4384120/Python/FacetExample.py
Andi, can you please update the file in the repo? thanks.
details: I should correct myself. The FacetExample was 'ported' to 4.x (with new fully qualified imports) already.
However in Java Lucene Release 4.7.0 [2014-02-26] some API Changes did break the code:
LUCENE-5339: The facet module was simplified/reworked to make the APIs more approachable to new users.
Note: when migrating to the new API, you must pass the Document that is returned
from FacetConfig.build() to IndexWriter.addDocument().
(Shai Erera, Gilad Barkai, Rob Muir, Mike McCandless)
see https://issues.apache.org/jira/browse/LUCENE-5339 for details!
(This was really a good and needed simplification of the Facet API!)
Some notes for those interested in facet search in Lucene:
- there are some Java samples available with Lucene Java, e.g.
http://svn.apache.org/repos/asf/lucene/dev/branches/lucene_solr_4_8/lucene/demo/src/java/org/apache/lucene/demo/facet/SimpleFacetsExample.java
(which I took as a reference)
- back in 4.4 there was some good documentation (besides Javadocs) - the "Apache Lucene Faceted Search "User's Guide
http://lucene.apache.org/core/4_4_0/facet/org/apache/lucene/facet/doc-files/userguide.html
I can't find it for more recent versions though and according to Changelog it was removed with 4.5 because it's outdated:
LUCENE-4894: remove facet userguide as it was outdated. Partially absorbed into
package's documentation and classes javadocs.
(Shai Erera)
(still think it's partially useful introductory reading though...)
- developer hint:
whereas in Java you can write
doc.add(new FacetField("Publish Date", "2010", "10", "20"));
here "Publish Date" is the 'dimension'
and "2010", "10", "20" is the 'path'
in Python you need to declare the path as an array argument:
doc.add(FacetField("Publish Date", ["2010", "10", "20"]))
Java introduced 'varargs' feature to pass unspecified number of argument to a method in JDK 5.0.
Python can do similar by using the *args which can be passed a list of arguments - as positional arguments.
Seems however JCC does for some (probably good) reason not map Java varargs to python *args.
regards,
Thomas
Re: PyLucene 4.8.0 - samples/FacetExample.py appears broken
Posted by Andi Vajda <va...@apache.org>.
> On May 30, 2014, at 10:58, Thomas Koch <ko...@orbiteam.de> wrote:
>
> Hello Ed,
> yes this is a 'known issue‘ - I contributed the FacetExample based on PyLucene 3.6.x. Meanwhile Lucene Facet API has changed and I shall adapt the example.
>
> regards,
Thanks Thomas !
Once you do that maybe it's time to make a 4.8.1 release too.
Let me know when you're ready with a fix.
Andi..
> Thomas
> --
>> Am 30.05.2014 um 18:09 schrieb Eduard Rozenberg <ed...@pobox.com>:
>>
>> Hello,
>>
>> I’m getting an error running this sample, I believe it is related to an old issue of Facet API having changed -
>> http://mail-archives.apache.org/mod_mbox/lucene-pylucene-dev/201302.mbox/%3calpine.OSX.2.01.1302101349450.6538@yuzu.local%3e
>>
>> $ python FacetExample.py
>>
>> Traceback (most recent call last):
>> File "FacetExample.py", line 51, in <module>
>> from org.apache.lucene.facet.index import FacetFields
>> ImportError: No module named index
>>
>> Regards,
>> —Ed
>
Re: PyLucene 4.8.0 - samples/FacetExample.py appears broken
Posted by Thomas Koch <ko...@orbiteam.de>.
Hello Ed,
yes this is a 'known issue‘ - I contributed the FacetExample based on PyLucene 3.6.x. Meanwhile Lucene Facet API has changed and I shall adapt the example.
regards,
Thomas
--
Am 30.05.2014 um 18:09 schrieb Eduard Rozenberg <ed...@pobox.com>:
> Hello,
>
> I’m getting an error running this sample, I believe it is related to an old issue of Facet API having changed -
> http://mail-archives.apache.org/mod_mbox/lucene-pylucene-dev/201302.mbox/%3calpine.OSX.2.01.1302101349450.6538@yuzu.local%3e
>
> $ python FacetExample.py
>
> Traceback (most recent call last):
> File "FacetExample.py", line 51, in <module>
> from org.apache.lucene.facet.index import FacetFields
> ImportError: No module named index
>
> Regards,
> —Ed
PyLucene 4.8.0 - samples/FacetExample.py appears broken
Posted by Eduard Rozenberg <ed...@pobox.com>.
Hello,
I’m getting an error running this sample, I believe it is related to an old issue of Facet API having changed -
http://mail-archives.apache.org/mod_mbox/lucene-pylucene-dev/201302.mbox/%3calpine.OSX.2.01.1302101349450.6538@yuzu.local%3e
$ python FacetExample.py
Traceback (most recent call last):
File "FacetExample.py", line 51, in <module>
from org.apache.lucene.facet.index import FacetFields
ImportError: No module named index
Regards,
—Ed
Re: PyLucene 4.8.0 - 'make install' with 'root=' missing a couple of files
Posted by Eduard Rozenberg <ed...@pobox.com>.
Hello Andi,
Thanks for your note. It took quite some days of effort but
I finally found a “golden path” to make a package out of
PyLucene.
I had to do ‘make’ and ‘make test’ without the setup.cfg
first. Otherwise 'make test' would dump a bunch of stuff
into my specified install root in a temporary directory and
then fail the tests entirely.
After ‘make’ and ‘make test’ succeed, I then put the
setup.cfg in the source folder and run ‘make install’ to
get the files to be copied to the alternate root specified
in the setup.cfg. Then I need to move and rename the
folders copied to the alternate root to match what the
install would normally do if it were allowed to install to
the usual root destination, because install apparently
behaves differently when given an alternate root. Then
I copy the missing _lucene.py and native_libs.txt files
from the source directory to the alternate root path.
Finally I created a doinst.sh script which uses sed to
add the proper path to easy-install.pth.
Then I take everything in the alternate root and make
a package out of it.
In case it helps anyone I put the various build and
script files in Dropbox, I’ll hopefully be submitting
them to slackbuilds.org sometime soon.
https://www.dropbox.com/sh/zsbw4uuhva2vy5h/AADKCmKyF-oOhKNTPiwppYELa
I think the reason it is so hard to package PyLucene
is that it is exceedingly free software, and doesn’t
like to be locked up inside of a package :).
Regards,
—Ed
On May 27, 2014, at 18:53, Andi Vajda <va...@apache.org> wrote:
>
> On Tue, 27 May 2014, Eduard Rozenberg wrote:
>
>> Hello folks,
>>
>> I?m working on packaging PyLucene to a Slackware
>> package by using a setup.cfg in the source directory
>> and redirecting the installation root to
>> /tmp/pylucene_installdir.
>>
>> I noticed that a couple of files are missing when doing
>> this alternate root install compared to the regular install.
>>
>> ########## setup.cfg ###########
>>
>> [easy_install]
>>
>> [build]
>>
>> [install]
>> root = /tmp/pylucene_installdir
>> compile = False
>> force = True
>> single-version-externally-managed = True
>>
>> ##############################
>>
>> I noticed that two files are missing when I do this root=
>> install compared to the regular install to /usr/lib?/
>> Are these two files below not necessary when
>> packaging PyLucene for distribution?
>>
>> Missing: native_libs.txt
>
> I don't know what this file, native_libs.txt, is for. Maybe a setuptools artifact ?
>
>> ------------------------------
>> Contains:
>> lucene/_lucene.so
>>
>>
>> Missing: _lucene.py
>
> Yes, that one you need.
> Did you try running pylucene tests without it ?
>
> Andi..
>
>> --------------------------
>> Contains:
>> def __bootstrap__():
>> global __bootstrap__, __loader__, __file__
>> import sys, pkg_resources, imp
>> __file__ = pkg_resources.resource_filename(__name__, '_lucene.so')
>> __loader__ = None; del __bootstrap__, __loader__
>> imp.load_dynamic(__name__,__file__)
>> __bootstrap__()
>>
>>
>> Thanks in advance!
>>
>> Regards,
>> ?Ed
Re: PyLucene 4.8.0 - 'make install' with 'root=' missing a couple
of files
Posted by Andi Vajda <va...@apache.org>.
On Tue, 27 May 2014, Eduard Rozenberg wrote:
> Hello folks,
>
> I?m working on packaging PyLucene to a Slackware
> package by using a setup.cfg in the source directory
> and redirecting the installation root to
> /tmp/pylucene_installdir.
>
> I noticed that a couple of files are missing when doing
> this alternate root install compared to the regular install.
>
> ########## setup.cfg ###########
>
> [easy_install]
>
> [build]
>
> [install]
> root = /tmp/pylucene_installdir
> compile = False
> force = True
> single-version-externally-managed = True
>
> ##############################
>
> I noticed that two files are missing when I do this root=
> install compared to the regular install to /usr/lib?/
> Are these two files below not necessary when
> packaging PyLucene for distribution?
>
> Missing: native_libs.txt
I don't know what this file, native_libs.txt, is for. Maybe a setuptools
artifact ?
> ------------------------------
> Contains:
> lucene/_lucene.so
>
>
> Missing: _lucene.py
Yes, that one you need.
Did you try running pylucene tests without it ?
Andi..
> --------------------------
> Contains:
> def __bootstrap__():
> global __bootstrap__, __loader__, __file__
> import sys, pkg_resources, imp
> __file__ = pkg_resources.resource_filename(__name__, '_lucene.so')
> __loader__ = None; del __bootstrap__, __loader__
> imp.load_dynamic(__name__,__file__)
> __bootstrap__()
>
>
> Thanks in advance!
>
> Regards,
> ?Ed
>