Full text indexing patch

[Andy Armstrong <an...@tagish.com>]

Here's a patch to be applied against the current CVS (as at 2004/03/11) 
which adds Lucene based full text indexing. In addition to applying the 
patch you need to add lucene-1.4-rc1-dev.jar (or similar I guess) to 
/java/lib.

To add a full text index to a collection use an index config like this:

   <index
     class="org.apache.xindice.core.indexer.LuceneIndexer"
     name="text-index"
     pattern="text@*"
     analyzer="org.apache.lucene.analysis.SimpleAnalyzer"
   />

If omitted analyzer defaults to the value shown above. To find out about 
other analyzers you'll need to check the Lucene documentation.

To query the full text index do something like this:

     String query = "some lucene query";
     TextQueryService tqs =
         (TextQueryService) col.getService("TextQueryService", "1.0");
     ResourceSet resultSet = tqs.query(query);

At the moment the implementation is pretty much devoid of any kind of 
XML:DB loveliness - it just lets you fire regular Lucene queries at the 
index and returns whole matching documents. Comments and criticism welcome.

-- 
Andy Armstrong, Tagish

Re: Full text indexing patch

[Andy Armstrong <an...@tagish.com>]

Murray Altheim wrote:
> Andy,
> 
> Not to bug you too much, but I have a meeting with my Ph.D. advisors
> on Thursday and I'd love to have some projected date for inclusion
> of full text searching in Xindice. I've been coding furiously for
> weeks and am trying to wrap up a preliminary version of my software
> for a release fairly soon. If not, they'll chop off my head (they
> do that in England I understand).

They do, yes :)

> Noting the patch availability back on the 11th and that you said
> it was working, I was wondering about the current status, and if any
> outstanding issues remain. If it's just a matter of downloading the
> current version of the patch and applying it, and including the
> Lucene jar, then great! I've obviously got some implementation work
> to do on my side too. Any clues, hints, etc. about how to use this
> would be most appreciated.

It was a patch against the then current CVS version. Vadim wanted to 
hold off actually merging it until after the 1.1b4 release (I think). 
That being the case and having lots of other work at the moment I 
decided to wait until Vadim's ready for it before doing any more on it.

> And if someone (perhaps someone else if you're busy) could explain
> how to apply the patch I'd write this up as a wiki page, as I'm
> sure there are others who'd like to have Lucene as an addon to
> Xindice.

Applying the patch would be a bit painful on Windows without something 
like Cygwin that supports patch I think. I could produce you a patched 
version of the current source.

Vadim: what's your thinking on when/if it might be appropriate to merge 
the FT code into the source?

-- 
Andy Armstrong, Tagish

Re: Full text indexing patch

[Murray Altheim <m....@open.ac.uk>]

Andy Armstrong wrote on 2004-03-11:
> Here's a patch to be applied against the current CVS (as at 2004/03/11) 
> which adds Lucene based full text indexing. In addition to applying the 
> patch you need to add lucene-1.4-rc1-dev.jar (or similar I guess) to 
> /java/lib.
[...]

Andy,

Not to bug you too much, but I have a meeting with my Ph.D. advisors
on Thursday and I'd love to have some projected date for inclusion
of full text searching in Xindice. I've been coding furiously for
weeks and am trying to wrap up a preliminary version of my software
for a release fairly soon. If not, they'll chop off my head (they
do that in England I understand).

Noting the patch availability back on the 11th and that you said
it was working, I was wondering about the current status, and if any
outstanding issues remain. If it's just a matter of downloading the
current version of the patch and applying it, and including the
Lucene jar, then great! I've obviously got some implementation work
to do on my side too. Any clues, hints, etc. about how to use this
would be most appreciated.

And if someone (perhaps someone else if you're busy) could explain
how to apply the patch I'd write this up as a wiki page, as I'm
sure there are others who'd like to have Lucene as an addon to
Xindice.

Thanks very much!

Murray

......................................................................
Murray Altheim                    http://kmi.open.ac.uk/people/murray/
Knowledge Media Institute
The Open University, Milton Keynes, Bucks, MK7 6AA, UK               .

    Here's what happens: Someone gets a cool idea. They go, "Hey!
    Here's an idea!" Someone else says, "Yah! I had that same idea!"
    There are big ideas, and little ideas. We want to understand big
    ideas, and let the little details sort themselves out. After the
    ideas thought and told, it's time to act. If you find yourself
    writing papers upon papers, organizing people for the sake of
    organizing people, or using fancy language and what not, you're
    dead. You gotta act. In our case, that means: Write code.
                                           -- Lion Kimbro, InterWiki

Re: Full text indexing patch

[Andy Armstrong <an...@tagish.com>]

Vadim Gritsenko wrote:

> Interesting. Couple of patch formatting suggestions... (a) please use 
> ALv2, Apache License version 2, as the rest of xindice sources, (b) 
> please use 4 spaces instead of tabs. :-)

Noted :)

Sorry. As I read through the patch (after posting it!) it occured to me 
some comment might have been nice too :)

> Now, about indexer itself... First, we need to get 1.1b4 out... For 
> this, I'm planning to write some short doc about current XPath features 
> - because XPath query result format was changed, and then I think 1.1b4 
> should be considered as ready. Once 1.1b4 is out, I can add your patch. 
> Second, what are your plans on evolving this indexer. I think next 
> logical step would be to discuss what text indexer pattern looks like 
> (element/attribute, as for other indexers? or may be it should include 
> path to element?), what should be query language (should it be lucene 
> query language, or any additions to it?), and results should be (whole 
> documents, or matching parts only? or configurable behavior?). What's 
> your opinion?

Yes to all of that really. At the moment it's just scratching and itch 
for me but I'm happy to keep working on it. I'm certain it would be 
nicer if only the matching portions of documents were returned. If I was 
to make that configurable I'd imagine that to be a property of the query 
rather than the way Xindice is set up.

On the implementation I don't have a clear understanding of how Xindice 
achieves thread safe access to the indexes. I'm hoping that's something 
that's enforced at a fairly high level because all I'm doing is some 
fairly half hearted synchronization which woulnd't properly protect the 
Lucene index from concurrent attempts to update it and search on it. 
Perhaps you can advise?

-- 
Andy Armstrong, Tagish

Re: Full text indexing patch

[Vadim Gritsenko <va...@reverycodes.com>]

Andy Armstrong wrote:

> Here's a patch to be applied against the current CVS (as at 
> 2004/03/11) which adds Lucene based full text indexing. In addition to 
> applying the patch you need to add lucene-1.4-rc1-dev.jar (or similar 
> I guess) to /java/lib.
>
> To add a full text index to a collection use an index config like this:
>
>   <index
>     class="org.apache.xindice.core.indexer.LuceneIndexer"
>     name="text-index"
>     pattern="text@*"
>     analyzer="org.apache.lucene.analysis.SimpleAnalyzer"
>   />
>
> If omitted analyzer defaults to the value shown above. To find out 
> about other analyzers you'll need to check the Lucene documentation.
>
> To query the full text index do something like this:
>
>     String query = "some lucene query";
>     TextQueryService tqs =
>         (TextQueryService) col.getService("TextQueryService", "1.0");
>     ResourceSet resultSet = tqs.query(query);
>
> At the moment the implementation is pretty much devoid of any kind of 
> XML:DB loveliness - it just lets you fire regular Lucene queries at 
> the index and returns whole matching documents. Comments and criticism 
> welcome.


Interesting. Couple of patch formatting suggestions... (a) please use 
ALv2, Apache License version 2, as the rest of xindice sources, (b) 
please use 4 spaces instead of tabs. :-)

Now, about indexer itself... First, we need to get 1.1b4 out... For 
this, I'm planning to write some short doc about current XPath features 
- because XPath query result format was changed, and then I think 1.1b4 
should be considered as ready. Once 1.1b4 is out, I can add your patch. 
Second, what are your plans on evolving this indexer. I think next 
logical step would be to discuss what text indexer pattern looks like 
(element/attribute, as for other indexers? or may be it should include 
path to element?), what should be query language (should it be lucene 
query language, or any additions to it?), and results should be (whole 
documents, or matching parts only? or configurable behavior?). What's 
your opinion?

Vadim