You are viewing a plain text version of this content. The canonical link for it is here.
Posted to xindice-dev@xml.apache.org by Andy Armstrong <an...@tagish.com> on 2004/03/10 13:42:49 UTC

Extending org.apache.xindice.core.indexer.Indexer

I'm currently working on integrating full text indexing and searching 
and as part of that I need to extend the o.a.x.c.i.Indexer interface to 
include

   public void startDocument(Key key) throws DBException;
   public void endDocument(Key key) throws DBException;

So that existing Indexers don't have to implement this methods I've 
actually done this:

public interface ExtendedIndexer extends Indexer {
	public void startDocument(Key key) throws DBException;
	public void endDocument(Key key) throws DBException;
}

and made IndexManager test for an Indexer being an instance of 
ExtendedIndexer in which case it invokes these methods at the 
appropriate times. The alternative would be just to add these to the 
Indexer interface and require all existing Indexers to implement the two 
extra methods.

ExtendedIndexer

+ doesn't affect existing Indexers
- needlessly complicates things

Add methods to Indexer

+ simple
- breaks existing Indexers

I guess it comes down to whether there /are/ any extant Indexers outside 
of the Xindice source. I guess perhaps not?

-- 
Andy Armstrong, Tagish



Re: Extending org.apache.xindice.core.indexer.Indexer

Posted by Andy Armstrong <an...@tagish.com>.
Vadim Gritsenko wrote:
> Can you explain a bit why these methods are needed, and why existing 
> Indexer interface is not enough?

Thanks for the sanity check Vadim. I've binned that whole silly idea. 
Everyone relax.

-- 
Andy Armstrong, Tagish


Re: Extending org.apache.xindice.core.indexer.Indexer

Posted by Terry Rosenbaum <Te...@amicas.com>.
Andy Armstrong wrote:

 > At the moment I'm not doing XPath search syntax. Searching looks like 
this:

 >  String query = "some text to find";
 >  TextQueryService tqs =
 >      (TextQueryService) col.getService("TextQueryService", "1.0");
 >  resultSet = tqs.query(query);

 > I'm open to suggestions for improvements / alternatives :)

This is a good thing since there is no way to provide
support for many of the possible Lucene queries
in the current XPath syntax.

A good addition would be implementation of a substring indexer
and use that to support standard XPath "contains"
and "ends-with" functions. The same index could be used
for both. "ends-with" is a special case of "contains" and the
indexed evaluation for "ends-with" could be "contains". The
"ends-with" predicate would be further resolved when the
full XPath evaluation runs following the indexed evaluation.
A substring indexer would index, perhaps, all 3-character
substrings. Indexed searches for 1- or 2-character strings
could either punt for the full collection scan, or implement as
a range query against the 3-character substring index. Optimal
performance would occur when the searched-for string was 3 characters.
Perhaps the index definition could specify the substring size for the
index.

o.a.x.core.query.XPathQueryResolver is where the indexed evaluations occur.

Compare/contrast the code in methods
Object evalValComparison(int op, String owner, int pos)
vs. funcContains(List args).

Unfortunately, job responsibilities prevent me from working on this
right now. :( When we are not close to the end of a product release
cycle, my employer will allow me to donate company-paid time to
the effort (as I have done in the past).

-Terry

> Terry Rosenbaum wrote:
>
>> One may want to implement a full text index on several
>> different index patterns. e.g. specify an index on some
>> element and an index on some attribute, Would your
>> implementation handle that case?
>
>
> Yes although right now it's concatenating all the text that matches a 
> particular pattern (x@y) within a document together and it returns the 
> whole document in the case of a match. I'm changing that behaviour 
> right now :)
>
>> Just out of curiosity, does your implementation support the
>> existing standard XPath approach to searching? What sort
>> of analyzer are you using? Did you decide to allow users
>> to specify the analyzer somehow?
>
>
> You get to specify the Lucene analyzer when you create the index via 
> an additional 'analyzer' attribute.
>
> At the moment I'm not doing XPath search syntax. Searching looks like 
> this:
>
>   String query = "some text to find";
>   TextQueryService tqs =
>       (TextQueryService) col.getService("TextQueryService", "1.0");
>   resultSet = tqs.query(query);
>
> I'm open to suggestions for improvements / alternatives :)
>

Re: Extending org.apache.xindice.core.indexer.Indexer

Posted by Andy Armstrong <an...@tagish.com>.
Terry Rosenbaum wrote:
> One may want to implement a full text index on several
> different index patterns. e.g. specify an index on some
> element and an index on some attribute, Would your
> implementation handle that case?

Yes although right now it's concatenating all the text that matches a 
particular pattern (x@y) within a document together and it returns the 
whole document in the case of a match. I'm changing that behaviour right 
now :)

> Just out of curiosity, does your implementation support the
> existing standard XPath approach to searching? What sort
> of analyzer are you using? Did you decide to allow users
> to specify the analyzer somehow?

You get to specify the Lucene analyzer when you create the index via an 
additional 'analyzer' attribute.

At the moment I'm not doing XPath search syntax. Searching looks like this:

   String query = "some text to find";
   TextQueryService tqs =
       (TextQueryService) col.getService("TextQueryService", "1.0");
   resultSet = tqs.query(query);

I'm open to suggestions for improvements / alternatives :)

-- 
Andy Armstrong, Tagish


Re: Extending org.apache.xindice.core.indexer.Indexer

Posted by Terry Rosenbaum <Te...@amicas.com>.
Andy Armstrong wrote:

 > My current approach is to have a one-to-one relationship between Xindice
 > documents and Lucene's concept of a document but it's just occured
 > to me that I should maybe have a Lucene document for each chunk
 > of text in a Xindice document.

One may want to implement a full text index on several
different index patterns. e.g. specify an index on some
element and an index on some attribute, Would your
implementation handle that case?

Just out of curiosity, does your implementation support the
existing standard XPath approach to searching? What sort
of analyzer are you using? Did you decide to allow users
to specify the analyzer somehow?

Great to see  you continuing work on this! Thanks.

-Terry

Andy Armstrong wrote:

> Vadim Gritsenko wrote:
>
>> Andy Armstrong wrote:
>>
>>> I'm currently working on integrating full text indexing and 
>>> searching and as part of that I need to extend the o.a.x.c.i.Indexer 
>>> interface to include
>>>
>>>   public void startDocument(Key key) throws DBException;
>>>   public void endDocument(Key key) throws DBException;
>>
>>
>> Can you explain a bit why these methods are needed, and why existing 
>> Indexer interface is not enough?
>
>
> I'm assuming (perhaps incorrectly) that Indexer.add() can be called 
> more than once for each document which doesn't sit happily with 
> Lucene's API.
>
> My current approach is to have a one-to-one relationship between 
> Xindice documents and Lucene's concept of a document but it's just 
> occured to me that I should maybe have a Lucene document for each 
> chunk of text in a Xindice document.
>
> Sorry; I seem to be thinking out loud now. Let me go and experiment a 
> little.
>

Re: Extending org.apache.xindice.core.indexer.Indexer

Posted by Andy Armstrong <an...@tagish.com>.
Vadim Gritsenko wrote:

> Andy Armstrong wrote:
> 
>> I'm currently working on integrating full text indexing and searching 
>> and as part of that I need to extend the o.a.x.c.i.Indexer interface 
>> to include
>>
>>   public void startDocument(Key key) throws DBException;
>>   public void endDocument(Key key) throws DBException;
> 
> Can you explain a bit why these methods are needed, and why existing 
> Indexer interface is not enough?

I'm assuming (perhaps incorrectly) that Indexer.add() can be called more 
than once for each document which doesn't sit happily with Lucene's API.

My current approach is to have a one-to-one relationship between Xindice 
documents and Lucene's concept of a document but it's just occured to me 
that I should maybe have a Lucene document for each chunk of text in a 
Xindice document.

Sorry; I seem to be thinking out loud now. Let me go and experiment a 
little.

-- 
Andy Armstrong, Tagish


Re: Extending org.apache.xindice.core.indexer.Indexer

Posted by Vadim Gritsenko <va...@reverycodes.com>.
Andy Armstrong wrote:

> I'm currently working on integrating full text indexing and searching 
> and as part of that I need to extend the o.a.x.c.i.Indexer interface 
> to include
>
>   public void startDocument(Key key) throws DBException;
>   public void endDocument(Key key) throws DBException;


Can you explain a bit why these methods are needed, and why existing 
Indexer interface is not enough?

Vadim