You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by ma...@yahoo.co.uk on 2004/06/19 20:45:31 UTC

Re: amusing interaction between advanced tokenizers and highlighter

A question before I dive into coding a fix: can I assume (for all analyzers) that the tokens produced by the tokenStream 
have the following property: 
   currentToken.startOffset() >= lastToken.startOffset()

The analyzers I have tested the highlighter with so far have the property:
   currentToken.startOffset() > lastToken.endOffset()
so aren't overlapping but I understand this isn't the case for others (all demonstrable examples of such "problem" analyzers 
would be appreciated for testing purposes).
If I can assume that tokenstreams always produce a zero or more increment in token.startOffset I think I can 
design a solution that still works using a single pass of the token stream.
I suspect an additional "flushText" method will be required on the Formatter interface to allow implementations
to use a buffer. This buffer would be required to accumulate overlapping token scores when trying to decide if a 
section of the original text merited any highlight markup.

Cheers
Mark


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


RE : amusing interaction between advanced tokenizers and highlighter

Posted by Rasik Pandey <ra...@ajlsm.com>.
> A question before I dive into coding a fix: can I assume (for
> all analyzers) that the tokens produced by the tokenStream
> have the following property:
>    currentToken.startOffset() >= lastToken.startOffset()
> 
> The analyzers I have tested the highlighter with so far have
> the property:
>    currentToken.startOffset() > lastToken.endOffset()
> so aren't overlapping but I understand this isn't the case for
> others (all demonstrable examples of such "problem" analyzers
> would be appreciated for testing purposes).

There is such an analyzer here
http://savannah.nongnu.org/projects/aramorph .

> If I can assume that tokenstreams always produce a zero or more
> increment in token.startOffset I think I can
> design a solution that still works using a single pass of the
> token stream.
> I suspect an additional "flushText" method will be required on
> the Formatter interface to allow implementations
> to use a buffer. This buffer would be required to accumulate
> overlapping token scores when trying to decide if a
> section of the original text merited any highlight markup.

I am not familiar with your most recent highlighter package, but I have implemented this myself with some older rudimentary highlighting code that just uses a Vector to keep track of all tokens for the same offset positions. Highlighting based on those tokens accumulated in the Vector is triggered when currentToken.startOffset() > lastToken.startOffset() is satisfied, after which the token Vector is simply cleared and the new token position tracking begins. Don't forget to make sure that the same input/term text isn't output/highlighted more than once for multiple output tokens.

Regards,
RBP 





---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


QueryParser handling a NOT query on its own

Posted by Allen Atamer <aa...@casebank.com>.
The Javadoc spec calls for one or more clauses in a query, but I had trouble
with a NOT query just on its own. For example

QueryParser.parse("my_field:-exclude") throws a parsing exception

Same with

QueryParser.parse("my_field:-(exclude)")
QueryParser.parse("my_field:(* AND -exclude")

The query QueryParser.parse("my_field:(-(exclude))") gives a legitimate
query that brings no results.

What I would expect is the following: If I have an index with 100 total
entries, and 20 records with the word "exclude" in them, then the above
queries should give 80 hits. There is no test case for this scenario in
TestQueryParser. Please confirm whether this is a bug or not,

Thank you,

Allen Atamer


---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org