You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Marjan Celikik <ce...@gmail.com> on 2008/01/09 21:13:39 UTC
Highlighting + phrase queries
Dear all,
Let's assume I have a phrase query and a document which contain the
phrase but also it contains separate
occurrences of each query term. How does the highlighter know that
should only display fragments which
contain phrases and not fragments which contain only the query words
(not as a phrase)?
Thank you in advance!
Marjan.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Highlighting + phrase queries
Posted by Mark Miller <ma...@gmail.com>.
It should work no problem with 2.2. What are the compile errors you are
getting?
If you send me a note directly I will send you a jar.
- Mark
Marjan Celikik wrote:
> Mark Miller wrote:
>> The contrib Highlighter doesn't know and highlights them all.
>>
>> Check out my patch here for position sensitive highlighting:
>> https://issues.apache.org/jira/browse/LUCENE-794
>>
>>
> It seems that the patch does not work with Lucene 2.2 as I get some
> compile errors. Is this really the case?
>
> Marjan.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Highlighting + phrase queries
Posted by Marjan Celikik <ce...@gmail.com>.
Marjan Celikik wrote:
> Mark Miller wrote:
>> The Highlighter works by comparing the TokenStream of the document
>> with the Tokens in the query. The TokenStream can be rebuilt from the
>> index if you use TermVectors with TokenSources or you can get it by
>> reanalyzing the document. Each Token from the TokenStream is checked
>> against Tokens in the query, and if there is a match you have a
>> Highlight. The original text is then reconstructed with the
>> Highlights from info in the TokenStream about original offsets into
>> the document for each Token. Also, there is a Fragment system that
>> will break apart the Highlighted text into score sorted text Fragments.
> OK, this is what I already knew...
>>
>> That is why the original contrib does not work with PhraseQuery's. It
>> simply matches Tokens from the query with those in the TokenStream.
>> LUCENE-794 takes the TokenStream and shoves it into a MemoryIndex.
>> Then, after converting the query to a SpanQuery approximation,
>> getSpans is called on the index for the query. The Spans provide a
>> bound on what positions should be Highlighted. Everything else is
>> done exactly like the original Highlighter (This is a patch that fits
>> into the original Highlighter framework that was developed, thereby
>> retaining all of its richness :) ).
>>
>>
> Thanks! This is what I needed. Still I don't know how to obtain the
> source code of your patch :(
ok, I will just apply the patch. Sorry.
Marjan.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Highlighting + phrase queries
Posted by Marjan Celikik <ce...@gmail.com>.
Mark Miller wrote:
> The Highlighter works by comparing the TokenStream of the document
> with the Tokens in the query. The TokenStream can be rebuilt from the
> index if you use TermVectors with TokenSources or you can get it by
> reanalyzing the document. Each Token from the TokenStream is checked
> against Tokens in the query, and if there is a match you have a
> Highlight. The original text is then reconstructed with the Highlights
> from info in the TokenStream about original offsets into the document
> for each Token. Also, there is a Fragment system that will break apart
> the Highlighted text into score sorted text Fragments.
OK, this is what I already knew...
>
> That is why the original contrib does not work with PhraseQuery's. It
> simply matches Tokens from the query with those in the TokenStream.
> LUCENE-794 takes the TokenStream and shoves it into a MemoryIndex.
> Then, after converting the query to a SpanQuery approximation,
> getSpans is called on the index for the query. The Spans provide a
> bound on what positions should be Highlighted. Everything else is done
> exactly like the original Highlighter (This is a patch that fits into
> the original Highlighter framework that was developed, thereby
> retaining all of its richness :) ).
>
>
Thanks! This is what I needed. Still I don't know how to obtain the
source code of your patch :(
Majan.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Highlighting + phrase queries
Posted by Mark Miller <ma...@gmail.com>.
I don't think you would see much of gain. Shoving the TokenStream into
the MemoryIndex is actually pretty fast and I wouldn't be surprised if
it was much faster than reading from disk. Most of the computational
time is spent in reconstructing the TokenStream, whether you use
term-vectors or re-analyze. Also, if the Query does not have any
position sensitive clauses, no MemoryIndex is created, so no worries there.
The great speed challenge of the current method (other than needing a
TokenStream created) is that it runs over each Token and stitches the
document together a piece at a time. This doesn't scale well on huge
docs. There are ways to cut this down and to just analyze the pertinent
Tokens as is done by a different patch. However, you'd need to have
TermVectors stored, and the concept doesn't fit with the current
Highlighter framework, which already has some significant functionality
and robustness.
- Mark
Marjan Celikik wrote:
> Mark Miller wrote:
>>
>> That is why the original contrib does not work with PhraseQuery's. It
>> simply matches Tokens from the query with those in the TokenStream.
>> LUCENE-794 takes the TokenStream and shoves it into a MemoryIndex.
>> Then, after converting the query to a SpanQuery approximation,
>> getSpans is called on the index for the query. The Spans provide a
>> bound on what positions should be Highlighted. Everything else is
>> done exactly like the original Highlighter (This is a patch that fits
>> into the original Highlighter framework that was developed, thereby
>> retaining all of its richness :) ).
>>
>>
> Mark, thanks for your patience! I have one final (conceptual,
> high-level) question concerning the usage of the MemoryIndex index
> over the TokenStream. Is it a good idea to
> store the procomputed MemoryIndex (conceptually speaking) as a field
> into each document at indexing time and then just load this
> precomputed index from
> disk (as you do with TermVector) such that you save extra computation
> for the highlighting?
>
> Marjan.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Highlighting + phrase queries
Posted by Marjan Celikik <ce...@gmail.com>.
Mark Miller wrote:
>
> That is why the original contrib does not work with PhraseQuery's. It
> simply matches Tokens from the query with those in the TokenStream.
> LUCENE-794 takes the TokenStream and shoves it into a MemoryIndex.
> Then, after converting the query to a SpanQuery approximation,
> getSpans is called on the index for the query. The Spans provide a
> bound on what positions should be Highlighted. Everything else is done
> exactly like the original Highlighter (This is a patch that fits into
> the original Highlighter framework that was developed, thereby
> retaining all of its richness :) ).
>
>
Mark, thanks for your patience! I have one final (conceptual,
high-level) question concerning the usage of the MemoryIndex index over
the TokenStream. Is it a good idea to
store the procomputed MemoryIndex (conceptually speaking) as a field
into each document at indexing time and then just load this precomputed
index from
disk (as you do with TermVector) such that you save extra computation
for the highlighting?
Marjan.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Highlighting + phrase queries
Posted by Mark Miller <ma...@gmail.com>.
The Highlighter works by comparing the TokenStream of the document with
the Tokens in the query. The TokenStream can be rebuilt from the index
if you use TermVectors with TokenSources or you can get it by
reanalyzing the document. Each Token from the TokenStream is checked
against Tokens in the query, and if there is a match you have a
Highlight. The original text is then reconstructed with the Highlights
from info in the TokenStream about original offsets into the document
for each Token. Also, there is a Fragment system that will break apart
the Highlighted text into score sorted text Fragments.
That is why the original contrib does not work with PhraseQuery's. It
simply matches Tokens from the query with those in the TokenStream.
LUCENE-794 takes the TokenStream and shoves it into a MemoryIndex. Then,
after converting the query to a SpanQuery approximation, getSpans is
called on the index for the query. The Spans provide a bound on what
positions should be Highlighted. Everything else is done exactly like
the original Highlighter (This is a patch that fits into the original
Highlighter framework that was developed, thereby retaining all of its
richness :) ).
Marjan Celikik wrote:
> Mark Miller wrote:
>> Oh yeah...something that you may not have seen is that this has a
>> dependency on MemoryIndex from contrib. You need that jar as well.
>>
>> - Mark
> Hm, I need the source code. How do I download the files from
> https://issues.apache.org/jira/browse/LUCENE-794 (all I see are some
> .patch files)?
>
> What I really need is a how the highlighter works in a nutshell... I
> am working on a publication and I want to have a reference to Lucene
> and its highlighting...
>
> Thanks again.
>
> Marjan.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Highlighting + phrase queries
Posted by Marjan Celikik <ce...@gmail.com>.
Mark Miller wrote:
> Oh yeah...something that you may not have seen is that this has a
> dependency on MemoryIndex from contrib. You need that jar as well.
>
> - Mark
Hm, I need the source code. How do I download the files from
https://issues.apache.org/jira/browse/LUCENE-794 (all I see are some
.patch files)?
What I really need is a how the highlighter works in a nutshell... I am
working on a publication and I want to have a reference to Lucene and
its highlighting...
Thanks again.
Marjan.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Highlighting + phrase queries
Posted by Mark Miller <ma...@gmail.com>.
Oh yeah...something that you may not have seen is that this has a
dependency on MemoryIndex from contrib. You need that jar as well.
- Mark
Marjan Celikik wrote:
> Mark Miller wrote:
>> The contrib Highlighter doesn't know and highlights them all.
>>
>> Check out my patch here for position sensitive highlighting:
>> https://issues.apache.org/jira/browse/LUCENE-794
>>
>>
> It seems that the patch does not work with Lucene 2.2 as I get some
> compile errors. Is this really the case?
>
> Marjan.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Highlighting + phrase queries
Posted by Marjan Celikik <ce...@gmail.com>.
Mark Miller wrote:
> The contrib Highlighter doesn't know and highlights them all.
>
> Check out my patch here for position sensitive highlighting:
> https://issues.apache.org/jira/browse/LUCENE-794
>
>
It seems that the patch does not work with Lucene 2.2 as I get some
compile errors. Is this really the case?
Marjan.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Highlighting + phrase queries
Posted by Mark Miller <ma...@gmail.com>.
It works exactly the same as the standard contrib Highlighter except
that it tries not to highlight spurious results for a positional query.
This is exact with Span queries, but more approximate for phrase
queries. The approximation is pretty darn good, but let me know if you
find a case that doesn't work. The Highlighter attempts to mimic the
PhraseQeury with a SpanQuery.
Marjan Celikik wrote:
> Mark Miller wrote:
>> The contrib Highlighter doesn't know and highlights them all.
>>
>> Check out my patch here for position sensitive highlighting:
>> https://issues.apache.org/jira/browse/LUCENE-794
> OK, before trying it out, I would like to know does the patch work for
> mixed queries, e.g. "a b" +c -d "f g" ?
>
> Thanks!
>
> Marjan.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Highlighting + phrase queries
Posted by Marjan Celikik <ce...@gmail.com>.
Mark Miller wrote:
> The contrib Highlighter doesn't know and highlights them all.
>
> Check out my patch here for position sensitive highlighting:
> https://issues.apache.org/jira/browse/LUCENE-794
OK, before trying it out, I would like to know does the patch work for
mixed queries, e.g. "a b" +c -d "f g" ?
Thanks!
Marjan.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Highlighting + phrase queries
Posted by Mark Miller <ma...@gmail.com>.
The contrib Highlighter doesn't know and highlights them all.
Check out my patch here for position sensitive highlighting:
https://issues.apache.org/jira/browse/LUCENE-794
Marjan Celikik wrote:
> Dear all,
>
> Let's assume I have a phrase query and a document which contain the
> phrase but also it contains separate
> occurrences of each query term. How does the highlighter know that
> should only display fragments which
> contain phrases and not fragments which contain only the query words
> (not as a phrase)?
>
> Thank you in advance!
>
> Marjan.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org