You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Sean O'Connor <se...@gmail.com> on 2005/06/16 18:03:46 UTC
Search Hit frequency and location
Hello,
I am trying to find the right approach for finding frequency (and,
slightly lower in priority, location) of search hits in a document. I
am working through the online documentation and the helpful "Lucene in
Action" book. There are several examples and explanations which seem
close, but not quite what I am looking for. Can anyone point me in the
right direction?
I have a set of queries, say 10000 different 'things' I want to
find. They range from single word matches (e.g. Lucene) to prefix
queries (e.g. index*) to phrases (e.g. "Lucene in Action", stopword
irrelevant, so "Lucene Action"?). I will also be delving into the more
advanced topics like proximity, fuzzy, snowball and such. For the
moment though, I will stick with the first three I mention, which I
believe translate to: TermQuery, PrefixQuery, and PhraseQuery.
How do I find how many hits occur in a document? I've seen the faq:
Is there a way to retrieve the original term positions during the search?
Yes, see the Javadoc for IndexReader.termPositions().
I'm probably missing the obvious here, but I assume this refers to
the analyzed terms (i.e. individual words, possibly transmogrified by
the analyzer).
I further assume that this does not directly relate to the results of
a search for "Lucene in Action". Where do I find information about the
search hits? Have I skimmed over this part of the API?
Thanks in advance,
Sean
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search Hit frequency and location
Posted by Paul Elschot <pa...@xs4all.nl>.
On Thursday 16 June 2005 21:03, Sean O'Connor wrote:
> Thanks for the clarification. I had assumed that to be the case, but
> assumptions have a tendency to come back and bite me in inappropriate
> places. By pointing that out, you've probably saved me from beating my
> head against the wall in the near future : -).
>
> The big stumbling block I have at the moment is understanding whether
> Terms can be used to find something like a phrase query, proximity
> query, or boolean query. I think the answer is no, two different
Terms are normally used as building blocks all of these.
> concepts. But I also tend to think that the wheel has already been
> invented to find how many times a phrase (i.e. "Lucene in Action")
> appears in a document. Before I go digging through the source code, and
> possibly creating some rather embarrassing hack(s), I thought I would
> check to see if there is a 'right' way to go about this.
Counting how often a query occurs is actually a simplification
of the normal query search. A PhraseQuery for the phrase
"lucene in action" provides an ExactPhraseScorer to score
each document. Have a look at the code of that scorer, its superclass
PhraseScorer only needs a minor modification to provide the phrase
frequency in a document as the score() for that document.
Getting from a Query to a Scorer is somewhat involved, but
it is documented in the javadocs of Weight.
An easy way to implement this is to copy the PhraseQuery
to one of your own so it uses your modified versions of (Exact)PhraseScorer.
Once this query works, override the corresponding get...() method in
your subclass of QueryParser to return that query instead of PhraseQuery.
For testing, have a look at TestPhraseQuery.java in the src/test directory.
> Alternatively, any suggestions on what to google, or where to look to
> educate myself would be welcome as well.
TermQuery and TermScorer make a good starting point. To save
some reading, ignore the explain() methods initially.
> Cheers,
> Sean
Proost,
Paul Elschot.
>
>
> Erik Hatcher wrote:
>
> >
> > On Jun 16, 2005, at 12:03 PM, Sean O'Connor wrote:
> >
> >> Yes, see the Javadoc for IndexReader.termPositions().
> >> I'm probably missing the obvious here, but I assume this refers to
> >> the analyzed terms (i.e. individual words, possibly transmogrified by
> >> the analyzer).
> >
> >
> > Just to respond to part of your mail:
> >
> > Terms do not necessarily come from analysis... they could be
> > specified directly using Field.Keyword() for example. Any _indexed_
> > field has term(s), with the possibility that the indexed field is
> > analyzed or not.
> >
> > Erik
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search Hit frequency and location
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Jun 16, 2005, at 3:03 PM, Sean O'Connor wrote:
> The big stumbling block I have at the moment is understanding
> whether Terms can be used to find something like a phrase query,
> proximity query, or boolean query. I think the answer is no, two
> different concepts.
Terms are the heart of searching. All queries boil down to finding
specific terms. A PhraseQuery finds terms exactly side-by-side
position-wise (or near by some slop factor). BooleanQuery is an
aggregation of other primitive queries, so its sort of special in
that it doesn't deal with terms directly, but all primitive
aggregated queries do. A PrefixQuery is not a "primitive" query - it
rewrites itself to a BooleanQuery OR'd with all the matching terms.
So the terms are the "atoms" of Lucene. Hopefully this is in-line
with the conceptual stumbling block.
> But I also tend to think that the wheel has already been invented
> to find how many times a phrase (i.e. "Lucene in Action") appears
> in a document. Before I go digging through the source code, and
> possibly creating some rather embarrassing hack(s), I thought I
> would check to see if there is a 'right' way to go about this.
Deep within PhraseQuery the number of occurrences is tracked and used
in scoring. But at the high level of searching, that detail is not
exposed. Using IndexSearcher.explain to see the tf() of a
PhraseQuery may help, but by default it'll be the square root of the
number of occurrences. (at least I hope I'm on track with that
explanation - please correct me if I've misspoken) I'd love to hear
how others have done this sort of thing - find the number of
occurrences of a phrase.
> Alternatively, any suggestions on what to google, or where to look
> to educate myself would be welcome as well.
You're in the right place. Keep asking - we're all here to learn and
share what we know.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search Hit frequency and location
Posted by Sean O'Connor <se...@oconeco.com>.
Thanks for the clarification. I had assumed that to be the case, but
assumptions have a tendency to come back and bite me in inappropriate
places. By pointing that out, you've probably saved me from beating my
head against the wall in the near future : -).
The big stumbling block I have at the moment is understanding whether
Terms can be used to find something like a phrase query, proximity
query, or boolean query. I think the answer is no, two different
concepts. But I also tend to think that the wheel has already been
invented to find how many times a phrase (i.e. "Lucene in Action")
appears in a document. Before I go digging through the source code, and
possibly creating some rather embarrassing hack(s), I thought I would
check to see if there is a 'right' way to go about this.
Alternatively, any suggestions on what to google, or where to look to
educate myself would be welcome as well.
Cheers,
Sean
Erik Hatcher wrote:
>
> On Jun 16, 2005, at 12:03 PM, Sean O'Connor wrote:
>
>> Yes, see the Javadoc for IndexReader.termPositions().
>> I'm probably missing the obvious here, but I assume this refers to
>> the analyzed terms (i.e. individual words, possibly transmogrified by
>> the analyzer).
>
>
> Just to respond to part of your mail:
>
> Terms do not necessarily come from analysis... they could be
> specified directly using Field.Keyword() for example. Any _indexed_
> field has term(s), with the possibility that the indexed field is
> analyzed or not.
>
> Erik
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Search Hit frequency and location
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Jun 16, 2005, at 12:03 PM, Sean O'Connor wrote:
> Yes, see the Javadoc for IndexReader.termPositions().
> I'm probably missing the obvious here, but I assume this refers to
> the analyzed terms (i.e. individual words, possibly transmogrified by
> the analyzer).
Just to respond to part of your mail:
Terms do not necessarily come from analysis... they could be
specified directly using Field.Keyword() for example. Any _indexed_
field has term(s), with the possibility that the indexed field is
analyzed or not.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org