Frequency of phrase

[Eric Jain <Er...@isb-sib.ch>]

This is somewhat related to a question sent to this list a while ago: Is 
there an efficient way to count the number of occurrences of a phrase (not 
term) in an index?

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Frequency of phrase

[Chris Hostetter <ho...@fucit.org>]

: > If you use a span query then you can get the actual number of phrase
: > instances.
:
: Thanks, good to know!

Just to clarify, i Doug means you can use the getSpans method of a
SpanNear query, and then count the iterations of next() untill you run
out.

Which is a really good idea I hadn't thought of before.

Another idea that was floating arround on the tip of my brain after the
recent comment about using a PhraseQUery and the idf, is that with a
custom Similarity that impliminted all of it's functions as constants, and
a HitCollector that just recorded the sum of the raw scores, you should
also be able to get the exact number of occurances of an (exact)
PhraseQuery.

You'd need to double check the score calculation to figure out exactly
which constants each of the functions should return -- I think it would
mainly be 1.0f, but the sumOfSquares aspect of the equation may make one
of them more interesting.  The only exception would be tf(float) which (I
think) should be an identity function.

Given other comments I've seen from people I trust that PhraseQueries are
faster then SpanNearQueries, that may be faster then the Span appraoch
Doug alluded too -- but it's also more complicated to get working.



-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Frequency of phrase

[Eric Jain <Er...@isb-sib.ch>]

Doug Cutting wrote:
> If you use a span query then you can get the actual number of phrase 
> instances.

Thanks, good to know!

In this case (need to suggest phrase queries to the user) I've now settled 
with dividing the number of hits for a potential phrase by the number of 
documents that contain all terms in the phrase. Seems to be fast and work 
well...


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Frequency of phrase

[Doug Cutting <cu...@apache.org>]

Eric Jain wrote:
> This gives you the number of documents containing the phrase, rather 
> than the number of occurrences of the phrase itself, but that may in 
> fact be good enough...

If you use a span query then you can get the actual number of phrase 
instances.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Frequency of phrase

[Eric Jain <Er...@isb-sib.ch>]

Dave Kor wrote:
> Not sure if this is what you want, but what I have done is to issue
> exact phrase queries to Lucene and counted the number of hits found.

This gives you the number of documents containing the phrase, rather than 
the number of occurrences of the phrase itself, but that may in fact be 
good enough...

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Frequency of phrase

[Dave Kor <da...@gmail.com>]

Not sure if this is what you want, but what I have done is to issue
exact phrase queries to Lucene and counted the number of hits found.

On 2/23/06, Eric Jain <Er...@isb-sib.ch> wrote:
> This is somewhat related to a question sent to this list a while ago: Is
> there an efficient way to count the number of occurrences of a phrase (not
> term) in an index?

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org