Retrieving Document Boosts

[Dan Climan <dc...@keepmedia.com>]

I was trying to test whether the Document Boosts I calculate and add during
indexing were being preserved correctly.
 
I understand that what's actually preserved by default is Field Boost *
Document Boost * lengthNorm
 
I'm using default similarity and initially had no field boosts or document
boosts so I would have expected my initially query of document boosts to be
1.0, but it ranged which likely means I'm not calculating it correctly.
 
Here's the java I tried to use to calculate the document boost:
 
            IndexReader ir = IndexReader.open(indexDir);
            IndexSearcher searcher = new IndexSearcher(ir) ;
            byte[] norms = ir.norms("FullText"); //FullText is the name of
the default field to be searched
            StandardAnalyzer sa = new StandardAnalyzer();
            Similarity sim = searcher.getSimilarity();
            
            TermEnum terms = ir.terms();
            int numTerms = 0;
            while (terms.next())
            {
                Term t = terms.term();
                
                if (t.field().equals("FullText"))
                    numTerms++;
            }
            double lengthNorm = 1.0 / Math.sqrt(numTerms); //since
lengthNorm was defined as 1/sqrt(numTerms) by default

            String key = "bush" // some term to be searched
            Query q2 = QueryParser.parse(key,  "FullText", sa);
            Hits hits2 = searcher.search(q2) ;
            float f = sim.decodeNorm(norms[hits2.id(0)]);  //ie get the norm
for the first hit returned in the search
            System.out.println("Boost: " + f / lengthNorm);
 
What am I missing in the calculation? I understand that there are precision
limitation, but the results I'm getting vary and are mostly in the range
71.68 to 143.35
 
Is there a faster way than iterating through the terms to calculate
lengthNorms?
 

Re: Retrieving Document Boosts

[Doug Cutting <cu...@apache.org>]

Dan Climan wrote:
>             TermEnum terms = ir.terms();
>             int numTerms = 0;
>             while (terms.next())
>             {
>                 Term t = terms.term();
>                 
>                 if (t.field().equals("FullText"))
>                     numTerms++;
>             }
>             double lengthNorm = 1.0 / Math.sqrt(numTerms); //since
> lengthNorm was defined as 1/sqrt(numTerms) by default

The numTerms is not the number of unique words in the collection, but 
rather the number of tokens in the document in question.  So, if you 
want to re-create this externally you could re-tokenize the text for the 
field and count the tokens.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-dev-help@jakarta.apache.org