You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Marvin Humphrey <ma...@rectangular.com> on 2006/04/27 20:58:28 UTC

Rich positions (was "boosting fields")

On Apr 27, 2006, at 9:41 AM, Doug Cutting wrote:

> karl wettin wrote:
>> My own immediate thought is to compromise by allowing boost per  
>> term  in document. Simply remove the norms-methods from the  
>> IndexReader and  add a new one to the TermEnum and fall back on  
>> the field boost. How  would the value be picked up by the scorer?
>> Boost per position, et.c. sounds very expensive.
>
> Indeed.  It will probably nearly double the size of indexes and  
> also increase search time.

I have been considering making a similar change to the KinoSearch  
file format.  Not having to cache norms radically cuts down on the  
time required to launch a fresh Searcher, especially if there aren't  
any deleted docs.  That's a win if you're launching a search app from  
scratch, like if you're running a web search under CGI rather than  
mod_perl.  It's also a win for refreshing a Searcher against a  
frequently updated index.

What I was considering was interleaving the document's score- 
multiplier norm byte between the VInts in the .frq file.  That would  
mean more disk i/o for processing terms when the term takes up more  
than a block on the file system, but at least the info would be  
contiguous.

I hadn't considered interleaving the score-multiplier into .prx, but  
that opens many possibilities.  Boost positions that appear near the  
top of the doc.  Boost positions if they occur within certain HTML  
tags.  Good stuff!

Moving away from cached norms was the second of three major changes  
to the file format on my agenda, and the one I was all but certain I  
wouldn't be able to sell to the Lucene community.  The first was  
using bytecounts at the head of Strings.

The third was storing start offsets and end offsets in the ProxFile.   
It rankles that much of the information from tis/frq/prx gets  
duplicated in the term vector files, but highlighting is most  
efficient when you know the offsets, and the primary index stops  
short of storing that information.  Currently, we have this:

     ProxFile (.prx) -->  <TermPositions>TermCount

How about this?

     ProxFile (.prx) -->  <TermPositions,TermOffsets>TermCount

To get highlighting info now, you retrieve a document's term vector  
information and then extract the offsets information for the precise  
term.  This format reverses the order: first you find the term, then  
you extract the offsets info for a particular doc.

I haven't implemented this change yet, so I'm not sure how it works  
out.  The current version of KinoSearch stores term vectors in  
the .fdt file, which is a win for locality of reference.  It sure  
would be nice to eliminate all that duplicated data, though.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Rich positions (was "boosting fields")

Posted by Marvin Humphrey <ma...@rectangular.com>.
Now that I think about it, putting the score-multiplier into the  
FreqFile does offer a benefit I hadn't considered before.  It makes  
it possible to tie the score multiplier to a term within a doc,  
rather than a field within a doc.

Say you have a doc with a "body" field that's 1000 terms long, with 3  
instances of "foo", right near the top.  Say you have another doc  
with a "body" that's 1000 terms long, and that it also has 3  
instances of "foo" but they're buried near the end, and therefore not  
as important.

Under the current implementation, these two docs will score  
identically against a query for "body:foo", as the freq is identical  
and so is the output of lengthNorm(1000).  But if you stuff the score  
multiplier into the FreqFile, a sophisticated indexing app could  
assign a higher score multiplier to the term "body:foo" in the first  
doc.

Associating a score multiplier with each position (a.k.a.  
WEIGHT_PER_POSITION, BOOST_PER_POSITION) would achieve the same end,  
but at the expense of much more processing per document, as the score  
would have to be built up position by position.   OTOH, it would  
produce more accurate results for queries where only certain  
positions ought to be considered, such as phrase queries.

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Rich positions (was "boosting fields")

Posted by Doug Cutting <cu...@apache.org>.
Marvin Humphrey wrote:
> Incidentally, how about calling it BOOST_PER_POSITION instead?

+1, that is more consistent with other naming.

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Rich positions (was "boosting fields")

Posted by Marvin Humphrey <ma...@rectangular.com>.
On Apr 27, 2006, at 12:17 PM, Doug Cutting wrote:

> Marvin Humphrey wrote:
>> Moving away from cached norms was the second of three major  
>> changes  to the file format on my agenda, and the one I was all  
>> but certain I  wouldn't be able to sell to the Lucene community.   
>> The first was  using bytecounts at the head of Strings.
>> The third was storing start offsets and end offsets in the  
>> ProxFile.   It rankles that much of the information from tis/frq/ 
>> prx gets  duplicated in the term vector files, but highlighting is  
>> most  efficient when you know the offsets, and the primary index  
>> stops  short of storing that information.  Currently, we have this:
>>     ProxFile (.prx) -->  <TermPositions>TermCount
>> How about this?
>>     ProxFile (.prx) -->  <TermPositions,TermOffsets>TermCount
>
> This would at least double the size of the .prx file, the largest  
> file in Lucene's index.  Yes it's useful, not not all folks will  
> use it.  So not all folks should have to pay for it.

Agreed.  I think it would at least triple the ProxFile, actually.  At  
least, I haven't thought of a compression scheme which could cram  
both start offset and end offset into fewer than two bytes on  
average.  But in theory, it would eliminate the need for the Term  
Vectors files, so if you need those now only for highlighting, it's a  
big gain.

> So perhaps instead we should simply allocate more bits in the  
> FieldInfo.  We could allocate bits for WEIGHT_PER_POSITION,  
> OFFSETS_IN_PRX, NORMS_IN_FRQ, OMIT_PRX, OMIT_FREQ, etc.  We can  
> increase the number of bits there by turning this into a VInt,  
> which would be back-compatible, no?

Using a VInt there sounds good 'n' clever to me.  Supporting all  
those different configs is another question.  "Flexibility is  
overrated." -- David Hansson.

It's charitable of you to include NORMS_IN_FRQ in that list, but in  
my mind, the idea was obsolesced the instant I saw  
WEIGHT_PER_POSITION.  Both enable fast launch of a Searcher.  That's  
the only benefit of NORMS_IN_FRQ, and it comes at the expense of  
increased file size in comparison to the same index without  
NORMS_IN_FRQ.  WEIGHT_PER_POSITION has much more potential.

My primary goal with enabling fast launch is to make it so only the  
largest installations have to worry about running under mod_perl and  
caching Searchers.  Simple installations, whether they are set up by  
a novice or by a sophisticated user who doesn't want to deal with  
mod_perl for a zillion possible reasons, should "just work" for as  
large an index as possible.  The main concern out-of-the-gate is ease  
of use, and I'm happy to trade off increased file size to get it.

To turn the idea on its head... I'm inclined to make  
WEIGHT_PER_POSITION the default behavior, and either deep-six the  
cacheable norms files altogether or "add" them as an "expert  
optimization": decreased file size and improved search speed at the  
expense of less control over scoring and slower startup.

Incidentally, how about calling it BOOST_PER_POSITION instead?

Marvin Humphrey
Rectangular Research
http://www.rectangular.com/


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Re: Rich positions (was "boosting fields")

Posted by Doug Cutting <cu...@apache.org>.
Marvin Humphrey wrote:
> Moving away from cached norms was the second of three major changes  to 
> the file format on my agenda, and the one I was all but certain I  
> wouldn't be able to sell to the Lucene community.  The first was  using 
> bytecounts at the head of Strings.
> 
> The third was storing start offsets and end offsets in the ProxFile.   
> It rankles that much of the information from tis/frq/prx gets  
> duplicated in the term vector files, but highlighting is most  efficient 
> when you know the offsets, and the primary index stops  short of storing 
> that information.  Currently, we have this:
> 
>     ProxFile (.prx) -->  <TermPositions>TermCount
> 
> How about this?
> 
>     ProxFile (.prx) -->  <TermPositions,TermOffsets>TermCount

This would at least double the size of the .prx file, the largest file 
in Lucene's index.  Yes it's useful, not not all folks will use it.  So 
not all folks should have to pay for it.  One way is to try to make it 
arbitrarily extensible, but to some degree, that's going to end up being 
language-specific.

So perhaps instead we should simply allocate more bits in the FieldInfo. 
  We could allocate bits for WEIGHT_PER_POSITION, OFFSETS_IN_PRX, 
NORMS_IN_FRQ, OMIT_PRX, OMIT_FREQ, etc.  We can increase the number of 
bits there by turning this into a VInt, which would be back-compatible, no?

Doug

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org