You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Marvin Humphrey <ma...@rectangular.com> on 2006/04/27 20:58:28 UTC
Rich positions (was "boosting fields")
On Apr 27, 2006, at 9:41 AM, Doug Cutting wrote:
> karl wettin wrote:
>> My own immediate thought is to compromise by allowing boost per
>> term in document. Simply remove the norms-methods from the
>> IndexReader and add a new one to the TermEnum and fall back on
>> the field boost. How would the value be picked up by the scorer?
>> Boost per position, et.c. sounds very expensive.
>
> Indeed. It will probably nearly double the size of indexes and
> also increase search time.
I have been considering making a similar change to the KinoSearch
file format. Not having to cache norms radically cuts down on the
time required to launch a fresh Searcher, especially if there aren't
any deleted docs. That's a win if you're launching a search app from
scratch, like if you're running a web search under CGI rather than
mod_perl. It's also a win for refreshing a Searcher against a
frequently updated index.
What I was considering was interleaving the document's score-
multiplier norm byte between the VInts in the .frq file. That would
mean more disk i/o for processing terms when the term takes up more
than a block on the file system, but at least the info would be
contiguous.
I hadn't considered interleaving the score-multiplier into .prx, but
that opens many possibilities. Boost positions that appear near the
top of the doc. Boost positions if they occur within certain HTML
tags. Good stuff!
Moving away from cached norms was the second of three major changes
to the file format on my agenda, and the one I was all but certain I
wouldn't be able to sell to the Lucene community. The first was
using bytecounts at the head of Strings.
The third was storing start offsets and end offsets in the ProxFile.
It rankles that much of the information from tis/frq/prx gets
duplicated in the term vector files, but highlighting is most
efficient when you know the offsets, and the primary index stops
short of storing that information. Currently, we have this:
ProxFile (.prx) --> <TermPositions>TermCount
How about this?
ProxFile (.prx) --> <TermPositions,TermOffsets>TermCount
To get highlighting info now, you retrieve a document's term vector
information and then extract the offsets information for the precise
term. This format reverses the order: first you find the term, then
you extract the offsets info for a particular doc.
I haven't implemented this change yet, so I'm not sure how it works
out. The current version of KinoSearch stores term vectors in
the .fdt file, which is a win for locality of reference. It sure
would be nice to eliminate all that duplicated data, though.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Rich positions (was "boosting fields")
Posted by Marvin Humphrey <ma...@rectangular.com>.
Now that I think about it, putting the score-multiplier into the
FreqFile does offer a benefit I hadn't considered before. It makes
it possible to tie the score multiplier to a term within a doc,
rather than a field within a doc.
Say you have a doc with a "body" field that's 1000 terms long, with 3
instances of "foo", right near the top. Say you have another doc
with a "body" that's 1000 terms long, and that it also has 3
instances of "foo" but they're buried near the end, and therefore not
as important.
Under the current implementation, these two docs will score
identically against a query for "body:foo", as the freq is identical
and so is the output of lengthNorm(1000). But if you stuff the score
multiplier into the FreqFile, a sophisticated indexing app could
assign a higher score multiplier to the term "body:foo" in the first
doc.
Associating a score multiplier with each position (a.k.a.
WEIGHT_PER_POSITION, BOOST_PER_POSITION) would achieve the same end,
but at the expense of much more processing per document, as the score
would have to be built up position by position. OTOH, it would
produce more accurate results for queries where only certain
positions ought to be considered, such as phrase queries.
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Rich positions (was "boosting fields")
Posted by Doug Cutting <cu...@apache.org>.
Marvin Humphrey wrote:
> Incidentally, how about calling it BOOST_PER_POSITION instead?
+1, that is more consistent with other naming.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Rich positions (was "boosting fields")
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Apr 27, 2006, at 12:17 PM, Doug Cutting wrote:
> Marvin Humphrey wrote:
>> Moving away from cached norms was the second of three major
>> changes to the file format on my agenda, and the one I was all
>> but certain I wouldn't be able to sell to the Lucene community.
>> The first was using bytecounts at the head of Strings.
>> The third was storing start offsets and end offsets in the
>> ProxFile. It rankles that much of the information from tis/frq/
>> prx gets duplicated in the term vector files, but highlighting is
>> most efficient when you know the offsets, and the primary index
>> stops short of storing that information. Currently, we have this:
>> ProxFile (.prx) --> <TermPositions>TermCount
>> How about this?
>> ProxFile (.prx) --> <TermPositions,TermOffsets>TermCount
>
> This would at least double the size of the .prx file, the largest
> file in Lucene's index. Yes it's useful, not not all folks will
> use it. So not all folks should have to pay for it.
Agreed. I think it would at least triple the ProxFile, actually. At
least, I haven't thought of a compression scheme which could cram
both start offset and end offset into fewer than two bytes on
average. But in theory, it would eliminate the need for the Term
Vectors files, so if you need those now only for highlighting, it's a
big gain.
> So perhaps instead we should simply allocate more bits in the
> FieldInfo. We could allocate bits for WEIGHT_PER_POSITION,
> OFFSETS_IN_PRX, NORMS_IN_FRQ, OMIT_PRX, OMIT_FREQ, etc. We can
> increase the number of bits there by turning this into a VInt,
> which would be back-compatible, no?
Using a VInt there sounds good 'n' clever to me. Supporting all
those different configs is another question. "Flexibility is
overrated." -- David Hansson.
It's charitable of you to include NORMS_IN_FRQ in that list, but in
my mind, the idea was obsolesced the instant I saw
WEIGHT_PER_POSITION. Both enable fast launch of a Searcher. That's
the only benefit of NORMS_IN_FRQ, and it comes at the expense of
increased file size in comparison to the same index without
NORMS_IN_FRQ. WEIGHT_PER_POSITION has much more potential.
My primary goal with enabling fast launch is to make it so only the
largest installations have to worry about running under mod_perl and
caching Searchers. Simple installations, whether they are set up by
a novice or by a sophisticated user who doesn't want to deal with
mod_perl for a zillion possible reasons, should "just work" for as
large an index as possible. The main concern out-of-the-gate is ease
of use, and I'm happy to trade off increased file size to get it.
To turn the idea on its head... I'm inclined to make
WEIGHT_PER_POSITION the default behavior, and either deep-six the
cacheable norms files altogether or "add" them as an "expert
optimization": decreased file size and improved search speed at the
expense of less control over scoring and slower startup.
Incidentally, how about calling it BOOST_PER_POSITION instead?
Marvin Humphrey
Rectangular Research
http://www.rectangular.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Rich positions (was "boosting fields")
Posted by Doug Cutting <cu...@apache.org>.
Marvin Humphrey wrote:
> Moving away from cached norms was the second of three major changes to
> the file format on my agenda, and the one I was all but certain I
> wouldn't be able to sell to the Lucene community. The first was using
> bytecounts at the head of Strings.
>
> The third was storing start offsets and end offsets in the ProxFile.
> It rankles that much of the information from tis/frq/prx gets
> duplicated in the term vector files, but highlighting is most efficient
> when you know the offsets, and the primary index stops short of storing
> that information. Currently, we have this:
>
> ProxFile (.prx) --> <TermPositions>TermCount
>
> How about this?
>
> ProxFile (.prx) --> <TermPositions,TermOffsets>TermCount
This would at least double the size of the .prx file, the largest file
in Lucene's index. Yes it's useful, not not all folks will use it. So
not all folks should have to pay for it. One way is to try to make it
arbitrarily extensible, but to some degree, that's going to end up being
language-specific.
So perhaps instead we should simply allocate more bits in the FieldInfo.
We could allocate bits for WEIGHT_PER_POSITION, OFFSETS_IN_PRX,
NORMS_IN_FRQ, OMIT_PRX, OMIT_FREQ, etc. We can increase the number of
bits there by turning this into a VInt, which would be back-compatible, no?
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org