You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucy.apache.org by Peter Karman <pe...@peknet.com> on 2012/08/28 05:09:45 UTC
[lucy-dev] custom scoring
I came up with several different subject lines for this email; I settled on the
one I would tend to search for later.
I spent last weekend working on a custom scoring project, involving subclasses
of Query, Compiler and Matcher[0]. It was a fateful trip, a real 3-hour tour.
My goal was to write the least amount of code possible in order to extend the
default TDF/IF weighting scheme, in order to achieve something like this:
package MyMatcher;
use base qw( Lucy::Search::Matcher );
sub score {
my $self = shift;
my $score = $self->SUPER::score(@_);
my $doc_reader = $self->get_doc_reader();
my $doc_id = $self->get_doc_id();
my $doc = $doc_reader->fetch_doc($doc_id);
return $score * $doc->{my_field_value};
}
If you grok the code above, you'll see that all I wanted to do was to affect the
score of a Doc at search time based on the value of a field in the Doc. That's
different than sorting by 'my_field_value' because I want the TDF/IF weighting
to still play a part in the score.
If you follow [0] you'll see that the "least amount of code possible" is
actually quite a lot of code. (Although it's quite possible I wrote more than I
needed to -- help very welcome.)
It was a good learning experience for me. However, I don't wish to impose that
learning experience on anyone else. I think it's time we take seriously Marvin's
long-standing desire to refactor how Query/Compiler/Matcher intersect. Reading
over [1] again I now understand more of where Nate was coming from, and I am
grateful for the ongoing dialog Marvin and Nate have had on this subject, as it
kept me company while I spelunked this weekend.
Here are some thoughts, in no particular order:
* I am intrigued by MatchEngine. Would it make what I'm trying to do above any
easier?
* It would be nice to have TermCompiler, TermMatcher (or whatever they end up
being called) made public so that it is easier to extend all the basic Query
types: PhraseQuery, RangeQuery, ProximityQuery. I ended up re-implementing all
the C logic in Perl which I just know is going to be much slower in tight loops
at search time.
* I actually like the original names (Weight, Scorer) more than Compiler and
Matcher. I understand the rationale for the change; the original names just have
more connotative meaning for me. Oh wait. I've been here before.[2] I have
changed my mind.
I started thinking of Compiler as WeightedQuery, and its relationship to Query
as similar to the relationship between Doc and HitDoc. I imagined code like:
my $query = $queryparser->parse('foo'); # $query isa Query
$query->apply_weight(searcher => $searcher); # $query isa WeightedQuery
if (!$query->weighted) {
die "can't score a search without weighting the query";
}
apply_weight() is like make_compiler(), I think. It can/should be called
internally in search(), but helpful when rolling your own.
* I was surprised to find that Compiler isa Query. That didn't really jive for
me with how Compiler keeps accessing parent(). I.e., a Query has-a Compiler. Why
must a Compiler is-a Query?
* The Matcher used by a TermCompiler is not a TermMatcher. It's a
PostlistSomeThingOrOther. Huh? I'm sure that's an optimization, but it still
caught be offguard and I had to hunt awhile.
I want to help make custom scoring easier to do, for me and for others. Is
MatchEngine the way forward? Help me understand.
[0]
https://github.com/karpet/search-query-dialect-lucy-perl/blob/master/t/03-compiler-matcher.t
[1] http://www.mail-archive.com/lucy-dev@incubator.apache.org/msg00342.html
[2] http://www.mail-archive.com/lucy-dev@incubator.apache.org/msg00310.html
--
Peter Karman . http://peknet.com/ . peter@peknet.com
[lucy-dev] [RFC] LucyX::Search::DelegateQuery [was Re: [lucy-dev] custom scoring]
Posted by Peter Karman <pe...@peknet.com>.
Thanks to Marvin's example code for custom Query classes, I have made a first
pass at an extension to ease the way for others:
https://github.com/karpet/lucyx-search-delegatequery
I'd appreciate feedback on the idea and implementation.
--
Peter Karman . http://peknet.com/ . peter@peknet.com
Re: [lucy-dev] custom scoring
Posted by Peter Karman <pe...@peknet.com>.
Marvin Humphrey wrote on 8/28/12 11:26 PM:
> On Tue, Aug 28, 2012 at 8:25 PM, Peter Karman <pe...@peknet.com> wrote:
>> in the current class setup, I would need something like this in make_matcher()
>> in MyCompiler then?
>>
>> sub make_matcher {
>> my $self = shift;
>> my $child_matcher = $self->SUPER::make_matcher(@_);
>> return MyMatcher->new( child => $child_matcher );
>> }
>>
>> or am I misunderstanding?
>
> I left a lot out. :)
[snip]
> Does that make sense?
>
it does. I'll play with that and see what I get.
thanks, Marvin.
--
Peter Karman . http://peknet.com/ . peter@peknet.com
Re: [lucy-dev] custom scoring
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Tue, Aug 28, 2012 at 8:25 PM, Peter Karman <pe...@peknet.com> wrote:
> in the current class setup, I would need something like this in make_matcher()
> in MyCompiler then?
>
> sub make_matcher {
> my $self = shift;
> my $child_matcher = $self->SUPER::make_matcher(@_);
> return MyMatcher->new( child => $child_matcher );
> }
>
> or am I misunderstanding?
I left a lot out. :)
I'd suggest generalizing your class and having it wrap an arbitrary Query.
{
package MyQuery;
use base qw( Lucy::Search::Query );
my %child;
sub new {
my ($class, %args) = @_;
my $child = delete $args{child};
confess "not a Query"
unless (blessed($child)
and $child->isa("Lucy::Search::Query"));
my $self = $class->SUPER::new(%args);
$child{$$self} = $child;
return $self;
}
...
sub make_compiler {
my ($self, %args) = @_;
my $child_compiler = $child{$$self}->make_compiler(%args);
my $compiler = MyCompiler->new(child => $child_compiler);
$compiler->normalize unless $args{subordinate};
return $compiler;
}
}
{
package MyCompiler;
use base qw( Lucy::Search::Compiler );
my %child;
sub new {
my ($class, %args) = @_;
my $child = delete $args{child};
my $self = $class->SUPER::new(%args);
$child{$$self} = $child;
return $self;
}
...
sub make_matcher {
my ($self, %args) = @_;
my $child_matcher = $child{$$self}->make_matcher(%args);
return unless $child_matcher;
my $sort_reader = $args{reader}->obtain("Lucy::Index::SortReader");
my $sort_cache = $sort_reader->fetch_sort_cache("magic");
return MyMatcher->new(
child => $child_matcher,
sort_cache => $sort_cache,
);
}
}
Does that make sense?
> I like the idea of making SortReader and SortCache public, just because I
> imagine it would be nice abuse them like this for other reasons. But I
> understand the once-public-always-public threshold.
It's so challenging to evolve sensible APIs...
Marvin Humphrey
Re: [lucy-dev] custom scoring
Posted by Peter Karman <pe...@peknet.com>.
Marvin Humphrey wrote on 8/28/12 11:14 AM:
> On Tue, Aug 28, 2012 at 7:47 AM, Peter Karman <pe...@peknet.com> wrote:
>> On 8/28/12 1:00 AM, Marvin Humphrey wrote:
>>> FWIW, speed only matters for Matchers.
>>
>> even when iterating over a PostingList? That's where I was expecting the
>> biggest perf hit. But that's with no evidence at all.
>
> Oh, sheesh, you're right about that. Correction:
>
> Speed generally doesn't matter for any activity that scales with the number of
> segments, rather than the number of documents.
>
> The make_matcher() method gets invoked once per segment, so it ordinarily is
> not a performance bottleneck. But in your case, you're iterating over a
> posting list within make_matcher(), an activity which scales with the number
> of documents.
/me nods
>
> FWIW, if we make SortReader and SortCache public, there's a more efficient way
> to do what you're doing: wrap a child Matcher and modify or override the score
> depending on what we find is the value in the SortCache for the doc id being
> scored.
>
> package MyMatcher;
> use base qw( Lucy::Search::Matcher );
>
> my %child;
> my %sort_cache;
>
> sub new {
> my ($class, %args) = @_;
> my $sort_cache = delete $args{sort_cache};
> my $child = delete $args{child};
in the current class setup, I would need something like this in make_matcher()
in MyCompiler then?
sub make_matcher {
my $self = shift;
my $child_matcher = $self->SUPER::make_matcher(@_);
return MyMatcher->new( child => $child_matcher );
}
or am I misunderstanding?
I don't think that SUPER::make_matcher() call works unless I've inherited from
TermCompiler instead of just Compiler.
> my $self = $class->SUPER::new(%args);
> $sort_cache{$$self} = $sort_cache;
> $child{$$self} = $child;
> return $self;
> }
>
> sub DESTROY {
> my $self = shift;
> delete $child{$$self};
> delete $sort_cache{$$self};
> $self->SUPER::DESTROY;
> }
>
> # Delegate next() and get_doc_id() to the child Matcher.
> sub next { $child{ ${+shift} }->next }
> sub get_doc_id { $child{ ${+shift} }->get_doc_id }
>
> my %magic_scores = (
> a => 100,
> b => 200,
> c => 300,
> d => 400,
> );
>
> sub score {
> my $self = shift;
>
> # Try for special score.
> my $doc_id = $self->get_doc_id;
> my $ord = $sort_cache{$$self}->ordinal($doc_id);
> my $value = $sort_cache{$$self}->value($ord);
> if ($value) {
> my $magic_score = $magic_scores{$value};
> return $magic_score if $magic_score;
> }
>
> # Fall back to child Matcher's score.
> return $child_matcher{$$self}->score;
> }
>
> Wrapping a child Query would also allow you to eliminate all that TFIDF stuff
> in your subclass.
>
> The same basic approach can work when fetching values with a DocReader instead
> of a SortCache. There would be some kind of speed loss for deserializing the
> entire document rather than fetching a value from a sort cache, but DocReader
> is already public.
>
I like the idea of making SortReader and SortCache public, just because I
imagine it would be nice abuse them like this for other reasons. But I
understand the once-public-always-public threshold.
--
Peter Karman . http://peknet.com/ . peter@peknet.com
Re: [lucy-dev] custom scoring
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Tue, Aug 28, 2012 at 7:47 AM, Peter Karman <pe...@peknet.com> wrote:
> On 8/28/12 1:00 AM, Marvin Humphrey wrote:
>> FWIW, speed only matters for Matchers.
>
> even when iterating over a PostingList? That's where I was expecting the
> biggest perf hit. But that's with no evidence at all.
Oh, sheesh, you're right about that. Correction:
Speed generally doesn't matter for any activity that scales with the number of
segments, rather than the number of documents.
The make_matcher() method gets invoked once per segment, so it ordinarily is
not a performance bottleneck. But in your case, you're iterating over a
posting list within make_matcher(), an activity which scales with the number
of documents.
FWIW, if we make SortReader and SortCache public, there's a more efficient way
to do what you're doing: wrap a child Matcher and modify or override the score
depending on what we find is the value in the SortCache for the doc id being
scored.
package MyMatcher;
use base qw( Lucy::Search::Matcher );
my %child;
my %sort_cache;
sub new {
my ($class, %args) = @_;
my $sort_cache = delete $args{sort_cache};
my $child = delete $args{child};
my $self = $class->SUPER::new(%args);
$sort_cache{$$self} = $sort_cache;
$child{$$self} = $child;
return $self;
}
sub DESTROY {
my $self = shift;
delete $child{$$self};
delete $sort_cache{$$self};
$self->SUPER::DESTROY;
}
# Delegate next() and get_doc_id() to the child Matcher.
sub next { $child{ ${+shift} }->next }
sub get_doc_id { $child{ ${+shift} }->get_doc_id }
my %magic_scores = (
a => 100,
b => 200,
c => 300,
d => 400,
);
sub score {
my $self = shift;
# Try for special score.
my $doc_id = $self->get_doc_id;
my $ord = $sort_cache{$$self}->ordinal($doc_id);
my $value = $sort_cache{$$self}->value($ord);
if ($value) {
my $magic_score = $magic_scores{$value};
return $magic_score if $magic_score;
}
# Fall back to child Matcher's score.
return $child_matcher{$$self}->score;
}
Wrapping a child Query would also allow you to eliminate all that TFIDF stuff
in your subclass.
The same basic approach can work when fetching values with a DocReader instead
of a SortCache. There would be some kind of speed loss for deserializing the
entire document rather than fetching a value from a sort cache, but DocReader
is already public.
Marvin Humphrey
Re: [lucy-dev] custom scoring
Posted by Peter Karman <pe...@peknet.com>.
On 8/28/12 1:00 AM, Marvin Humphrey wrote:
> (I've divided my reply in half for the sake of digestibility...)
>
> On Mon, Aug 27, 2012 at 8:09 PM, Peter Karman<pe...@peknet.com> wrote:
>> If you follow [0] you'll see that the "least amount of code possible" is
>> actually quite a lot of code. (Although it's quite possible I wrote more
>> than I needed to -- help very welcome.)
>
> I looked, and I don't see a lot of ways to shrink that code. (Should `DELETE`
> have been `DESTROY`, though?)
ah yes, thank you. /me suffering from context-switching overload.
>> * It would be nice to have TermCompiler, TermMatcher (or whatever they end up
>> being called) made public so that it is easier to extend all the basic Query
>> types: PhraseQuery, RangeQuery, ProximityQuery. I ended up re-implementing all
>> the C logic in Perl which I just know is going to be much slower in tight loops
>> at search time.
>
> FWIW, speed only matters for Matchers.
even when iterating over a PostingList? That's where I was expecting the
biggest perf hit. But that's with no evidence at all.
Or maybe I should be doing that in Matcher instead of Compiler?
>
> Under the MatchEngine model, Lucy::Search::TermCompiler would become
> Lucy::TFIDF::TFIDFTermQuery and would no longer be linked directly with
> TermQuery (TermQuery_Make_Compiler() would be eliminated, and its replacement
> TermQuery_Make_Weighted_Query() would either be an abstract method or return
> NULL.) Once the connection to TermQuery is severed, there's less risk in
> making that class public.
cool.
>
>> * I actually like the original names (Weight, Scorer) more than Compiler and
>> Matcher. I understand the rationale for the change; the original names just have
>> more connotative meaning for me. Oh wait. I've been here before.[2] I have
>> changed my mind.
>
> Heh. I'm pleased with "Matcher". "Compiler" was a mistake, but "Weight" also
> sucks.
>
> Hopefully, we can just eliminate Compiler and break the impasse.
+1
>
>> * I was surprised to find that Compiler isa Query. That didn't really jive for
>> me with how Compiler keeps accessing parent(). I.e., a Query has-a Compiler. Why
>> must a Compiler is-a Query?
>
> This actually has more to do with Searcher. If a Compiler wasn't a Query,
> we'd need either additional parameters or additional methods inside Searcher.
>
> In Lucene, most search methods have two implentations, differentiated by
> method signature overloading -- one of them takes a Query, and the other takes
> a Weight. The method which takes a Query manufactures a Weight and dispatches
> to the method which takes a Weight.
>
> (If you think Lucy is convoluted with regards to Query processing inside
> Searcher, try spelunking Lucene sometime -- it's _way_ worse.)
/me takes your word for it.
>
>> * The Matcher used by a TermCompiler is not a TermMatcher. It's a
>> PostlistSomeThingOrOther. Huh? I'm sure that's an optimization, but it still
>> caught be offguard and I had to hunt awhile.
>
> If we go with MatchEngine and eliminate pluggable posting support, I think we
> can rip out those layers of indirection.
>
> Before:
>
> ScorePostingMatcher, which subclasses TermMatcher and wraps a
> SegPostingList which wraps an InStream.
>
> After:
>
> TFIDFTermMatcher which wraps an InStream.
>
cool.
--
Peter Karman . http://peknet.com/ . peter@peknet.com
Re: [lucy-dev] custom scoring
Posted by Marvin Humphrey <ma...@rectangular.com>.
(I've divided my reply in half for the sake of digestibility...)
On Mon, Aug 27, 2012 at 8:09 PM, Peter Karman <pe...@peknet.com> wrote:
> If you follow [0] you'll see that the "least amount of code possible" is
> actually quite a lot of code. (Although it's quite possible I wrote more
> than I needed to -- help very welcome.)
I looked, and I don't see a lot of ways to shrink that code. (Should `DELETE`
have been `DESTROY`, though?)
> It was a good learning experience for me. However, I don't wish to impose that
> learning experience on anyone else.
Some of this is TFIDF. Some of it is extra layers of indirection within a
compiled TermQuery... There isn't a silver bullet, but maybe we can make some
progress...
> * It would be nice to have TermCompiler, TermMatcher (or whatever they end up
> being called) made public so that it is easier to extend all the basic Query
> types: PhraseQuery, RangeQuery, ProximityQuery. I ended up re-implementing all
> the C logic in Perl which I just know is going to be much slower in tight loops
> at search time.
FWIW, speed only matters for Matchers.
Under the MatchEngine model, Lucy::Search::TermCompiler would become
Lucy::TFIDF::TFIDFTermQuery and would no longer be linked directly with
TermQuery (TermQuery_Make_Compiler() would be eliminated, and its replacement
TermQuery_Make_Weighted_Query() would either be an abstract method or return
NULL.) Once the connection to TermQuery is severed, there's less risk in
making that class public.
> * I actually like the original names (Weight, Scorer) more than Compiler and
> Matcher. I understand the rationale for the change; the original names just have
> more connotative meaning for me. Oh wait. I've been here before.[2] I have
> changed my mind.
Heh. I'm pleased with "Matcher". "Compiler" was a mistake, but "Weight" also
sucks.
Hopefully, we can just eliminate Compiler and break the impasse.
> * I was surprised to find that Compiler isa Query. That didn't really jive for
> me with how Compiler keeps accessing parent(). I.e., a Query has-a Compiler. Why
> must a Compiler is-a Query?
This actually has more to do with Searcher. If a Compiler wasn't a Query,
we'd need either additional parameters or additional methods inside Searcher.
In Lucene, most search methods have two implentations, differentiated by
method signature overloading -- one of them takes a Query, and the other takes
a Weight. The method which takes a Query manufactures a Weight and dispatches
to the method which takes a Weight.
(If you think Lucy is convoluted with regards to Query processing inside
Searcher, try spelunking Lucene sometime -- it's _way_ worse.)
> * The Matcher used by a TermCompiler is not a TermMatcher. It's a
> PostlistSomeThingOrOther. Huh? I'm sure that's an optimization, but it still
> caught be offguard and I had to hunt awhile.
If we go with MatchEngine and eliminate pluggable posting support, I think we
can rip out those layers of indirection.
Before:
ScorePostingMatcher, which subclasses TermMatcher and wraps a
SegPostingList which wraps an InStream.
After:
TFIDFTermMatcher which wraps an InStream.
Marvin Humphrey
Re: [lucy-dev] custom scoring
Posted by Peter Karman <pe...@peknet.com>.
Marvin Humphrey wrote on 8/28/12 12:56 AM:
> On Mon, Aug 27, 2012 at 8:09 PM, Peter Karman <pe...@peknet.com> wrote:
>> * I am intrigued by MatchEngine. Would it make what I'm trying to do above
>> any easier?
>
> While it wouldn't diminish the raw volume of code, I think it would make
> things easier to understand.
>
> * It allows us to eliminate Compiler.
> * People creating custom query subclasses would have the option of writing
> only a Query/Matcher pair rather than a Query/weighted-Query/Matcher trio.
I like both those things.
>
> Here's a rapid prototype for TFIDFMatchEngine:
>
> package Lucy::TFIDF::TFIDFMatchEngine;
> use base qw( Lucy::Search::MatchEngine );
>
> sub weight_query {
> my ($self, %args) = @_;
> my ($query, $searcher) = @args{qw( query searcher )};
> my $query_class = ref($query);
> if ($query_class eq 'Lucy::Search::TermQuery') {
> return Lucy::TFIDF::TFIDFTermQuery->new(
> parent => $query,
> searcher => $searcher,
> );
> }
> elsif ($query_class eq 'Lucy::Search::PhraseQuery') {
> return Lucy::TFIDF::TFIDFPhraseQuery->new(
> parent => $query,
> searcher => $searcher,
> );
> }
> ... # ...and so on for all core Query classes.
> else {
> # Unknown Query class, so fall back to delegating weighting to the
> # Query itself. (Since TFIDFMatchEngine doesn't know how to
> # produce a weighted query for this Query class, we have to hope
> # that this is a custom Query class written with TFIDFMatchEngine
> # as a target and that it knows how to create a weighted query
> # appropriate for TFIDFMatchEngine.)
> return $query->make_weighted_query(
> searcher => $searcher
> match_engine => $self,
> );
> }
> }
>
ok, that all makes sense to me. Could we make the Lucy::TFIDF::* classes public
so that (as in my case) they could be extended, rather than just replaced?
>> I started thinking of Compiler as WeightedQuery, and its relationship to Query
>> as similar to the relationship between Doc and HitDoc. I imagined code like:
>>
>> my $query = $queryparser->parse('foo'); # $query isa Query
>> $query->apply_weight(searcher => $searcher); # $query isa WeightedQuery
>> if (!$query->weighted) {
>> die "can't score a search without weighting the query";
>> }
>
> FWIW, mutating Query objects is what Lucene originally did, back in
> version 1.3 IIRC. :P
>
man. I am so old and in the weigh.
> Here's something very similar:
>
> my $query = $queryparser->parse('foo');
> my $weighted_query = $match_engine->weight_query(
> searcher => $searcher,
> query => $query
> );
> if (!$weighted_query->weighted) {
> die "can't score a search without weighting the query";
> }
>
> The difference is that while your sample code mutates $query, this version
> uses a factory method which generates a new $weighted_query and leaves the
> original $query untouched.
>
> There are a lot of reasons that we want to avoid mutating the original Query
> object. For starters, we have long wanted to sever the connection between our
> Query classes and TFIDF (or any other scoring model) -- but if they must hold
> weighting information, they have to know about the scoring model.
>
+1 I get why the mutation is to be avoided.
--
Peter Karman . http://peknet.com/ . peter@peknet.com
Re: [lucy-dev] custom scoring
Posted by Marvin Humphrey <ma...@rectangular.com>.
On Mon, Aug 27, 2012 at 8:09 PM, Peter Karman <pe...@peknet.com> wrote:
> * I am intrigued by MatchEngine. Would it make what I'm trying to do above
> any easier?
While it wouldn't diminish the raw volume of code, I think it would make
things easier to understand.
* It allows us to eliminate Compiler.
* People creating custom query subclasses would have the option of writing
only a Query/Matcher pair rather than a Query/weighted-Query/Matcher trio.
Here's a rapid prototype for TFIDFMatchEngine:
package Lucy::TFIDF::TFIDFMatchEngine;
use base qw( Lucy::Search::MatchEngine );
sub weight_query {
my ($self, %args) = @_;
my ($query, $searcher) = @args{qw( query searcher )};
my $query_class = ref($query);
if ($query_class eq 'Lucy::Search::TermQuery') {
return Lucy::TFIDF::TFIDFTermQuery->new(
parent => $query,
searcher => $searcher,
);
}
elsif ($query_class eq 'Lucy::Search::PhraseQuery') {
return Lucy::TFIDF::TFIDFPhraseQuery->new(
parent => $query,
searcher => $searcher,
);
}
... # ...and so on for all core Query classes.
else {
# Unknown Query class, so fall back to delegating weighting to the
# Query itself. (Since TFIDFMatchEngine doesn't know how to
# produce a weighted query for this Query class, we have to hope
# that this is a custom Query class written with TFIDFMatchEngine
# as a target and that it knows how to create a weighted query
# appropriate for TFIDFMatchEngine.)
return $query->make_weighted_query(
searcher => $searcher
match_engine => $self,
);
}
}
> I started thinking of Compiler as WeightedQuery, and its relationship to Query
> as similar to the relationship between Doc and HitDoc. I imagined code like:
>
> my $query = $queryparser->parse('foo'); # $query isa Query
> $query->apply_weight(searcher => $searcher); # $query isa WeightedQuery
> if (!$query->weighted) {
> die "can't score a search without weighting the query";
> }
FWIW, mutating Query objects is what Lucene originally did, back in
version 1.3 IIRC. :P
Here's something very similar:
my $query = $queryparser->parse('foo');
my $weighted_query = $match_engine->weight_query(
searcher => $searcher,
query => $query
);
if (!$weighted_query->weighted) {
die "can't score a search without weighting the query";
}
The difference is that while your sample code mutates $query, this version
uses a factory method which generates a new $weighted_query and leaves the
original $query untouched.
There are a lot of reasons that we want to avoid mutating the original Query
object. For starters, we have long wanted to sever the connection between our
Query classes and TFIDF (or any other scoring model) -- but if they must hold
weighting information, they have to know about the scoring model.
Marvin Humphrey