theoretical maximum score

[Peter Keegan <pe...@gmail.com>]

Is it possible to compute a theoretical maximum score for a given query if
constraints are placed on 'tf' and 'lengthNorm'? If so, scores could be
compared to a 'perfect score' (a feature request from our customers)

Here are some related threads on this:

In this thread:

http://www.nabble.com/Newbie-questions-re%3A-scoring-td4228776.html#a4228776

Hoss writes:

> the only way I can think of to fairly compare scores from queries for
> foo:bar with queries for yak:baz is to normalize them relative a maximum
> possible score across the entire term query space -- but finding that
> maximum is a pretty complicated problem just for simple term queries ...
> when you start talking about more complicated query structures you really
> get messy -- and even then it's only fair as long as the query structures
> are identical, you can never compare the scores from apples and oranges

And in this thread:

http://www.nabble.com/non-relative-scoring-td8956299.html#a8956299

Walt writes:

> A tf.idf engine, like Lucene, might not have a maximum score.
> What if a document contains the word a thousand times?
> A million times?

It seems that if 'tf' is limited to a max value and 'lengthNorm' is a
constant, it might be possible, at least for 'simple' term queries. But Hoss
says that things get messing with complicated queries.

Could someone elaborate a bit? Does the index contain enough info to do this
efficiently?
I realize that scores values must be interpreted 'carefully', but I'm seeing
a push to get more leverage from the absolute values, not just the relative
values.

Peter

Re: theoretical maximum score

[Paul Elschot <pa...@xs4all.nl>]

Op Saturday 17 May 2008 00:04:31 schreef Chris Hostetter:
> : Is it possible to compute a theoretical maximum score for a given
> : query if constraints are placed on 'tf' and 'lengthNorm'? If so,
> : scores could be compared to a 'perfect score' (a feature request
> : from our customers)
 
I think a theoretical maximum score is only going to work when that
maximum applies to queries of any structure. So, start with the
simplest query, associate it with a theoretical maximum score, and
then for each possible combination of subqueries ((weighted)
and/or/phrase/span) make sure that the subscore values are
combined into another value that has the same theoretical
maximum.

Have a look here to start:
https://issues.apache.org/jira/browse/LUCENE-293

Regards,
Paul Elschot

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: theoretical maximum score

[Chris Hostetter <ho...@fucit.org>]

: Is it possible to compute a theoretical maximum score for a given query if
: constraints are placed on 'tf' and 'lengthNorm'? If so, scores could be
: compared to a 'perfect score' (a feature request from our customers)

without thinking about it two hard, you'd also need to constrain:
 * field and document boosts (which are combined with lengthNorm to create 
   the fieldNorm)
 * query time boosts
 * idf (there's no law that says Similarity.idf has to return a number 
   less then 1) 
 * queryNorm

...you'd also need to use query structures that are simple -- a 
ValueSourceQuery can break all the rules it wants -- but if you used only 
basic types of queries (boolean, term, phrase) and you imposed those 
constraints you could probably pull it off.

the key thing to watch out for is that even if you can do it, and you can 
start to say meaninful things like "doc A scored X out of a max possible Y 
against query Q" that doesn't neccessarily help you compare that with doc 
B which scored V out of a max possible W against query R ... even if X/Y == V/W 
... because the *structure* and complexity of Q and R play havock 
with the scores.

but comparisons like that are what people are going to start to do as 
soome as you give them a number like that.  People are going to start to 
tihnk "well doc A is an N% match for Q, and doc B is an N% match for R, 
but A is clearly a better match Q then B is for R .. what the heck?"

...i would do a lot of "subjective" testing using a variety of queries of 
various complexities before i put any faith in producing a number like 
that.

I suspect what you'd find is that the constraints you need to impose in 
order to give you a meaningful number wind up hobbling Lucene so much it 
doesn't do a very good job of scoring anything.




-Hoss


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org