You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Александр Аристов <cl...@mail.ru> on 2008/08/07 21:05:41 UTC
Re[2]: lucene scoring
I want implement searching with ability to set so-called a confidence level below which I would treat documents as garbage. I cannot defile the level per query as the level should be relevant for all documents.
With current scoring implementation the level would mean nothing. I don't believe that since that time (the thread is of 2005year) nothing has been made towards the resolving the issue.
Do you think any workarounds like implementing more sophisticated queries so that we have approximately the same normalization values?
Alex
-----Original Message-----
From: Grant Ingersoll <gs...@apache.org>
To: java-dev@lucene.apache.org
Date: Thu, 7 Aug 2008 10:20:10 -0400
Subject: Re: lucene scoring
My understanding is this is an area of research in Info. Retrieval in
general. There is some attempt at this with the query normalization
factor in the scoring model, but my understanding is one shouldn't
rely on it. You might try searching Google scholar (or MSN Academic
Live, which I prefer :-) ). I think Hoss's response in that thread
you link, however, is right on. The scores for Lucene are only meant
for comparison within the given search result. I would even go so far
as to say the scores may not be comparable for the same query done at
different times, unless absolutely nothing has changed in the index.
My question back is what are you trying to do that you think you need
it?
On Aug 7, 2008, at 2:10 AM, Александр Аристов wrote:
> Hi people
>
> What is the best way to implement scoring so that it become possible
> to compare scores obtained from different queries. Full problem
> description is here (clear and short) http://thread.gmane.org/gmane.comp.jakarta.lucene.user/10760/focus=10810
>
> I know about possible usage of TopFieldDocCollector or disabling the
> Hits class but how will we normalize the results?
>
> thanks
>
> Александр
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Re[2]: lucene scoring
Posted by Doron Cohen <cd...@gmail.com>.
Following suggestion is weaker than the requested functionality, but
maybe you'll find the concept useful to ignore so called "garbage" results.
Assume that the query is a simple OR query made of a few words.
By examining the frequencies of these words in the index
(their DFs) devise a synthetic document which is the worst
document you will be willing to accept as a useful result.
Alternatively ignore DFs, but create a few documents like this -
each perhaps containing one or few of the query words (and likely
many other words). Now virtually add the synthetic document(s)
to the index. Can be done by creating a small in memory index,
and creating a multiIndexReader on top of the real index and
the dummy one. Now execute the query, with a filter that
accepts only the synthetic documents. The score of the worst
acceptable document(s) can be used as a threshold when
running the query on the original index.
It is inefficient - should be done for each query, and would be hard
to implement for general queries, and I never tried it...
Doron
2008/8/8 Александр Аристов <cl...@mail.ru>
> Query independent means that the threshold should have the same relevance
> for all queries and discard found docs below it. Current scoring
> implementation doesn't give guaranties that, say two documents found in two
> queries and which got the same score 0.5 are of the same quality.
>
> I don't want discarding docs from being indexed, no. But I want to be sure
> that two docs with the same score in two different queries have the same
> quality (they contain the same set of found terms, lenght etc.)
>
> Alexander
>
> -----Original Message-----
> From: Andrzej Bialecki <ab...@getopt.org>
> To: java-dev@lucene.apache.org
> Date: Thu, 07 Aug 2008 22:44:46 +0200
> Subject: Re: lucene scoring
>
>
> Александр Аристов wrote:
> > I want implement searching with ability to set so-called a confidence
> > level below which I would treat documents as garbage. I cannot defile
> > the level per query as the level should be relevant for all
> > documents.
>
> Hmm .. I'm not sure if I understand it properly - if the level is
> query-independent, then it's a constant factor, which you can put in a
> field during the index creation - and then you could use a Filter or
> FunctionQuery to exclude documents with this factor below the threshold.
>
> --
> Best regards,
> Andrzej Bialecki <><
> ___. ___ ___ ___ _ _ __________________________________
> [__ || __|__/|__||\/| Information Retrieval, Semantic Web
> ___|||__|| \| || | Embedded Unix, System Integration
> http://www.sigram.com Contact: info at sigram dot com
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re[2]: lucene scoring
Posted by Александр Аристов <cl...@mail.ru>.
Query independent means that the threshold should have the same relevance for all queries and discard found docs below it. Current scoring implementation doesn't give guaranties that, say two documents found in two queries and which got the same score 0.5 are of the same quality.
I don't want discarding docs from being indexed, no. But I want to be sure that two docs with the same score in two different queries have the same quality (they contain the same set of found terms, lenght etc.)
Alexander
-----Original Message-----
From: Andrzej Bialecki <ab...@getopt.org>
To: java-dev@lucene.apache.org
Date: Thu, 07 Aug 2008 22:44:46 +0200
Subject: Re: lucene scoring
Александр Аристов wrote:
> I want implement searching with ability to set so-called a confidence
> level below which I would treat documents as garbage. I cannot defile
> the level per query as the level should be relevant for all
> documents.
Hmm .. I'm not sure if I understand it properly - if the level is
query-independent, then it's a constant factor, which you can put in a
field during the index creation - and then you could use a Filter or
FunctionQuery to exclude documents with this factor below the threshold.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: lucene scoring
Posted by Andrzej Bialecki <ab...@getopt.org>.
Александр Аристов wrote:
> I want implement searching with ability to set so-called a confidence
> level below which I would treat documents as garbage. I cannot defile
> the level per query as the level should be relevant for all
> documents.
Hmm .. I'm not sure if I understand it properly - if the level is
query-independent, then it's a constant factor, which you can put in a
field during the index creation - and then you could use a Filter or
FunctionQuery to exclude documents with this factor below the threshold.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Re[4]: lucene scoring
Posted by "J. Delgado" <jo...@gmail.com>.
The only score that I can think of that can measure "quality" across
different queries are invariant scores such as pagerank. That is to score
the document on its general information value and then use that as a filter
regardless of the query. This is very different than the problem of trying
to nomalize the score on the same query over different shards (indexes) in a
federated query setting, which has been researched extensively.
The reason why two queries have different "scale" for scores is because of
the probabilistic nature of the algorithms which view word occurences as
independent random variables. Thus the occurence of each word in a document
is treated as an independent event. Joint and conditional probabilities can
estimated looking at word co-occurence, which could be used to compare two
specific results (i.e. how relevant is document X to both "baby kittens" and
"death metal" or if "baby kitten" is present in a doc how likely is that
"death metal" is present too), but to use the TF-IDF based score as as
absolute measure is like trying to compare Pears with Apples. Trying to
nomalize it is an ill-defined task.
-- J.D.
2008/8/8 Александр Аристов <cl...@mail.ru>
> Relevance ranking is an option but we still won't be able compare results.
> Lets say we have distributed searching - in this case top 10 from one server
> is not the same as those which are from another. Even worse we may get that
> in the resulting set a document with most top score is worse than others.
>
> what if we disable normalization or make it constant will results be
> absolutely dummy?
>
> And anther approach, can we calculate most possible top value? Or just
> maybe approximation of it? we then would be able to compare results with it.
>
> Alex
>
>
> -----Original Message-----
> From: Grant Ingersoll <gs...@apache.org>
> To: java-dev@lucene.apache.org
> Date: Thu, 7 Aug 2008 15:54:41 -0400
> Subject: Re: Re[2]: lucene scoring
>
>
> On Aug 7, 2008, at 3:05 PM, Александр Аристов wrote:
>
> > I want implement searching with ability to set so-called a
> > confidence level below which I would treat documents as garbage. I
> > cannot defile the level per query as the level should be relevant
> > for all documents.
> >
> > With current scoring implementation the level would mean nothing. I
> > don't believe that since that time (the thread is of 2005year)
> > nothing has been made towards the resolving the issue.
>
> That's because there is no resolution to be had, as far as I know, but
> I'm open to suggestions (patches are even better.) What would it mean
> to say that a score of 0.5 for "baby kittens" is comparable to a score
> of 0.5 for "death metal"? Like I said, I don't think that 0.5 for
> "baby kittens" is even comparable later if you added other documents
> that contain any of the query terms.
>
> >
> >
> > Do you think any workarounds like implementing more sophisticated
> > queries so that we have approximately the same normalization values?
>
> I just don't think you will be successful with this, and I don't
> believe it is a Lucene issue alone, but one that applies to all search
> engines, but I could be wrong.
>
> I get what you are trying to do, though, I've wanted to do it from
> time to time. Another approach may be to look for significant
> differences between scores w/in a result set. For example, if doc 1
> is 0.8, doc 2 is 0.79 and then doc 3 is 0.2, then maybe one could
> argue that doc 3 is garbage, but even that is somewhat of a stretch.
> Garbage truly is in the eye of the beholder.
>
> Another option is to do more relevance tuning to make sure your top 10
> are as good as possible so that your garbage is minimized.
>
> -Grant
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
Re[4]: lucene scoring
Posted by Александр Аристов <cl...@mail.ru>.
Relevance ranking is an option but we still won't be able compare results. Lets say we have distributed searching - in this case top 10 from one server is not the same as those which are from another. Even worse we may get that in the resulting set a document with most top score is worse than others.
what if we disable normalization or make it constant will results be absolutely dummy?
And anther approach, can we calculate most possible top value? Or just maybe approximation of it? we then would be able to compare results with it.
Alex
-----Original Message-----
From: Grant Ingersoll <gs...@apache.org>
To: java-dev@lucene.apache.org
Date: Thu, 7 Aug 2008 15:54:41 -0400
Subject: Re: Re[2]: lucene scoring
On Aug 7, 2008, at 3:05 PM, Александр Аристов wrote:
> I want implement searching with ability to set so-called a
> confidence level below which I would treat documents as garbage. I
> cannot defile the level per query as the level should be relevant
> for all documents.
>
> With current scoring implementation the level would mean nothing. I
> don't believe that since that time (the thread is of 2005year)
> nothing has been made towards the resolving the issue.
That's because there is no resolution to be had, as far as I know, but
I'm open to suggestions (patches are even better.) What would it mean
to say that a score of 0.5 for "baby kittens" is comparable to a score
of 0.5 for "death metal"? Like I said, I don't think that 0.5 for
"baby kittens" is even comparable later if you added other documents
that contain any of the query terms.
>
>
> Do you think any workarounds like implementing more sophisticated
> queries so that we have approximately the same normalization values?
I just don't think you will be successful with this, and I don't
believe it is a Lucene issue alone, but one that applies to all search
engines, but I could be wrong.
I get what you are trying to do, though, I've wanted to do it from
time to time. Another approach may be to look for significant
differences between scores w/in a result set. For example, if doc 1
is 0.8, doc 2 is 0.79 and then doc 3 is 0.2, then maybe one could
argue that doc 3 is garbage, but even that is somewhat of a stretch.
Garbage truly is in the eye of the beholder.
Another option is to do more relevance tuning to make sure your top 10
are as good as possible so that your garbage is minimized.
-Grant
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Re[2]: lucene scoring
Posted by Grant Ingersoll <gs...@apache.org>.
On Aug 7, 2008, at 3:05 PM, Александр Аристов wrote:
> I want implement searching with ability to set so-called a
> confidence level below which I would treat documents as garbage. I
> cannot defile the level per query as the level should be relevant
> for all documents.
>
> With current scoring implementation the level would mean nothing. I
> don't believe that since that time (the thread is of 2005year)
> nothing has been made towards the resolving the issue.
That's because there is no resolution to be had, as far as I know, but
I'm open to suggestions (patches are even better.) What would it mean
to say that a score of 0.5 for "baby kittens" is comparable to a score
of 0.5 for "death metal"? Like I said, I don't think that 0.5 for
"baby kittens" is even comparable later if you added other documents
that contain any of the query terms.
>
>
> Do you think any workarounds like implementing more sophisticated
> queries so that we have approximately the same normalization values?
I just don't think you will be successful with this, and I don't
believe it is a Lucene issue alone, but one that applies to all search
engines, but I could be wrong.
I get what you are trying to do, though, I've wanted to do it from
time to time. Another approach may be to look for significant
differences between scores w/in a result set. For example, if doc 1
is 0.8, doc 2 is 0.79 and then doc 3 is 0.2, then maybe one could
argue that doc 3 is garbage, but even that is somewhat of a stretch.
Garbage truly is in the eye of the beholder.
Another option is to do more relevance tuning to make sure your top 10
are as good as possible so that your garbage is minimized.
-Grant
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org