[jira] [Commented] (LUCENE-8216) Better cross-field scoring

["Alessandro Benedetti (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-8216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17357258#comment-17357258 ] 

Alessandro Benedetti commented on LUCENE-8216:
----------------------------------------------

hi [~jim.ferenczi] I am investigating BM25F in Lucene and Solr and I ended up here: org/apache/lucene/sandbox/search/CombinedFieldQuery.java:289 
When calculating the IDF in BM25F we do that across fields, so as far as I explored the matter in my investigation yet the Document Frequency for a term T should be:

Number of documents in the corpus that contains the term T (in any field).

So effectively it would be the cardinality of the set that is the Union of all the posting lists for such term across the various fields.

From a quick look at your code, the document frequency is just calcolated as the max document frequency, across all the field involved (and that is actually the lower bound of the real blended document frequency).
Was it done with this approximation for simplicity, or there's any other reason?

> Better cross-field scoring
> --------------------------
>
>                 Key: LUCENE-8216
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8216
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Adrien Grand
>            Assignee: Jim Ferenczi
>            Priority: Major
>             Fix For: 8.0
>
>         Attachments: LUCENE-8216.patch, LUCENE-8216.patch
>
>
> I'd like Lucene to have better support for scoring across multiple fields. Today we have BlendedTermQuery which tries to help there but it probably tries to do too much on some aspects (handling cross-field term queries AND synonyms) and too little on other ones (it tries to merge index-level statistics, but not per-document statistics like tf and norm).
> Maybe we could implement something like BM25F so that queries across multiple fields would retain the benefits of BM25 like the fact that the impact of the term frequency saturates quickly, which is not the case with BlendedTermQuery if you have occurrences across many fields.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org