You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by 承諾一輩子 <50...@qq.com.INVALID> on 2023/08/09 09:38:11 UTC
What is the approximate processing mechanism for field length？
Dear colleague:
I am a drive development engineer works in ZTE corporation from China.
Recently in learning lucene source code.There is a question that puzzled me for a long time, as follows:
How to understand the approximate handing mechanism for field length "dl" in the BM25Scorer class? For example, "keywords" field has 78 tokens. I think its field_length(dl) is 78, but lucene handled as 76(approximate) as described in function explainTF(Explaination freq, long norm).
Thank you very much for your reading and look forward to your answer!
Koo
Drive development engineer
Re: What is the approximate processing mechanism for field length？
Posted by Mikhail Khludnev <mk...@apache.org>.
Hello, Koo.
I can share my shallow understanding of this algorithm.
Approximate field length is called norm and stored in a byte per document
for sake of compactness (implying performance). Norms are encoded
via org.apache.lucene.util.SmallFloat#intToByte4.
Thus, there might be just 256 different values of approximate field
length, and they are decoded by SmallFloat.byte4ToInt((byte) i) and cached
as floats in BM25Similarity#LENGTH_TABLE.
Float lengths might seem unexpected, but if you trace LENGTH_TABLE usage
you'll see that these values are used to compute ratios (floats),
therefore it's fine to use floats as inputs for score calculations.
As far as I understand, due to approximation nature:
field_length(dl)<= field_length(dl+1)
WDYT?
On Wed, Aug 9, 2023 at 5:43 PM 承諾一輩子 <50...@qq.com.invalid> wrote:
> Dear colleague:
> I am a drive development engineer works in ZTE corporation
> from China.
> Recently in learning lucene source code.There is a question
> that puzzled me for a long time, as follows:
> How to understand the approximate handing mechanism for field
> length "dl" in the BM25Scorer class? For example, "keywords" field has 78
> tokens. I think its field_length(dl) is 78, but lucene handled as
> 76(approximate) as described in function explainTF(Explaination freq, long
> norm).
> Thank you very much for your reading and look forward to your
> answer!
>
>
> Koo
> Drive development engineer
--
Sincerely yours
Mikhail Khludnev
Re: What is the approximate processing mechanism for field length？
Posted by Jean-Marie Lallouet <la...@gmail.com>.
Did the analyzer for this field remove some tokens?
Sent from my iPhone
> On 9. Aug 2023, at 16:43, 承諾一輩子 <50...@qq.com.invalid> wrote:
>
> Dear colleague:
> I am a drive development engineer works in ZTE corporation from China.
> Recently in learning lucene source code.There is a question that puzzled me for a long time, as follows:
> How to understand the approximate handing mechanism for field length "dl" in the BM25Scorer class? For example, "keywords" field has 78 tokens. I think its field_length(dl) is 78, but lucene handled as 76(approximate) as described in function explainTF(Explaination freq, long norm).
> Thank you very much for your reading and look forward to your answer!
>
>
> Koo
> Drive development engineer
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org