You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Stian Brattland <st...@octetnest.no> on 2022/09/17 17:30:56 UTC
How is fieldNorm calculated when omitNorms is set to true?
Hi,
I have a Solr (3.2.0) instance with omitNorms=true on all fields. This has
been done in an attempt to not penalize documents which have many terms.
What puzzles me is that, despite omitNorms=true, the fieldNorm is still
calculated and affects the score.
---
3.7455106 = (MATCH) fieldWeight(track_hierarchynode:pop in 258465),
product of:
2.236068 = tf(termFreq(track_hierarchynode:pop)=5)
6.700173 = idf(docFreq=6960, maxDocs=2080776)
0.25 = fieldNorm(field=track_hierarchynode, doc=258465)
---
How is the fieldNorm value calculated when omitNorms=true?
Kind regards,
Stian Brattland
Re: How is fieldNorm calculated when omitNorms is set to true?
Posted by Stian Brattland <st...@octetnest.no>.
Hi Thomas,
Yes, I will most likely go ahead and set up new instances for Solr 9 and
rebuild the index. I'd be surprised if the original schema from 3.2.0 is
100% compatible with Solr 9.
Thanks :-)
Regards,
Stian
tir. 20. sep. 2022 kl. 11:04 skrev Thomas Corthals <th...@klascement.net>:
> Hi Stian,
>
> You can't upgrade across more than one major version. You'll have to
> reindex against a new Solr install.
>
> The ref guide has more on configuring similarity:
>
> https://solr.apache.org/guide/solr/latest/indexing-guide/schema-elements.html#similarity
>
> Wikipedia has an explanation of the BM25 ranking function:
> https://en.wikipedia.org/wiki/Okapi_BM25
>
> Kind regards,
>
> Thomas
>
> Op ma 19 sep. 2022 om 08:59 schreef Stian Brattland <st...@octetnest.no>:
>
> > Hi Thomas,
> >
> > Yes, it may be about time to upgrade from 3.2.0. I think I will have to
> > look into that. Thanks for sharing your insight and configuration
> example.
> >
> > Kind regards,
> > Stian
> >
> >
> >
> > søn. 18. sep. 2022 kl. 16:21 skrev Thomas Corthals <
> thomas@klascement.net
> > >:
> >
> > > Hi Stian,
> > >
> > > We have the same issue with our documents. I fixed that by setting b =
> 0
> > in
> > > our schema for BM25 similarity.
> > >
> > > <similarity class="solr.BM25SimilarityFactory">
> > > <float name="b">0</float>
> > > </similarity>
> > >
> > > I don't know if BM25 can be used with your version of Solr. Personally
> I
> > > think it's worth upgrading for.
> > >
> > > Thomas
> > >
> > >
> > > Op za 17 sep. 2022 om 19:30 schreef Stian Brattland <
> stian@octetnest.no
> > >:
> > >
> > > > Hi,
> > > >
> > > > I have a Solr (3.2.0) instance with omitNorms=true on all fields.
> This
> > > has
> > > > been done in an attempt to not penalize documents which have many
> > terms.
> > > >
> > > > What puzzles me is that, despite omitNorms=true, the fieldNorm is
> still
> > > > calculated and affects the score.
> > > >
> > > > ---
> > > > 3.7455106 = (MATCH) fieldWeight(track_hierarchynode:pop in
> 258465),
> > > > product of:
> > > > 2.236068 = tf(termFreq(track_hierarchynode:pop)=5)
> > > > 6.700173 = idf(docFreq=6960, maxDocs=2080776)
> > > > 0.25 = fieldNorm(field=track_hierarchynode, doc=258465)
> > > > ---
> > > >
> > > > How is the fieldNorm value calculated when omitNorms=true?
> > > >
> > > > Kind regards,
> > > > Stian Brattland
> > > >
> > >
> >
>
Re: How is fieldNorm calculated when omitNorms is set to true?
Posted by Thomas Corthals <th...@klascement.net>.
Hi Stian,
You can't upgrade across more than one major version. You'll have to
reindex against a new Solr install.
The ref guide has more on configuring similarity:
https://solr.apache.org/guide/solr/latest/indexing-guide/schema-elements.html#similarity
Wikipedia has an explanation of the BM25 ranking function:
https://en.wikipedia.org/wiki/Okapi_BM25
Kind regards,
Thomas
Op ma 19 sep. 2022 om 08:59 schreef Stian Brattland <st...@octetnest.no>:
> Hi Thomas,
>
> Yes, it may be about time to upgrade from 3.2.0. I think I will have to
> look into that. Thanks for sharing your insight and configuration example.
>
> Kind regards,
> Stian
>
>
>
> søn. 18. sep. 2022 kl. 16:21 skrev Thomas Corthals <thomas@klascement.net
> >:
>
> > Hi Stian,
> >
> > We have the same issue with our documents. I fixed that by setting b = 0
> in
> > our schema for BM25 similarity.
> >
> > <similarity class="solr.BM25SimilarityFactory">
> > <float name="b">0</float>
> > </similarity>
> >
> > I don't know if BM25 can be used with your version of Solr. Personally I
> > think it's worth upgrading for.
> >
> > Thomas
> >
> >
> > Op za 17 sep. 2022 om 19:30 schreef Stian Brattland <stian@octetnest.no
> >:
> >
> > > Hi,
> > >
> > > I have a Solr (3.2.0) instance with omitNorms=true on all fields. This
> > has
> > > been done in an attempt to not penalize documents which have many
> terms.
> > >
> > > What puzzles me is that, despite omitNorms=true, the fieldNorm is still
> > > calculated and affects the score.
> > >
> > > ---
> > > 3.7455106 = (MATCH) fieldWeight(track_hierarchynode:pop in 258465),
> > > product of:
> > > 2.236068 = tf(termFreq(track_hierarchynode:pop)=5)
> > > 6.700173 = idf(docFreq=6960, maxDocs=2080776)
> > > 0.25 = fieldNorm(field=track_hierarchynode, doc=258465)
> > > ---
> > >
> > > How is the fieldNorm value calculated when omitNorms=true?
> > >
> > > Kind regards,
> > > Stian Brattland
> > >
> >
>
Re: How is fieldNorm calculated when omitNorms is set to true?
Posted by Thomas Corthals <th...@klascement.net>.
Hi Stian,
You can't upgrade across more than one major version. You'll have to
reindex against a new Solr install.
The ref guide has more on configuring similarity:
https://solr.apache.org/guide/solr/latest/indexing-guide/schema-elements.html#similarity
Wikipedia has an explanation of the BM25 ranking function:
https://en.wikipedia.org/wiki/Okapi_BM25
Kind regards,
Thomas
Op ma 19 sep. 2022 om 08:59 schreef Stian Brattland <st...@octetnest.no>:
> Hi Thomas,
>
> Yes, it may be about time to upgrade from 3.2.0. I think I will have to
> look into that. Thanks for sharing your insight and configuration example.
>
> Kind regards,
> Stian
>
>
>
> søn. 18. sep. 2022 kl. 16:21 skrev Thomas Corthals <thomas@klascement.net
> >:
>
> > Hi Stian,
> >
> > We have the same issue with our documents. I fixed that by setting b = 0
> in
> > our schema for BM25 similarity.
> >
> > <similarity class="solr.BM25SimilarityFactory">
> > <float name="b">0</float>
> > </similarity>
> >
> > I don't know if BM25 can be used with your version of Solr. Personally I
> > think it's worth upgrading for.
> >
> > Thomas
> >
> >
> > Op za 17 sep. 2022 om 19:30 schreef Stian Brattland <stian@octetnest.no
> >:
> >
> > > Hi,
> > >
> > > I have a Solr (3.2.0) instance with omitNorms=true on all fields. This
> > has
> > > been done in an attempt to not penalize documents which have many
> terms.
> > >
> > > What puzzles me is that, despite omitNorms=true, the fieldNorm is still
> > > calculated and affects the score.
> > >
> > > ---
> > > 3.7455106 = (MATCH) fieldWeight(track_hierarchynode:pop in 258465),
> > > product of:
> > > 2.236068 = tf(termFreq(track_hierarchynode:pop)=5)
> > > 6.700173 = idf(docFreq=6960, maxDocs=2080776)
> > > 0.25 = fieldNorm(field=track_hierarchynode, doc=258465)
> > > ---
> > >
> > > How is the fieldNorm value calculated when omitNorms=true?
> > >
> > > Kind regards,
> > > Stian Brattland
> > >
> >
>
Re: How is fieldNorm calculated when omitNorms is set to true?
Posted by Stian Brattland <st...@octetnest.no>.
Hi Thomas,
Yes, it may be about time to upgrade from 3.2.0. I think I will have to
look into that. Thanks for sharing your insight and configuration example.
Kind regards,
Stian
søn. 18. sep. 2022 kl. 16:21 skrev Thomas Corthals <th...@klascement.net>:
> Hi Stian,
>
> We have the same issue with our documents. I fixed that by setting b = 0 in
> our schema for BM25 similarity.
>
> <similarity class="solr.BM25SimilarityFactory">
> <float name="b">0</float>
> </similarity>
>
> I don't know if BM25 can be used with your version of Solr. Personally I
> think it's worth upgrading for.
>
> Thomas
>
>
> Op za 17 sep. 2022 om 19:30 schreef Stian Brattland <st...@octetnest.no>:
>
> > Hi,
> >
> > I have a Solr (3.2.0) instance with omitNorms=true on all fields. This
> has
> > been done in an attempt to not penalize documents which have many terms.
> >
> > What puzzles me is that, despite omitNorms=true, the fieldNorm is still
> > calculated and affects the score.
> >
> > ---
> > 3.7455106 = (MATCH) fieldWeight(track_hierarchynode:pop in 258465),
> > product of:
> > 2.236068 = tf(termFreq(track_hierarchynode:pop)=5)
> > 6.700173 = idf(docFreq=6960, maxDocs=2080776)
> > 0.25 = fieldNorm(field=track_hierarchynode, doc=258465)
> > ---
> >
> > How is the fieldNorm value calculated when omitNorms=true?
> >
> > Kind regards,
> > Stian Brattland
> >
>
Re: How is fieldNorm calculated when omitNorms is set to true?
Posted by Thomas Corthals <th...@klascement.net>.
Hi Stian,
We have the same issue with our documents. I fixed that by setting b = 0 in
our schema for BM25 similarity.
<similarity class="solr.BM25SimilarityFactory">
<float name="b">0</float>
</similarity>
I don't know if BM25 can be used with your version of Solr. Personally I
think it's worth upgrading for.
Thomas
Op za 17 sep. 2022 om 19:30 schreef Stian Brattland <st...@octetnest.no>:
> Hi,
>
> I have a Solr (3.2.0) instance with omitNorms=true on all fields. This has
> been done in an attempt to not penalize documents which have many terms.
>
> What puzzles me is that, despite omitNorms=true, the fieldNorm is still
> calculated and affects the score.
>
> ---
> 3.7455106 = (MATCH) fieldWeight(track_hierarchynode:pop in 258465),
> product of:
> 2.236068 = tf(termFreq(track_hierarchynode:pop)=5)
> 6.700173 = idf(docFreq=6960, maxDocs=2080776)
> 0.25 = fieldNorm(field=track_hierarchynode, doc=258465)
> ---
>
> How is the fieldNorm value calculated when omitNorms=true?
>
> Kind regards,
> Stian Brattland
>
Re: How is fieldNorm calculated when omitNorms is set to true?
Posted by Thomas Corthals <th...@klascement.net>.
Hi Stian,
We have the same issue with our documents. I fixed that by setting b = 0 in
our schema for BM25 similarity.
<similarity class="solr.BM25SimilarityFactory">
<float name="b">0</float>
</similarity>
I don't know if BM25 can be used with your version of Solr. Personally I
think it's worth upgrading for.
Thomas
Op za 17 sep. 2022 om 19:30 schreef Stian Brattland <st...@octetnest.no>:
> Hi,
>
> I have a Solr (3.2.0) instance with omitNorms=true on all fields. This has
> been done in an attempt to not penalize documents which have many terms.
>
> What puzzles me is that, despite omitNorms=true, the fieldNorm is still
> calculated and affects the score.
>
> ---
> 3.7455106 = (MATCH) fieldWeight(track_hierarchynode:pop in 258465),
> product of:
> 2.236068 = tf(termFreq(track_hierarchynode:pop)=5)
> 6.700173 = idf(docFreq=6960, maxDocs=2080776)
> 0.25 = fieldNorm(field=track_hierarchynode, doc=258465)
> ---
>
> How is the fieldNorm value calculated when omitNorms=true?
>
> Kind regards,
> Stian Brattland
>