You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "asfimport (via GitHub)" <gi...@apache.org> on 2026/01/16 16:30:02 UTC
[I] Bypass total frequency check if field uses custom term frequency [LUCENE-10048] [lucene]
asfimport opened a new issue, #11086:
URL: https://github.com/apache/lucene/issues/11086
For all fields whose index option is not **IndexOptions.NONE**. There is a check on per field total token count (i.e. field-length) to ensure we don't index too many tokens. This is done by accumulating the token's **TermFrequencyAttribute.**
Given that currently Lucene allows custom term frequency attached to each token and the usage of the frequency can be pretty wild. It is possible to have the following case where the check fails with only a few tokens that have large frequencies. Currently Lucene will skip indexing the whole document.
**"foo|<very large number> bar|<very large number>"**
What should be way to inform the indexing chain not to check the field length?
A related observation, when custom term frequency is in use, user is not likely to use the similarity for this field. Maybe we can offer a way to specify that, too?
---
Migrated from [LUCENE-10048](https://issues.apache.org/jira/browse/LUCENE-10048) by Tony Xu (@Tony-X), 1 vote, resolved Aug 13 2021
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Bypass total frequency check if field uses custom term frequency [LUCENE-10048] [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov closed issue #11086: Bypass total frequency check if field uses custom term frequency [LUCENE-10048]
URL: https://github.com/apache/lucene/issues/11086
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [I] Bypass total frequency check if field uses custom term frequency [LUCENE-10048] [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on issue #11086:
URL: https://github.com/apache/lucene/issues/11086#issuecomment-3760846479
I want to come back to this one again. I did spend some time implementing an encoding for custom term frequencies using a floating point term frequency score that is basically a 12-bit encoding that only encodes nonegative integer numbers, and whose exponent is always nonnegative, with the idea being that the encoded bits of the maximum value look like a reasonably-bounded integer. While we can get this to work, it does have some drawbacks. The main one is that the loss of precision becomes noticeable. Possibly not in a truly meaningful way, but it does produce different search results for some queries. We could increase the precision, but then the encoding gets larger and we run into overflow issues again. For example, if our maximum encoded value uses 16 bits, ie it is 65535 viewed as an int, then we can only have 64K of these terms in a field before we overflow.
The other challenge is that using a custom floating point encoding does introduce some slowdown because we have to decode the values, and we read a lot of these values per query per document and we can't rely on any intrinsics for our 12-bit encoding.
Instead we are using a much simpler solution which is to positively identify fields with custom term frequencies (rather than trying to infer from having norms disabled or something, as was tried in a previous patch), and then use this knowledge to avoid the overflow in DefaultIndexingChain by treating each term occurrence as frequency of 1, while we consider the encoded "term frequency" written to term dict/postings as a term "score" that has nothing to do with frequency of occurrence which is really the spirit of the original custom term freq feature.
What I want to understand is what the practical consequences would be if someone were to, say, compute the default Similarity over such a field (we don't plan to do that, but someone could, and it shouldn't break in a horrible way). Anyway I'll post a small patch that will probably explain what I'm talking about better than this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org