[jira] [Created] (LUCENE-8501) An ability to define the sum method for custom term frequencies

["Olli Kuonanoja (JIRA)" <ji...@apache.org>]

Olli Kuonanoja created LUCENE-8501:
--------------------------------------

             Summary: An ability to define the sum method for custom term frequencies
                 Key: LUCENE-8501
                 URL: https://issues.apache.org/jira/browse/LUCENE-8501
             Project: Lucene - Core
          Issue Type: Improvement
          Components: core/index
            Reporter: Olli Kuonanoja


Custom term frequencies allows expert users to index and score in custom ways, however, _DefaultIndexingChain_ adds a limitation to this as the sum of frequencies can't overflow
{code:java}
try {
    invertState.length = Math.addExact(invertState.length, invertState.termFreqAttribute.getTermFrequency());
} catch (ArithmeticException ae) {
    throw new IllegalArgumentException("too many tokens for field \"" + field.name() + "\"");
}
{code}
This might become an issue if for example the frequency data is encoded in a different way, say the specific scorer works with float frequencies.

The sum method can be added to _TermFrequencyAttribute_ to get something like
{code:java}
invertState.length = invertState.termFreqAttribute.addFrequency(invertState.length);
{code}
so users may define the summing method and avoid the owerflow exceptions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org