[jira] [Commented] (LUCENE-8501) An ability to define the sum method for custom term frequencies

["Adrien Grand (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-8501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617272#comment-16617272 ] 

Adrien Grand commented on LUCENE-8501:
--------------------------------------

Storing floats as term frequencies not only raises issues with field length, but also with storage efficiency since that will likely add 4 bytes for every (term,document) pair of your dataset. For instance Lucene's FeatureField only retains 16 bits of the 32 bits of floats: no sign bit, 8 exponent bits like regular floats, and 8 mantissa bits instead of 23. It makes the overhead of floats as term frequencies drop to 2 bytes per (term,document) pair instead of 4: https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/lucene/core/src/java/org/apache/lucene/document/FeatureField.java#L153-L154 Would a similar approach work for you too?

> An ability to define the sum method for custom term frequencies
> ---------------------------------------------------------------
>
>                 Key: LUCENE-8501
>                 URL: https://issues.apache.org/jira/browse/LUCENE-8501
>             Project: Lucene - Core
>          Issue Type: Improvement
>          Components: core/index
>            Reporter: Olli Kuonanoja
>            Priority: Major
>
> Custom term frequencies allows expert users to index and score in custom ways, however, _DefaultIndexingChain_ adds a limitation to this as the sum of frequencies can't overflow
> {code:java}
> try {
>     invertState.length = Math.addExact(invertState.length, invertState.termFreqAttribute.getTermFrequency());
> } catch (ArithmeticException ae) {
>     throw new IllegalArgumentException("too many tokens for field \"" + field.name() + "\"");
> }
> {code}
> This might become an issue if for example the frequency data is encoded in a different way, say the specific scorer works with float frequencies.
> The sum method can be added to _TermFrequencyAttribute_ to get something like
> {code:java}
> invertState.length = invertState.termFreqAttribute.addFrequency(invertState.length);
> {code}
> so users may define the summing method and avoid the owerflow exceptions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org