You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Michael Froh <ms...@gmail.com> on 2025/05/16 19:29:44 UTC
Increasing precision for sparse vector search
Hi folks,
I was recently (yesterday) pulled into a conversation with a data science
colleague about sparse vector search, which led me to learn how
FeatureFields work.
While it sounds like they should work to get things started, the science
guy was concerned about precision on their weights. Apparently their
experiments have been using 32-bit floats (with a 23 bit mantissa), whereas
FeatureField is stuck with an 8-bit mantissa. In theory, we could modify
FeatureField to use IEEE-754 half-precision floats (versus the current
truncated single-precision floats) giving us 10 bits of mantissa.
I found a thread (
https://lists.apache.org/thread/dcwm71rnjz7mz69ntwy04co3615zdl6w) from
Ankur Goel from 2020 on a similar subject. Not really sure where he ended
up going with it, though. (Though it sounds like his use-case might have
been different.)
I'm guessing the 16-bit limitation on FeatureField values is because the
custom term frequencies get added to the invertState.length at
https://github.com/apache/lucene/blob/b0ebaaed56505048d2c0bdfb7d71a3a082d23822/lucene/core/src/java/org/apache/lucene/index/IndexingChain.java#L1284,
and we're trying to avoid overflow?
Are there any other good options for sparse vector storage in Lucene? I'd
love something kind of like FeatureField, but where the values are legit
"payload" values stored in postings without a couple of extra hops of
indirection, but also without pretending that they're term frequencies.
Does anyone else have a use for something like that?
Thanks!
Froh
Re: Increasing precision for sparse vector search
Posted by Michael Froh <ms...@gmail.com>.
Forgot to close the loop on this one and was traveling last week.
I've gone back to the science folks with the suggestion that we experiment
with 16 bits. I suspect that you're right and 16 bits will be good enough.
It is good to know that we can play with it a bit, provided that we keep
under the field length overflow. For example, if we know that there are
fewer than 127 non-zero values for a given doc, I think we could expand to
24 bits. (I'm probably off-by-one in the count of values or bits.)
Hopefully it won't be necessary, though.
Thanks Adrien!
On Fri, May 16, 2025 at 1:51 PM Adrien Grand <jp...@gmail.com> wrote:
> Your understanding is correct. For reference, using only 16 bits was
> convenient to avoid overflowing the field length, but storage efficiency
> was a motivation as well. Using 32 bits per frequency instead of 16 would
> significantly increase the size of the inverted index.
>
> We could look into what it would take to support greater precision, on the
> other hand it would be interesting to check if more precision is actually
> needed. Historically, some people have raised concerns about e.g. how
> Lucene encodes length normalization factors on a single byte, but no
> benchmark that I'm aware of ever concluded that it hurts in practice. My
> understanding is also that weights are commonly stored as bfloat16, which
> have less precision (one less bit of mantissa) than FeatureField.
>
>
> On Fri, May 16, 2025 at 9:30 PM Michael Froh <ms...@gmail.com> wrote:
>
>> Hi folks,
>>
>> I was recently (yesterday) pulled into a conversation with a data science
>> colleague about sparse vector search, which led me to learn how
>> FeatureFields work.
>>
>> While it sounds like they should work to get things started, the science
>> guy was concerned about precision on their weights. Apparently their
>> experiments have been using 32-bit floats (with a 23 bit mantissa), whereas
>> FeatureField is stuck with an 8-bit mantissa. In theory, we could modify
>> FeatureField to use IEEE-754 half-precision floats (versus the current
>> truncated single-precision floats) giving us 10 bits of mantissa.
>>
>> I found a thread (
>> https://lists.apache.org/thread/dcwm71rnjz7mz69ntwy04co3615zdl6w) from
>> Ankur Goel from 2020 on a similar subject. Not really sure where he ended
>> up going with it, though. (Though it sounds like his use-case might have
>> been different.)
>>
>> I'm guessing the 16-bit limitation on FeatureField values is because the
>> custom term frequencies get added to the invertState.length at
>> https://github.com/apache/lucene/blob/b0ebaaed56505048d2c0bdfb7d71a3a082d23822/lucene/core/src/java/org/apache/lucene/index/IndexingChain.java#L1284,
>> and we're trying to avoid overflow?
>>
>> Are there any other good options for sparse vector storage in Lucene? I'd
>> love something kind of like FeatureField, but where the values are legit
>> "payload" values stored in postings without a couple of extra hops of
>> indirection, but also without pretending that they're term frequencies.
>> Does anyone else have a use for something like that?
>>
>> Thanks!
>> Froh
>>
>
>
> --
> Adrien
>
Re: Increasing precision for sparse vector search
Posted by Adrien Grand <jp...@gmail.com>.
Your understanding is correct. For reference, using only 16 bits was
convenient to avoid overflowing the field length, but storage efficiency
was a motivation as well. Using 32 bits per frequency instead of 16 would
significantly increase the size of the inverted index.
We could look into what it would take to support greater precision, on the
other hand it would be interesting to check if more precision is actually
needed. Historically, some people have raised concerns about e.g. how
Lucene encodes length normalization factors on a single byte, but no
benchmark that I'm aware of ever concluded that it hurts in practice. My
understanding is also that weights are commonly stored as bfloat16, which
have less precision (one less bit of mantissa) than FeatureField.
On Fri, May 16, 2025 at 9:30 PM Michael Froh <ms...@gmail.com> wrote:
> Hi folks,
>
> I was recently (yesterday) pulled into a conversation with a data science
> colleague about sparse vector search, which led me to learn how
> FeatureFields work.
>
> While it sounds like they should work to get things started, the science
> guy was concerned about precision on their weights. Apparently their
> experiments have been using 32-bit floats (with a 23 bit mantissa), whereas
> FeatureField is stuck with an 8-bit mantissa. In theory, we could modify
> FeatureField to use IEEE-754 half-precision floats (versus the current
> truncated single-precision floats) giving us 10 bits of mantissa.
>
> I found a thread (
> https://lists.apache.org/thread/dcwm71rnjz7mz69ntwy04co3615zdl6w) from
> Ankur Goel from 2020 on a similar subject. Not really sure where he ended
> up going with it, though. (Though it sounds like his use-case might have
> been different.)
>
> I'm guessing the 16-bit limitation on FeatureField values is because the
> custom term frequencies get added to the invertState.length at
> https://github.com/apache/lucene/blob/b0ebaaed56505048d2c0bdfb7d71a3a082d23822/lucene/core/src/java/org/apache/lucene/index/IndexingChain.java#L1284,
> and we're trying to avoid overflow?
>
> Are there any other good options for sparse vector storage in Lucene? I'd
> love something kind of like FeatureField, but where the values are legit
> "payload" values stored in postings without a couple of extra hops of
> indirection, but also without pretending that they're term frequencies.
> Does anyone else have a use for something like that?
>
> Thanks!
> Froh
>
--
Adrien