normalizing vectors

[Michael Sokolov <ms...@gmail.com>]

I have been working on getting benchmarks working on the GloVe public
data set and spent a while chasing down a bug with VectorValues.search
that turned out to be a bug with the data (sort of)! When comparing
vectors using an angular (dot product) measure, one has to normalize
by the vectors' lengths. Given that the only purpose of such vectors
is to compare them using dot-product, it would be sensible to
normalize them *in advance* to unit length, rather than doing so for
every comparison, yet this is not how this dataset at least is
distributed on the internet, and widely-referenced benchmarking
software such as ann-benchmarks assumes that code will handle such
details internally.

I'm trying to see how we should handle this use case. We could provide
a convenience function for normalizing while indexing. But should we?
Would it happen when creating an IndexableField? When flushing? It's a
little strange if you index a vector, and then retrieve it and its
value is different! Alternatively we could simply expect users to
perform such normalization, and throw an error if vectors intended for
comparison using dot product (which is specified when adding a value)
are not unit-length. But then again this is a somewhat costly
operation that is only a safety measure, and users who already
normalized their vectors would pay the cost needlessly.

For now, I'm doing nothing, but I wonder if we could offer users some help here.

-Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: normalizing vectors

[Michael Sokolov <ms...@gmail.com>]

Thanks, Tomoko, I think you're right. Documentation and a helper function
should do it. I see now that ann-benchmarks toolkit can be made to do the
work external to the indexing process.

On Wed, Nov 25, 2020, 6:52 PM Tomoko Uchida <to...@gmail.com>
wrote:

> Hi Mike,
> I'm looking forward to that vector search is available at 9.0, thanks for
> your hard work on it.
>
> > Alternatively we could simply expect users to
> > perform such normalization, and throw an error if vectors intended for
> > comparison using dot product (which is specified when adding a value)
> > are not unit-length.
>
> For simplicity, we could assume normalized vectors as inputs and just
> document it - without any checks? Meanwhile some utility functions (e.g.,
> o.a.l.util.VectorUtil) for it could be helpful.
>
> Tomoko
>
>
> 2020年11月26日(木) 6:23 Michael Sokolov <ms...@gmail.com>:
>
>> I have been working on getting benchmarks working on the GloVe public
>> data set and spent a while chasing down a bug with VectorValues.search
>> that turned out to be a bug with the data (sort of)! When comparing
>> vectors using an angular (dot product) measure, one has to normalize
>> by the vectors' lengths. Given that the only purpose of such vectors
>> is to compare them using dot-product, it would be sensible to
>> normalize them *in advance* to unit length, rather than doing so for
>> every comparison, yet this is not how this dataset at least is
>> distributed on the internet, and widely-referenced benchmarking
>> software such as ann-benchmarks assumes that code will handle such
>> details internally.
>>
>> I'm trying to see how we should handle this use case. We could provide
>> a convenience function for normalizing while indexing. But should we?
>> Would it happen when creating an IndexableField? When flushing? It's a
>> little strange if you index a vector, and then retrieve it and its
>> value is different! Alternatively we could simply expect users to
>> perform such normalization, and throw an error if vectors intended for
>> comparison using dot product (which is specified when adding a value)
>> are not unit-length. But then again this is a somewhat costly
>> operation that is only a safety measure, and users who already
>> normalized their vectors would pay the cost needlessly.
>>
>> For now, I'm doing nothing, but I wonder if we could offer users some
>> help here.
>>
>> -Mike
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>>
>>

Re: normalizing vectors

[Tomoko Uchida <to...@gmail.com>]

Hi Mike,
I'm looking forward to that vector search is available at 9.0, thanks for
your hard work on it.

> Alternatively we could simply expect users to
> perform such normalization, and throw an error if vectors intended for
> comparison using dot product (which is specified when adding a value)
> are not unit-length.

For simplicity, we could assume normalized vectors as inputs and just
document it - without any checks? Meanwhile some utility functions (e.g.,
o.a.l.util.VectorUtil) for it could be helpful.

Tomoko


2020年11月26日(木) 6:23 Michael Sokolov <ms...@gmail.com>:

> I have been working on getting benchmarks working on the GloVe public
> data set and spent a while chasing down a bug with VectorValues.search
> that turned out to be a bug with the data (sort of)! When comparing
> vectors using an angular (dot product) measure, one has to normalize
> by the vectors' lengths. Given that the only purpose of such vectors
> is to compare them using dot-product, it would be sensible to
> normalize them *in advance* to unit length, rather than doing so for
> every comparison, yet this is not how this dataset at least is
> distributed on the internet, and widely-referenced benchmarking
> software such as ann-benchmarks assumes that code will handle such
> details internally.
>
> I'm trying to see how we should handle this use case. We could provide
> a convenience function for normalizing while indexing. But should we?
> Would it happen when creating an IndexableField? When flushing? It's a
> little strange if you index a vector, and then retrieve it and its
> value is different! Alternatively we could simply expect users to
> perform such normalization, and throw an error if vectors intended for
> comparison using dot product (which is specified when adding a value)
> are not unit-length. But then again this is a somewhat costly
> operation that is only a safety measure, and users who already
> normalized their vectors would pay the cost needlessly.
>
> For now, I'm doing nothing, but I wonder if we could offer users some help
> here.
>
> -Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>