You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@lucene.apache.org by Jonathan Ellis <jb...@gmail.com> on 2023/06/09 21:38:03 UTC

Fix for vector math precision

Hi all,

I ran into a bug where the cosine of a large vector taken with itself
returned NaN.  (Cosine of equal vectors should always be 1.)  I put
together a PR to do the internal math of the cosine function with double,
before returning the result as a float:
https://github.com/apache/lucene/pull/12281

-- 
Jonathan Ellis
co-founder, http://www.datastax.com
@spyced

Re: Fix for vector math precision

Posted by Jonathan Ellis <jb...@gmail.com>.

Instead of doing double math, new PR to limit vector components to smaller
than 1E17 to prevent overflow: https://github.com/apache/lucene/pull/12373

On Fri, Jun 9, 2023 at 4:38 PM Jonathan Ellis <jb...@gmail.com> wrote:

> Hi all,
>
> I ran into a bug where the cosine of a large vector taken with itself
> returned NaN.  (Cosine of equal vectors should always be 1.)  I put
> together a PR to do the internal math of the cosine function with double,
> before returning the result as a float:
> https://github.com/apache/lucene/pull/12281
>
> --
> Jonathan Ellis
> co-founder, http://www.datastax.com
> @spyced
>


-- 
Jonathan Ellis
co-founder, http://www.datastax.com
@spyced