[jira] [Commented] (LUCENE-10471) Increase the number of dims for KNN vectors to 2048

["Robert Muir (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-10471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17508140#comment-17508140 ] 

Robert Muir commented on LUCENE-10471:
--------------------------------------

I don't "strongly object" but I question the approach of just raising the limit to satisfy whatever shitty models people come up with. At some point we should have a limit, and people should do dimensionality reduction.

> Increase the number of dims for KNN vectors to 2048
> ---------------------------------------------------
>
>                 Key: LUCENE-10471
>                 URL: https://issues.apache.org/jira/browse/LUCENE-10471
>             Project: Lucene - Core
>          Issue Type: Wish
>            Reporter: Mayya Sharipova
>            Priority: Trivial
>
> The current maximum allowed number of dimensions is equal to 1024. But we see in practice a couple well-known models that produce vectors with > 1024 dimensions (e.g [mobilenet_v2|https://tfhub.dev/google/imagenet/mobilenet_v2_035_224/feature_vector/1] uses 1280d vectors, OpenAI / GPT-3 Babbage uses 2048d vectors). Increasing max dims to `2048` will satisfy these use cases.
> I am wondering if anybody has strong objections against this.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org