[jira] [Commented] (LUCENE-10471) Increase the number of dims for KNN vectors to 2048

["Stanislav (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/LUCENE-10471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17565240#comment-17565240 ] 

Stanislav commented on LUCENE-10471:
------------------------------------

I don't think there is a trend to increase dimensionality. Only few models have feature dimensions more than 2048.

Most of modern neural networks (ViT and whole Bert family) have dimensions less than 1k. 

However there are still many models like ms-resnet or EfficientNet that operate in range from 1k to 2048. 

And they are most common models for image embedding and vector search.

Current limit is forcing to do dimensionally reduction for pretty standard shapes. 

 

> Increase the number of dims for KNN vectors to 2048
> ---------------------------------------------------
>
>                 Key: LUCENE-10471
>                 URL: https://issues.apache.org/jira/browse/LUCENE-10471
>             Project: Lucene - Core
>          Issue Type: Wish
>            Reporter: Mayya Sharipova
>            Priority: Trivial
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> The current maximum allowed number of dimensions is equal to 1024. But we see in practice a couple well-known models that produce vectors with > 1024 dimensions (e.g [mobilenet_v2|https://tfhub.dev/google/imagenet/mobilenet_v2_035_224/feature_vector/1] uses 1280d vectors, OpenAI / GPT-3 Babbage uses 2048d vectors). Increasing max dims to `2048` will satisfy these use cases.
> I am wondering if anybody has strong objections against this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org