You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "Chris M. Hostetter (Jira)" <ji...@apache.org> on 2023/10/31 22:41:00 UTC
[jira] [Resolved] (SOLR-17045) DenseVectorField w/ vectorDimension > 1024 no longer works by default
[ https://issues.apache.org/jira/browse/SOLR-17045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris M. Hostetter resolved SOLR-17045.
---------------------------------------
Fix Version/s: main (10.0)
9.5
Resolution: Fixed
> DenseVectorField w/ vectorDimension > 1024 no longer works by default
> ---------------------------------------------------------------------
>
> Key: SOLR-17045
> URL: https://issues.apache.org/jira/browse/SOLR-17045
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Affects Versions: 9.3
> Reporter: Chris M. Hostetter
> Assignee: Chris M. Hostetter
> Priority: Major
> Fix For: main (10.0), 9.5
>
>
> Starting with Solr 9.4, configuring a {{DenseVectorField}} w/ {{vectorDimension}} > 1024 no longer works by default. There is no error on startup, but when indexing you'll get errors like...
> {noformat}
> 2> => org.apache.solr.common.SolrException: Exception writing document id 2 to the index; possible analysis error: Field [vector]vector's dimensions must be <= [1024]; got 1600
> 2> at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:329)
> 2> org.apache.solr.common.SolrException: Exception writing document id 2 to the index; possible analysis error: Field [vector]vector's dimensions must be <= [1024]; got 1600
> 2> at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:329) ~[solr-core-9.4.0.jar:9.4.0 71e101bb37497f730078d9afe1991b60d10bfe96 - stillalex - 2023-10-10 19:10:39]
> 2> at org.apache.solr.update.processor.RunUpdateProcessorFactory$RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:76) ~[solr-core-9.4.0.jar:9.4.0 71e101bb37497f730078d9afe1991b60d10bfe96 - stillalex - 2023-10-10 19:10:39]
> ...
> 2> Caused by: java.lang.IllegalArgumentException: Field [vector]vector's dimensions must be <= [1024]; got 1600
> 2> at org.apache.lucene.index.IndexingChain.validateMaxVectorDimension(IndexingChain.java:843) ~[lucene-core-9.8.0.jar:9.8.0 d914b3722bd5b8ef31ccf7e8ddc638a87fd648db - 2023-09-21 21:57:47]
> ...
> {noformat}
> This is because Lucene 9.8 moved the dimension size limitation to the codec – and while Solr 9.4's {{SchemaCodecFactory}} was updated to implement a per-field {{SolrDelegatingKnnVectorsFormat}} that respected the {{vectorDimension}} configured for each {{DenseVectorField}} *the {{SchemaCodecFactory}} _is not implicitly used by default in Solr_* – nor is it explicitly configured in the {{_default}} configset.
> {panel:title=Known Work Around}
> Existing {{DenseVectorField}} who encounter this error when upgrading to Solr >= 9.4, or new user attempting to use {{DenseVectorField}} with {{vectorDimension}} > 1024, need to explicitly configure {{SchemaCodecFactory}} in the solrconfig.xml for each collection.
> {panel}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org