You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@solr.apache.org by Alastair Porter <al...@porter.net.nz> on 2022/12/13 19:22:32 UTC
Dense Vector Search fieldType default parameters result in index error
Hi there,
I'm using solr 9.1 to implement dense vector search using the documentation
at
https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html
According to the guide, a valid fieldType configuration is:
<fieldType name="knn_vector" class="solr.DenseVectorField"
vectorDimension="4" similarityFunction="cosine"/>
Using the schema API, I added the following configuration:
$ grep knn_vector managed-schema.xml
<fieldType name="knn_vector" class="solr.DenseVectorField"
vectorDimension="4" similarityFunction="euclidean"/>
<field name="vector" type="knn_vector" multiValued="false" stored="true"/>
However, when I try to submit a document containing a vector to the server,
an error occurs:
(truncated traceback):
2022-12-13 19:10:08.763 ERROR (qtp3540494-54) [ x:fsnn]
o.a.s.s.HttpSolrCall org.apache.solr.common.SolrException: null KNN
algorithm is not supported => org.apache.solr.common.SolrException: null
KNN algorithm is not supported
at
org.apache.solr.core.SchemaCodecFactory$1.getKnnVectorsFormatForField(SchemaCodecFactory.java:134)
org.apache.solr.common.SolrException: null KNN algorithm is not supported
at
org.apache.solr.core.SchemaCodecFactory$1.getKnnVectorsFormatForField(SchemaCodecFactory.java:134)
~[?:?]
at
org.apache.lucene.codecs.lucene92.Lucene92Codec$3.getKnnVectorsFormatForField(Lucene92Codec.java:101)
~[?:?]
at
org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.getInstance(PerFieldKnnVectorsFormat.java:148)
~[?:?]
at
org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.writeField(PerFieldKnnVectorsFormat.java:107)
~[?:?]
at
org.apache.lucene.index.VectorValuesWriter.flush(VectorValuesWriter.java:145)
~[?:?]
at
org.apache.lucene.index.IndexingChain.writeVectors(IndexingChain.java:461)
~[?:?]
After having a dig through the source, I found
https://github.com/apache/solr/pull/910 which added this check. I note that
the documentation about dense vector search does describe the knnAlgorithm
parameter, but says that it is optional with a default value of "hnsw".
It seems that the patch changed the behaviour of this check to raise an
error if the knnAlgorithm is null/missing, rather than only if the value
isn't the expected "hnsw".
If I update the fieldType in the schema to include the knnAlgorithm,
hnswMaxConnections, and hnswBeamWidth parameters then I am able to
successfully index and search on this field.
Does this appear to be a bug in the code? Or if the parameter is expected
to always be set, perhaps a bug in the documentation.
Thanks,
Alastair
Re: Dense Vector Search fieldType default parameters result in index error
Posted by Alessandro Benedetti <a....@sease.io>.
Thanks, Alastair!
We'll keep you updated!
--------------------------
*Alessandro Benedetti*
Director @ Sease Ltd.
*Apache Lucene/Solr Committer*
*Apache Solr PMC Member*
e-mail: a.benedetti@sease.io
*Sease* - Information Retrieval Applied
Consulting | Training | Open Source
Website: Sease.io <http://sease.io/>
LinkedIn <https://linkedin.com/company/sease-ltd> | Twitter
<https://twitter.com/seaseltd> | Youtube
<https://www.youtube.com/channel/UCDx86ZKLYNpI3gzMercM7BQ> | Github
<https://github.com/seaseltd>
On Wed, 14 Dec 2022 at 18:41, Alastair Porter <al...@porter.net.nz>
wrote:
> I opened https://issues.apache.org/jira/browse/SOLR-16588
>
> Alastair
>
> On Wed, 14 Dec 2022 at 08:52, Alastair Porter <al...@porter.net.nz>
> wrote:
>
> > Hi Alessandro,
> > Thanks for the confirmation of the bug.
> > I'd be happy to create an issue for this, but I don't have a jira
> account,
> > and I see that it's closed for signups due to spam. I wrote here because
> I
> > was unable to find a private contact for solr (as suggested in
> > https://infra.apache.org/jira-guidelines.html#who)
> > If you or someone else from the project can contact me on this email
> > address I'd be happy to send you the necessary details to create the
> > account so that I can raise the issue.
> >
> > Alastair
> >
> > On Wed, 14 Dec 2022 at 02:50, Alessandro Benedetti <a.benedetti@sease.io
> >
> > wrote:
> >
> >> Hi Alastair,
> >> That's definitely not intended and we'll fix it as soon as possible (it
> >> will be available in the next release).
> >>
> >> Feel free to create a Jira issue on the Apache Solr project with this
> >> details (and I'll be able to link it in the Pull Request)
> >>
> >> Cheers
> >>
> >> On Wed, 14 Dec 2022, 04:22 Alastair Porter, <al...@porter.net.nz>
> >> wrote:
> >>
> >> > Hi there,
> >> > I'm using solr 9.1 to implement dense vector search using the
> >> documentation
> >> > at
> >> >
> >> >
> >>
> https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html
> >> >
> >> > According to the guide, a valid fieldType configuration is:
> >> >
> >> > <fieldType name="knn_vector" class="solr.DenseVectorField"
> >> > vectorDimension="4" similarityFunction="cosine"/>
> >> >
> >> > Using the schema API, I added the following configuration:
> >> > $ grep knn_vector managed-schema.xml
> >> > <fieldType name="knn_vector" class="solr.DenseVectorField"
> >> > vectorDimension="4" similarityFunction="euclidean"/>
> >> > <field name="vector" type="knn_vector" multiValued="false"
> >> > stored="true"/>
> >> >
> >> > However, when I try to submit a document containing a vector to the
> >> server,
> >> > an error occurs:
> >> > (truncated traceback):
> >> > 2022-12-13 19:10:08.763 ERROR (qtp3540494-54) [ x:fsnn]
> >> > o.a.s.s.HttpSolrCall org.apache.solr.common.SolrException: null KNN
> >> > algorithm is not supported => org.apache.solr.common.SolrException:
> null
> >> > KNN algorithm is not supported
> >> > at
> >> >
> >> >
> >>
> org.apache.solr.core.SchemaCodecFactory$1.getKnnVectorsFormatForField(SchemaCodecFactory.java:134)
> >> > org.apache.solr.common.SolrException: null KNN algorithm is not
> >> supported
> >> > at
> >> >
> >> >
> >>
> org.apache.solr.core.SchemaCodecFactory$1.getKnnVectorsFormatForField(SchemaCodecFactory.java:134)
> >> > ~[?:?]
> >> > at
> >> >
> >> >
> >>
> org.apache.lucene.codecs.lucene92.Lucene92Codec$3.getKnnVectorsFormatForField(Lucene92Codec.java:101)
> >> > ~[?:?]
> >> > at
> >> >
> >> >
> >>
> org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.getInstance(PerFieldKnnVectorsFormat.java:148)
> >> > ~[?:?]
> >> > at
> >> >
> >> >
> >>
> org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.writeField(PerFieldKnnVectorsFormat.java:107)
> >> > ~[?:?]
> >> > at
> >> >
> >> >
> >>
> org.apache.lucene.index.VectorValuesWriter.flush(VectorValuesWriter.java:145)
> >> > ~[?:?]
> >> > at
> >> >
> >>
> org.apache.lucene.index.IndexingChain.writeVectors(IndexingChain.java:461)
> >> > ~[?:?]
> >> >
> >> > After having a dig through the source, I found
> >> > https://github.com/apache/solr/pull/910 which added this check. I
> note
> >> > that
> >> > the documentation about dense vector search does describe the
> >> knnAlgorithm
> >> > parameter, but says that it is optional with a default value of
> "hnsw".
> >> > It seems that the patch changed the behaviour of this check to raise
> an
> >> > error if the knnAlgorithm is null/missing, rather than only if the
> value
> >> > isn't the expected "hnsw".
> >> >
> >> > If I update the fieldType in the schema to include the knnAlgorithm,
> >> > hnswMaxConnections, and hnswBeamWidth parameters then I am able to
> >> > successfully index and search on this field.
> >> >
> >> > Does this appear to be a bug in the code? Or if the parameter is
> >> expected
> >> > to always be set, perhaps a bug in the documentation.
> >> >
> >> > Thanks,
> >> > Alastair
> >> >
> >>
> >
>
Re: Dense Vector Search fieldType default parameters result in index error
Posted by Alastair Porter <al...@porter.net.nz>.
I opened https://issues.apache.org/jira/browse/SOLR-16588
Alastair
On Wed, 14 Dec 2022 at 08:52, Alastair Porter <al...@porter.net.nz>
wrote:
> Hi Alessandro,
> Thanks for the confirmation of the bug.
> I'd be happy to create an issue for this, but I don't have a jira account,
> and I see that it's closed for signups due to spam. I wrote here because I
> was unable to find a private contact for solr (as suggested in
> https://infra.apache.org/jira-guidelines.html#who)
> If you or someone else from the project can contact me on this email
> address I'd be happy to send you the necessary details to create the
> account so that I can raise the issue.
>
> Alastair
>
> On Wed, 14 Dec 2022 at 02:50, Alessandro Benedetti <a....@sease.io>
> wrote:
>
>> Hi Alastair,
>> That's definitely not intended and we'll fix it as soon as possible (it
>> will be available in the next release).
>>
>> Feel free to create a Jira issue on the Apache Solr project with this
>> details (and I'll be able to link it in the Pull Request)
>>
>> Cheers
>>
>> On Wed, 14 Dec 2022, 04:22 Alastair Porter, <al...@porter.net.nz>
>> wrote:
>>
>> > Hi there,
>> > I'm using solr 9.1 to implement dense vector search using the
>> documentation
>> > at
>> >
>> >
>> https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html
>> >
>> > According to the guide, a valid fieldType configuration is:
>> >
>> > <fieldType name="knn_vector" class="solr.DenseVectorField"
>> > vectorDimension="4" similarityFunction="cosine"/>
>> >
>> > Using the schema API, I added the following configuration:
>> > $ grep knn_vector managed-schema.xml
>> > <fieldType name="knn_vector" class="solr.DenseVectorField"
>> > vectorDimension="4" similarityFunction="euclidean"/>
>> > <field name="vector" type="knn_vector" multiValued="false"
>> > stored="true"/>
>> >
>> > However, when I try to submit a document containing a vector to the
>> server,
>> > an error occurs:
>> > (truncated traceback):
>> > 2022-12-13 19:10:08.763 ERROR (qtp3540494-54) [ x:fsnn]
>> > o.a.s.s.HttpSolrCall org.apache.solr.common.SolrException: null KNN
>> > algorithm is not supported => org.apache.solr.common.SolrException: null
>> > KNN algorithm is not supported
>> > at
>> >
>> >
>> org.apache.solr.core.SchemaCodecFactory$1.getKnnVectorsFormatForField(SchemaCodecFactory.java:134)
>> > org.apache.solr.common.SolrException: null KNN algorithm is not
>> supported
>> > at
>> >
>> >
>> org.apache.solr.core.SchemaCodecFactory$1.getKnnVectorsFormatForField(SchemaCodecFactory.java:134)
>> > ~[?:?]
>> > at
>> >
>> >
>> org.apache.lucene.codecs.lucene92.Lucene92Codec$3.getKnnVectorsFormatForField(Lucene92Codec.java:101)
>> > ~[?:?]
>> > at
>> >
>> >
>> org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.getInstance(PerFieldKnnVectorsFormat.java:148)
>> > ~[?:?]
>> > at
>> >
>> >
>> org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.writeField(PerFieldKnnVectorsFormat.java:107)
>> > ~[?:?]
>> > at
>> >
>> >
>> org.apache.lucene.index.VectorValuesWriter.flush(VectorValuesWriter.java:145)
>> > ~[?:?]
>> > at
>> >
>> org.apache.lucene.index.IndexingChain.writeVectors(IndexingChain.java:461)
>> > ~[?:?]
>> >
>> > After having a dig through the source, I found
>> > https://github.com/apache/solr/pull/910 which added this check. I note
>> > that
>> > the documentation about dense vector search does describe the
>> knnAlgorithm
>> > parameter, but says that it is optional with a default value of "hnsw".
>> > It seems that the patch changed the behaviour of this check to raise an
>> > error if the knnAlgorithm is null/missing, rather than only if the value
>> > isn't the expected "hnsw".
>> >
>> > If I update the fieldType in the schema to include the knnAlgorithm,
>> > hnswMaxConnections, and hnswBeamWidth parameters then I am able to
>> > successfully index and search on this field.
>> >
>> > Does this appear to be a bug in the code? Or if the parameter is
>> expected
>> > to always be set, perhaps a bug in the documentation.
>> >
>> > Thanks,
>> > Alastair
>> >
>>
>
Re: Dense Vector Search fieldType default parameters result in index error
Posted by Alastair Porter <al...@porter.net.nz>.
Hi Alessandro,
Thanks for the confirmation of the bug.
I'd be happy to create an issue for this, but I don't have a jira account,
and I see that it's closed for signups due to spam. I wrote here because I
was unable to find a private contact for solr (as suggested in
https://infra.apache.org/jira-guidelines.html#who)
If you or someone else from the project can contact me on this email
address I'd be happy to send you the necessary details to create the
account so that I can raise the issue.
Alastair
On Wed, 14 Dec 2022 at 02:50, Alessandro Benedetti <a....@sease.io>
wrote:
> Hi Alastair,
> That's definitely not intended and we'll fix it as soon as possible (it
> will be available in the next release).
>
> Feel free to create a Jira issue on the Apache Solr project with this
> details (and I'll be able to link it in the Pull Request)
>
> Cheers
>
> On Wed, 14 Dec 2022, 04:22 Alastair Porter, <al...@porter.net.nz>
> wrote:
>
> > Hi there,
> > I'm using solr 9.1 to implement dense vector search using the
> documentation
> > at
> >
> >
> https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html
> >
> > According to the guide, a valid fieldType configuration is:
> >
> > <fieldType name="knn_vector" class="solr.DenseVectorField"
> > vectorDimension="4" similarityFunction="cosine"/>
> >
> > Using the schema API, I added the following configuration:
> > $ grep knn_vector managed-schema.xml
> > <fieldType name="knn_vector" class="solr.DenseVectorField"
> > vectorDimension="4" similarityFunction="euclidean"/>
> > <field name="vector" type="knn_vector" multiValued="false"
> > stored="true"/>
> >
> > However, when I try to submit a document containing a vector to the
> server,
> > an error occurs:
> > (truncated traceback):
> > 2022-12-13 19:10:08.763 ERROR (qtp3540494-54) [ x:fsnn]
> > o.a.s.s.HttpSolrCall org.apache.solr.common.SolrException: null KNN
> > algorithm is not supported => org.apache.solr.common.SolrException: null
> > KNN algorithm is not supported
> > at
> >
> >
> org.apache.solr.core.SchemaCodecFactory$1.getKnnVectorsFormatForField(SchemaCodecFactory.java:134)
> > org.apache.solr.common.SolrException: null KNN algorithm is not supported
> > at
> >
> >
> org.apache.solr.core.SchemaCodecFactory$1.getKnnVectorsFormatForField(SchemaCodecFactory.java:134)
> > ~[?:?]
> > at
> >
> >
> org.apache.lucene.codecs.lucene92.Lucene92Codec$3.getKnnVectorsFormatForField(Lucene92Codec.java:101)
> > ~[?:?]
> > at
> >
> >
> org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.getInstance(PerFieldKnnVectorsFormat.java:148)
> > ~[?:?]
> > at
> >
> >
> org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.writeField(PerFieldKnnVectorsFormat.java:107)
> > ~[?:?]
> > at
> >
> >
> org.apache.lucene.index.VectorValuesWriter.flush(VectorValuesWriter.java:145)
> > ~[?:?]
> > at
> >
> org.apache.lucene.index.IndexingChain.writeVectors(IndexingChain.java:461)
> > ~[?:?]
> >
> > After having a dig through the source, I found
> > https://github.com/apache/solr/pull/910 which added this check. I note
> > that
> > the documentation about dense vector search does describe the
> knnAlgorithm
> > parameter, but says that it is optional with a default value of "hnsw".
> > It seems that the patch changed the behaviour of this check to raise an
> > error if the knnAlgorithm is null/missing, rather than only if the value
> > isn't the expected "hnsw".
> >
> > If I update the fieldType in the schema to include the knnAlgorithm,
> > hnswMaxConnections, and hnswBeamWidth parameters then I am able to
> > successfully index and search on this field.
> >
> > Does this appear to be a bug in the code? Or if the parameter is expected
> > to always be set, perhaps a bug in the documentation.
> >
> > Thanks,
> > Alastair
> >
>
Re: Dense Vector Search fieldType default parameters result in index error
Posted by Alessandro Benedetti <a....@sease.io>.
Hi Alastair,
That's definitely not intended and we'll fix it as soon as possible (it
will be available in the next release).
Feel free to create a Jira issue on the Apache Solr project with this
details (and I'll be able to link it in the Pull Request)
Cheers
On Wed, 14 Dec 2022, 04:22 Alastair Porter, <al...@porter.net.nz> wrote:
> Hi there,
> I'm using solr 9.1 to implement dense vector search using the documentation
> at
>
> https://solr.apache.org/guide/solr/latest/query-guide/dense-vector-search.html
>
> According to the guide, a valid fieldType configuration is:
>
> <fieldType name="knn_vector" class="solr.DenseVectorField"
> vectorDimension="4" similarityFunction="cosine"/>
>
> Using the schema API, I added the following configuration:
> $ grep knn_vector managed-schema.xml
> <fieldType name="knn_vector" class="solr.DenseVectorField"
> vectorDimension="4" similarityFunction="euclidean"/>
> <field name="vector" type="knn_vector" multiValued="false"
> stored="true"/>
>
> However, when I try to submit a document containing a vector to the server,
> an error occurs:
> (truncated traceback):
> 2022-12-13 19:10:08.763 ERROR (qtp3540494-54) [ x:fsnn]
> o.a.s.s.HttpSolrCall org.apache.solr.common.SolrException: null KNN
> algorithm is not supported => org.apache.solr.common.SolrException: null
> KNN algorithm is not supported
> at
>
> org.apache.solr.core.SchemaCodecFactory$1.getKnnVectorsFormatForField(SchemaCodecFactory.java:134)
> org.apache.solr.common.SolrException: null KNN algorithm is not supported
> at
>
> org.apache.solr.core.SchemaCodecFactory$1.getKnnVectorsFormatForField(SchemaCodecFactory.java:134)
> ~[?:?]
> at
>
> org.apache.lucene.codecs.lucene92.Lucene92Codec$3.getKnnVectorsFormatForField(Lucene92Codec.java:101)
> ~[?:?]
> at
>
> org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.getInstance(PerFieldKnnVectorsFormat.java:148)
> ~[?:?]
> at
>
> org.apache.lucene.codecs.perfield.PerFieldKnnVectorsFormat$FieldsWriter.writeField(PerFieldKnnVectorsFormat.java:107)
> ~[?:?]
> at
>
> org.apache.lucene.index.VectorValuesWriter.flush(VectorValuesWriter.java:145)
> ~[?:?]
> at
> org.apache.lucene.index.IndexingChain.writeVectors(IndexingChain.java:461)
> ~[?:?]
>
> After having a dig through the source, I found
> https://github.com/apache/solr/pull/910 which added this check. I note
> that
> the documentation about dense vector search does describe the knnAlgorithm
> parameter, but says that it is optional with a default value of "hnsw".
> It seems that the patch changed the behaviour of this check to raise an
> error if the knnAlgorithm is null/missing, rather than only if the value
> isn't the expected "hnsw".
>
> If I update the fieldType in the schema to include the knnAlgorithm,
> hnswMaxConnections, and hnswBeamWidth parameters then I am able to
> successfully index and search on this field.
>
> Does this appear to be a bug in the code? Or if the parameter is expected
> to always be set, perhaps a bug in the documentation.
>
> Thanks,
> Alastair
>