You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "shubhamvishu (via GitHub)" <gi...@apache.org> on 2023/09/11 02:03:49 UTC
[GitHub] [lucene] shubhamvishu opened a new pull request, #12548: Add API to compute vector similarity in DoubleValuesSource
shubhamvishu opened a new pull request, #12548:
URL: https://github.com/apache/lucene/pull/12548
### Description
This PR addresses the issue #12394. It adds an API **`similarityToQueryVector`** to `DoubleValuesSource` to compute vector similarity scores between the query vector and the `KnnByteVectorField`/`KnnFloatVectorField` for documents using the 2 new DVS implementations (`ByteVectorSimilarityValuesSource` for byte vectors and `FloatVectorSimilarityValuesSource` for float vectors). Below are the method signatures added to DVS in this PR:
- `DoubleValues similarityToQueryVector(LeafReaderContext ctx, float[] queryVector, String vectorField) (uses ByteVectorSimilarityValuesSource)`
- `DoubleValues similarityToQueryVector(LeafReaderContext ctx, byte[] queryVector, String vectorField) (uses FloatVectorSimilarityValuesSource)`
Closes #12394
<!--
If this is your first contribution to Lucene, please make sure you have reviewed the contribution guide.
https://github.com/apache/lucene/blob/main/CONTRIBUTING.md
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Ability to compute vector similarity scores with DoubleValuesSource [lucene]
Posted by "shubhamvishu (via GitHub)" <gi...@apache.org>.
shubhamvishu commented on code in PR #12548:
URL: https://github.com/apache/lucene/pull/12548#discussion_r1357312134
##########
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##########
@@ -172,6 +173,52 @@ public LongValuesSource rewrite(IndexSearcher searcher) throws IOException {
}
}
+ /**
+ * Returns a DoubleValues instance for computing the vector similarity score per document against
+ * the byte query vector
+ *
+ * @param ctx the context for which to return the DoubleValues
+ * @param queryVector byte query vector
+ * @param vectorField knn byte field name
+ * @return DoubleValues instance
+ * @throws IOException if an {@link IOException} occurs
+ */
+ public static DoubleValues similarityToQueryVector(
Review Comment:
Yes I didn't mean to move all the static methods here(sorry for the confusion). My idea was along your suggestion, should we also move the private static classes that are in this class(a lot in number, example - [one](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java#L309-L372), [two](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java#L396-L556) and more) which is bloating this class out into another pkg-private class lets say `DoubleValuesSourceUtil` (the only implication being we would make these private classes pkg-private which is fine I think?). This way we could only keep the required API methods in DVS class and move out the rest stuff making this lean and readable. If that makes sense I'll put up these changes too with the followup CR(like below). Let me know what do you think?
Here is an short example commit I made to explain what I was saying - https://github.com/shubhamvishu/lucene/commit/604f1fdcad96a3c5e09356190645b6f2ec16bb2c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] shubhamvishu commented on a diff in pull request #12548: Ability to compute vector similarity scores with DoubleValuesSource
Posted by "shubhamvishu (via GitHub)" <gi...@apache.org>.
shubhamvishu commented on code in PR #12548:
URL: https://github.com/apache/lucene/pull/12548#discussion_r1342112409
##########
lucene/core/src/java/org/apache/lucene/search/ByteVectorSimilarityValuesSource.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Objects;
+import org.apache.lucene.index.ByteVectorValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.VectorSimilarityFunction;
+
+/**
+ * A {@link DoubleValuesSource} which computes the vector similarity scores between the query vector
+ * and the {@link org.apache.lucene.document.KnnByteVectorField} for documents.
+ */
+class ByteVectorSimilarityValuesSource extends DoubleValuesSource {
+ private final byte[] queryVector;
+ private final String fieldName;
+
+ public ByteVectorSimilarityValuesSource(byte[] vector, String fieldName) {
+ this.queryVector = vector;
+ this.fieldName = fieldName;
+ }
+
+ @Override
+ public DoubleValues getValues(LeafReaderContext ctx, DoubleValues scores) throws IOException {
+ final ByteVectorValues vectorValues = ctx.reader().getByteVectorValues(fieldName);
+ VectorSimilarityFunction function =
+ ctx.reader().getFieldInfos().fieldInfo(fieldName).getVectorSimilarityFunction();
+ return new DoubleValues() {
+ @Override
+ public double doubleValue() throws IOException {
+ return function.compare(queryVector, vectorValues.vectorValue());
+ }
+
+ @Override
+ public boolean advanceExact(int doc) throws IOException {
+ if (doc >= vectorValues.docID()
Review Comment:
I'll change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Ability to compute vector similarity scores with DoubleValuesSource [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on code in PR #12548:
URL: https://github.com/apache/lucene/pull/12548#discussion_r1357027508
##########
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##########
@@ -172,6 +173,52 @@ public LongValuesSource rewrite(IndexSearcher searcher) throws IOException {
}
}
+ /**
+ * Returns a DoubleValues instance for computing the vector similarity score per document against
+ * the byte query vector
+ *
+ * @param ctx the context for which to return the DoubleValues
+ * @param queryVector byte query vector
+ * @param vectorField knn byte field name
+ * @return DoubleValues instance
+ * @throws IOException if an {@link IOException} occurs
+ */
+ public static DoubleValues similarityToQueryVector(
Review Comment:
Let's move these to VectorSimilaritySource? This file seem to be a dumping ground for all the things; let's not pile on.
##########
lucene/core/src/java/org/apache/lucene/search/ByteVectorSimilarityValuesSource.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Objects;
+import org.apache.lucene.index.ByteVectorValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.VectorSimilarityFunction;
+
+/**
+ * A {@link DoubleValuesSource} which computes the vector similarity scores between the query vector
Review Comment:
nit: my English profs always said to use "that" in place of "which" when possible
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Ability to compute vector similarity scores with DoubleValuesSource [lucene]
Posted by "shubhamvishu (via GitHub)" <gi...@apache.org>.
shubhamvishu commented on PR #12548:
URL: https://github.com/apache/lucene/pull/12548#issuecomment-1759892123
> oh I missed that @benwtrent had approved -- it's OK w/me to merge as-is, just had some idea about organizing the static methods into a different class...
Thanks @msokolov ! I'll open up a follow up PR to address your ideas if that sounds good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Ability to compute vector similarity scores with DoubleValuesSource [lucene]
Posted by "shubhamvishu (via GitHub)" <gi...@apache.org>.
shubhamvishu commented on PR #12548:
URL: https://github.com/apache/lucene/pull/12548#issuecomment-1761428061
@msokolov @benwtrent Raised a followup PR #12671 for this and also moved some other relevant static classes etc out of DVS(it now looks much better to me). Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Ability to compute vector similarity scores with DoubleValuesSource [lucene]
Posted by "shubhamvishu (via GitHub)" <gi...@apache.org>.
shubhamvishu commented on PR #12548:
URL: https://github.com/apache/lucene/pull/12548#issuecomment-1759689286
Thanks for reviewing!
@benwtrent I have added a CHANGES.txt entry now, could you help me merging this if all looks good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] shubhamvishu commented on a diff in pull request #12548: Ability to compute vector similarity scores with DoubleValuesSource
Posted by "shubhamvishu (via GitHub)" <gi...@apache.org>.
shubhamvishu commented on code in PR #12548:
URL: https://github.com/apache/lucene/pull/12548#discussion_r1342112910
##########
lucene/core/src/java/org/apache/lucene/search/FloatVectorSimilarityValuesSource.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Objects;
+import org.apache.lucene.index.FloatVectorValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.VectorSimilarityFunction;
+
+/**
+ * A {@link DoubleValuesSource} which computes the vector similarity scores between the query vector
+ * and the {@link org.apache.lucene.document.KnnFloatVectorField} for documents.
+ */
+class FloatVectorSimilarityValuesSource extends DoubleValuesSource {
Review Comment:
There are some common bits. I have moved those to `VectorSimilarityValuesSource` now. Let me know if this looks more better or cleaner?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Ability to compute vector similarity scores with DoubleValuesSource [lucene]
Posted by "shubhamvishu (via GitHub)" <gi...@apache.org>.
shubhamvishu commented on code in PR #12548:
URL: https://github.com/apache/lucene/pull/12548#discussion_r1357050474
##########
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##########
@@ -172,6 +173,52 @@ public LongValuesSource rewrite(IndexSearcher searcher) throws IOException {
}
}
+ /**
+ * Returns a DoubleValues instance for computing the vector similarity score per document against
+ * the byte query vector
+ *
+ * @param ctx the context for which to return the DoubleValues
+ * @param queryVector byte query vector
+ * @param vectorField knn byte field name
+ * @return DoubleValues instance
+ * @throws IOException if an {@link IOException} occurs
+ */
+ public static DoubleValues similarityToQueryVector(
Review Comment:
Sure. I see different DVS use cases implemented as static classes in this class. How about we move those static classes and functions(like you pointed) into `DoubleValuesSourceUtil` maybe?. I'll open a followup PR to address that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] shubhamvishu commented on a diff in pull request #12548: Ability to compute vector similarity scores with DoubleValuesSource
Posted by "shubhamvishu (via GitHub)" <gi...@apache.org>.
shubhamvishu commented on code in PR #12548:
URL: https://github.com/apache/lucene/pull/12548#discussion_r1342112773
##########
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##########
@@ -172,6 +173,40 @@ public LongValuesSource rewrite(IndexSearcher searcher) throws IOException {
}
}
+ /**
+ * Returns a DoubleValues instance for computing the vector similarity score per document against
+ * the byte query vector
+ *
+ * @param ctx the context for which to return the DoubleValues
+ * @param queryVector byte query vector
+ * @param vectorField knn byte field name
+ * @return DoubleValues instance
+ * @throws IOException if an {@link IOException} occurs
+ */
+ public static DoubleValues similarityToQueryVector(
+ LeafReaderContext ctx, byte[] queryVector, String vectorField) throws IOException {
+ assert ctx.reader().getFieldInfos().fieldInfo(vectorField).getVectorEncoding()
Review Comment:
Makes sense! But I wonder if thats the case in many places in codebase?. I think we should rather throw an `IAE` in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] shubhamvishu commented on a diff in pull request #12548: Ability to compute vector similarity scores with DoubleValuesSource
Posted by "shubhamvishu (via GitHub)" <gi...@apache.org>.
shubhamvishu commented on code in PR #12548:
URL: https://github.com/apache/lucene/pull/12548#discussion_r1342113181
##########
lucene/core/src/test/org/apache/lucene/search/TestVectorSimilarityValuesSource.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.Field;
+import org.apache.lucene.document.KnnByteVectorField;
+import org.apache.lucene.document.KnnFloatVectorField;
+import org.apache.lucene.document.SortedDocValuesField;
+import org.apache.lucene.document.StringField;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.tests.analysis.MockAnalyzer;
+import org.apache.lucene.tests.index.RandomIndexWriter;
+import org.apache.lucene.tests.util.LuceneTestCase;
+import org.apache.lucene.util.BytesRef;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+
+public class TestVectorSimilarityValuesSource extends LuceneTestCase {
+ private static Directory dir;
+ private static Analyzer analyzer;
+ private static IndexReader reader;
+ private static IndexSearcher searcher;
+
+ @BeforeClass
+ public static void beforeClass() throws Exception {
+ dir = newDirectory();
+ analyzer = new MockAnalyzer(random());
+ IndexWriterConfig iwConfig = newIndexWriterConfig(analyzer);
+ iwConfig.setMergePolicy(newLogMergePolicy());
+ RandomIndexWriter iw = new RandomIndexWriter(random(), dir, iwConfig);
+
+ Document document = new Document();
+ document.add(new StringField("id", "1", Field.Store.NO));
+ document.add(new SortedDocValuesField("id", new BytesRef("1")));
+ document.add(new KnnFloatVectorField("knnFloatField1", new float[] {1.f, 2.f, 3.f}));
+ document.add(
+ new KnnFloatVectorField(
+ "knnFloatField2",
+ new float[] {2.2f, -3.2f, -3.1f},
+ VectorSimilarityFunction.DOT_PRODUCT));
+ document.add(
+ new KnnFloatVectorField(
+ "knnFloatField3", new float[] {4.5f, 10.3f, -7.f}, VectorSimilarityFunction.COSINE));
+ document.add(
+ new KnnFloatVectorField(
+ "knnFloatField4",
+ new float[] {-1.3f, 1.0f, 1.0f},
+ VectorSimilarityFunction.MAXIMUM_INNER_PRODUCT));
+ document.add(new KnnFloatVectorField("knnFloatField5", new float[] {-6.7f, -1.0f, -0.9f}));
+ document.add(new KnnByteVectorField("knnByteField1", new byte[] {106, 80, 127}));
+ document.add(
+ new KnnByteVectorField(
+ "knnByteField2", new byte[] {4, 2, 3}, VectorSimilarityFunction.DOT_PRODUCT));
+ document.add(
+ new KnnByteVectorField(
+ "knnByteField3", new byte[] {-121, -64, -1}, VectorSimilarityFunction.COSINE));
+ document.add(
+ new KnnByteVectorField(
+ "knnByteField4",
+ new byte[] {-127, 127, 127},
+ VectorSimilarityFunction.MAXIMUM_INNER_PRODUCT));
+ iw.addDocument(document);
+
+ Document document2 = new Document();
+ document2.add(new StringField("id", "2", Field.Store.NO));
+ document2.add(new SortedDocValuesField("id", new BytesRef("2")));
+ document2.add(new KnnFloatVectorField("knnFloatField1", new float[] {1.f, 2.f, 3.f}));
+ document2.add(
+ new KnnFloatVectorField(
+ "knnFloatField2",
+ new float[] {-5.2f, 8.7f, 3.1f},
+ VectorSimilarityFunction.DOT_PRODUCT));
+ document2.add(
+ new KnnFloatVectorField(
+ "knnFloatField3", new float[] {0.2f, -3.2f, 3.1f}, VectorSimilarityFunction.COSINE));
+ document2.add(new KnnFloatVectorField("knnFloatField5", new float[] {2.f, 13.2f, 9.1f}));
+ document2.add(new KnnByteVectorField("knnByteField1", new byte[] {1, -2, -30}));
+ document2.add(
+ new KnnByteVectorField(
+ "knnByteField2", new byte[] {40, 21, 3}, VectorSimilarityFunction.DOT_PRODUCT));
+ document2.add(
+ new KnnByteVectorField(
+ "knnByteField3", new byte[] {9, 2, 3}, VectorSimilarityFunction.COSINE));
+ document2.add(
+ new KnnByteVectorField(
+ "knnByteField4",
+ new byte[] {14, 29, 31},
+ VectorSimilarityFunction.MAXIMUM_INNER_PRODUCT));
+ iw.addDocument(document2);
+
+ Document document3 = new Document();
+ document3.add(new StringField("id", "3", Field.Store.NO));
+ document3.add(new SortedDocValuesField("id", new BytesRef("3")));
+ document3.add(new KnnFloatVectorField("knnFloatField1", new float[] {1.f, 2.f, 3.f}));
+ document3.add(
+ new KnnFloatVectorField(
+ "knnFloatField2", new float[] {-8.f, 7.f, -6.f}, VectorSimilarityFunction.DOT_PRODUCT));
+ document3.add(new KnnFloatVectorField("knnFloatField5", new float[] {5.2f, 3.2f, 3.1f}));
+ document3.add(new KnnByteVectorField("knnByteField1", new byte[] {-128, 0, 127}));
+ document3.add(
+ new KnnByteVectorField(
+ "knnByteField2", new byte[] {-1, -2, -3}, VectorSimilarityFunction.DOT_PRODUCT));
+ document3.add(
+ new KnnByteVectorField(
+ "knnByteField3", new byte[] {4, 2, 3}, VectorSimilarityFunction.COSINE));
+ document3.add(
+ new KnnByteVectorField(
+ "knnByteField4",
+ new byte[] {-4, -2, -128},
+ VectorSimilarityFunction.MAXIMUM_INNER_PRODUCT));
+ document3.add(new KnnByteVectorField("knnByteField5", new byte[] {-120, -2, 3}));
+ iw.addDocument(document3);
+
+ reader = iw.getReader();
+ searcher = newSearcher(reader);
+ iw.close();
+ }
+
+ @AfterClass
+ public static void afterClass() throws Exception {
Review Comment:
Ahh good catch! It didn't struck my eyes. I'll change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Ability to compute vector similarity scores with DoubleValuesSource [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #12548:
URL: https://github.com/apache/lucene/pull/12548#issuecomment-1759701427
@shubhamvishu running CI :). I will see about merging and such soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Ability to compute vector similarity scores with DoubleValuesSource [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent commented on PR #12548:
URL: https://github.com/apache/lucene/pull/12548#issuecomment-1760058838
These are good ideas @msokolov , @shubhamvishu a follow up PR is most welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Ability to compute vector similarity scores with DoubleValuesSource [lucene]
Posted by "shubhamvishu (via GitHub)" <gi...@apache.org>.
shubhamvishu commented on code in PR #12548:
URL: https://github.com/apache/lucene/pull/12548#discussion_r1357050474
##########
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##########
@@ -172,6 +173,52 @@ public LongValuesSource rewrite(IndexSearcher searcher) throws IOException {
}
}
+ /**
+ * Returns a DoubleValues instance for computing the vector similarity score per document against
+ * the byte query vector
+ *
+ * @param ctx the context for which to return the DoubleValues
+ * @param queryVector byte query vector
+ * @param vectorField knn byte field name
+ * @return DoubleValues instance
+ * @throws IOException if an {@link IOException} occurs
+ */
+ public static DoubleValues similarityToQueryVector(
Review Comment:
Sure. I see different DVS use cases implemented as static classes here making this less readable. How about we move those static classes and functions(like you pointed) into a new class `DoubleValuesSourceUtil` maybe(only thing those classes are private currently)?. I'll open a followup PR to address that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Ability to compute vector similarity scores with DoubleValuesSource [lucene]
Posted by "stefanvodita (via GitHub)" <gi...@apache.org>.
stefanvodita commented on code in PR #12548:
URL: https://github.com/apache/lucene/pull/12548#discussion_r1347266594
##########
lucene/core/src/java/org/apache/lucene/search/FloatVectorSimilarityValuesSource.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Objects;
+import org.apache.lucene.index.FloatVectorValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.VectorSimilarityFunction;
+
+/**
+ * A {@link DoubleValuesSource} which computes the vector similarity scores between the query vector
+ * and the {@link org.apache.lucene.document.KnnFloatVectorField} for documents.
+ */
+class FloatVectorSimilarityValuesSource extends DoubleValuesSource {
Review Comment:
Great! I think this is the right approach.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] stefanvodita commented on a diff in pull request #12548: Ability to compute vector similarity scores with DoubleValuesSource
Posted by "stefanvodita (via GitHub)" <gi...@apache.org>.
stefanvodita commented on code in PR #12548:
URL: https://github.com/apache/lucene/pull/12548#discussion_r1341952008
##########
lucene/core/src/test/org/apache/lucene/search/TestVectorSimilarityValuesSource.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.Field;
+import org.apache.lucene.document.KnnByteVectorField;
+import org.apache.lucene.document.KnnFloatVectorField;
+import org.apache.lucene.document.SortedDocValuesField;
+import org.apache.lucene.document.StringField;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.tests.analysis.MockAnalyzer;
+import org.apache.lucene.tests.index.RandomIndexWriter;
+import org.apache.lucene.tests.util.LuceneTestCase;
+import org.apache.lucene.util.BytesRef;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+
+public class TestVectorSimilarityValuesSource extends LuceneTestCase {
+ private static Directory dir;
+ private static Analyzer analyzer;
+ private static IndexReader reader;
+ private static IndexSearcher searcher;
+
+ @BeforeClass
+ public static void beforeClass() throws Exception {
+ dir = newDirectory();
+ analyzer = new MockAnalyzer(random());
+ IndexWriterConfig iwConfig = newIndexWriterConfig(analyzer);
+ iwConfig.setMergePolicy(newLogMergePolicy());
+ RandomIndexWriter iw = new RandomIndexWriter(random(), dir, iwConfig);
+
+ Document document = new Document();
+ document.add(new StringField("id", "1", Field.Store.NO));
+ document.add(new SortedDocValuesField("id", new BytesRef("1")));
+ document.add(new KnnFloatVectorField("knnFloatField1", new float[] {1.f, 2.f, 3.f}));
+ document.add(
+ new KnnFloatVectorField(
+ "knnFloatField2",
+ new float[] {2.2f, -3.2f, -3.1f},
+ VectorSimilarityFunction.DOT_PRODUCT));
+ document.add(
+ new KnnFloatVectorField(
+ "knnFloatField3", new float[] {4.5f, 10.3f, -7.f}, VectorSimilarityFunction.COSINE));
+ document.add(
+ new KnnFloatVectorField(
+ "knnFloatField4",
+ new float[] {-1.3f, 1.0f, 1.0f},
+ VectorSimilarityFunction.MAXIMUM_INNER_PRODUCT));
+ document.add(new KnnFloatVectorField("knnFloatField5", new float[] {-6.7f, -1.0f, -0.9f}));
+ document.add(new KnnByteVectorField("knnByteField1", new byte[] {106, 80, 127}));
+ document.add(
+ new KnnByteVectorField(
+ "knnByteField2", new byte[] {4, 2, 3}, VectorSimilarityFunction.DOT_PRODUCT));
+ document.add(
+ new KnnByteVectorField(
+ "knnByteField3", new byte[] {-121, -64, -1}, VectorSimilarityFunction.COSINE));
+ document.add(
+ new KnnByteVectorField(
+ "knnByteField4",
+ new byte[] {-127, 127, 127},
+ VectorSimilarityFunction.MAXIMUM_INNER_PRODUCT));
+ iw.addDocument(document);
+
+ Document document2 = new Document();
+ document2.add(new StringField("id", "2", Field.Store.NO));
+ document2.add(new SortedDocValuesField("id", new BytesRef("2")));
+ document2.add(new KnnFloatVectorField("knnFloatField1", new float[] {1.f, 2.f, 3.f}));
+ document2.add(
+ new KnnFloatVectorField(
+ "knnFloatField2",
+ new float[] {-5.2f, 8.7f, 3.1f},
+ VectorSimilarityFunction.DOT_PRODUCT));
+ document2.add(
+ new KnnFloatVectorField(
+ "knnFloatField3", new float[] {0.2f, -3.2f, 3.1f}, VectorSimilarityFunction.COSINE));
+ document2.add(new KnnFloatVectorField("knnFloatField5", new float[] {2.f, 13.2f, 9.1f}));
+ document2.add(new KnnByteVectorField("knnByteField1", new byte[] {1, -2, -30}));
+ document2.add(
+ new KnnByteVectorField(
+ "knnByteField2", new byte[] {40, 21, 3}, VectorSimilarityFunction.DOT_PRODUCT));
+ document2.add(
+ new KnnByteVectorField(
+ "knnByteField3", new byte[] {9, 2, 3}, VectorSimilarityFunction.COSINE));
+ document2.add(
+ new KnnByteVectorField(
+ "knnByteField4",
+ new byte[] {14, 29, 31},
+ VectorSimilarityFunction.MAXIMUM_INNER_PRODUCT));
+ iw.addDocument(document2);
+
+ Document document3 = new Document();
+ document3.add(new StringField("id", "3", Field.Store.NO));
+ document3.add(new SortedDocValuesField("id", new BytesRef("3")));
+ document3.add(new KnnFloatVectorField("knnFloatField1", new float[] {1.f, 2.f, 3.f}));
+ document3.add(
+ new KnnFloatVectorField(
+ "knnFloatField2", new float[] {-8.f, 7.f, -6.f}, VectorSimilarityFunction.DOT_PRODUCT));
+ document3.add(new KnnFloatVectorField("knnFloatField5", new float[] {5.2f, 3.2f, 3.1f}));
+ document3.add(new KnnByteVectorField("knnByteField1", new byte[] {-128, 0, 127}));
+ document3.add(
+ new KnnByteVectorField(
+ "knnByteField2", new byte[] {-1, -2, -3}, VectorSimilarityFunction.DOT_PRODUCT));
+ document3.add(
+ new KnnByteVectorField(
+ "knnByteField3", new byte[] {4, 2, 3}, VectorSimilarityFunction.COSINE));
+ document3.add(
+ new KnnByteVectorField(
+ "knnByteField4",
+ new byte[] {-4, -2, -128},
+ VectorSimilarityFunction.MAXIMUM_INNER_PRODUCT));
+ document3.add(new KnnByteVectorField("knnByteField5", new byte[] {-120, -2, 3}));
+ iw.addDocument(document3);
+
+ reader = iw.getReader();
+ searcher = newSearcher(reader);
+ iw.close();
+ }
+
+ @AfterClass
+ public static void afterClass() throws Exception {
Review Comment:
You've previously committed a [change](https://github.com/apache/lucene/commit/368dbffef3d2d44256fe63d4f026e76e211d9eb6) replacing many separate `.close()` calls with `IOUtils.close()`. Why not use that here?
##########
lucene/core/src/java/org/apache/lucene/search/FloatVectorSimilarityValuesSource.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Objects;
+import org.apache.lucene.index.FloatVectorValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.VectorSimilarityFunction;
+
+/**
+ * A {@link DoubleValuesSource} which computes the vector similarity scores between the query vector
+ * and the {@link org.apache.lucene.document.KnnFloatVectorField} for documents.
+ */
+class FloatVectorSimilarityValuesSource extends DoubleValuesSource {
Review Comment:
This and `BytesVectorSimilarityValuesSource` look very similar. Should they inherit from a `VectorSimilarityValuesSource` to ensure we can add common functionality in the future?
##########
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##########
@@ -172,6 +173,40 @@ public LongValuesSource rewrite(IndexSearcher searcher) throws IOException {
}
}
+ /**
+ * Returns a DoubleValues instance for computing the vector similarity score per document against
+ * the byte query vector
+ *
+ * @param ctx the context for which to return the DoubleValues
+ * @param queryVector byte query vector
+ * @param vectorField knn byte field name
+ * @return DoubleValues instance
+ * @throws IOException if an {@link IOException} occurs
+ */
+ public static DoubleValues similarityToQueryVector(
+ LeafReaderContext ctx, byte[] queryVector, String vectorField) throws IOException {
+ assert ctx.reader().getFieldInfos().fieldInfo(vectorField).getVectorEncoding()
Review Comment:
What do you think of throwing an exception even when assertions are not enabled?
##########
lucene/core/src/java/org/apache/lucene/search/ByteVectorSimilarityValuesSource.java:
##########
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.Objects;
+import org.apache.lucene.index.ByteVectorValues;
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.VectorSimilarityFunction;
+
+/**
+ * A {@link DoubleValuesSource} which computes the vector similarity scores between the query vector
+ * and the {@link org.apache.lucene.document.KnnByteVectorField} for documents.
+ */
+class ByteVectorSimilarityValuesSource extends DoubleValuesSource {
+ private final byte[] queryVector;
+ private final String fieldName;
+
+ public ByteVectorSimilarityValuesSource(byte[] vector, String fieldName) {
+ this.queryVector = vector;
+ this.fieldName = fieldName;
+ }
+
+ @Override
+ public DoubleValues getValues(LeafReaderContext ctx, DoubleValues scores) throws IOException {
+ final ByteVectorValues vectorValues = ctx.reader().getByteVectorValues(fieldName);
+ VectorSimilarityFunction function =
+ ctx.reader().getFieldInfos().fieldInfo(fieldName).getVectorSimilarityFunction();
+ return new DoubleValues() {
+ @Override
+ public double doubleValue() throws IOException {
+ return function.compare(queryVector, vectorValues.vectorValue());
+ }
+
+ @Override
+ public boolean advanceExact(int doc) throws IOException {
+ if (doc >= vectorValues.docID()
Review Comment:
This is equivalent to `return doc >= vectorValues.docID() && (vectorValues.docID() == doc || vectorValues.advance(doc) == doc);`, which avoids the branch.
##########
lucene/core/src/test/org/apache/lucene/search/TestVectorSimilarityValuesSource.java:
##########
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.search;
+
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.Field;
+import org.apache.lucene.document.KnnByteVectorField;
+import org.apache.lucene.document.KnnFloatVectorField;
+import org.apache.lucene.document.SortedDocValuesField;
+import org.apache.lucene.document.StringField;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.index.VectorSimilarityFunction;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.tests.analysis.MockAnalyzer;
+import org.apache.lucene.tests.index.RandomIndexWriter;
+import org.apache.lucene.tests.util.LuceneTestCase;
+import org.apache.lucene.util.BytesRef;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+
+public class TestVectorSimilarityValuesSource extends LuceneTestCase {
Review Comment:
Thank you for adding a thorough test!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Ability to compute vector similarity scores with DoubleValuesSource [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on code in PR #12548:
URL: https://github.com/apache/lucene/pull/12548#discussion_r1357190877
##########
lucene/core/src/java/org/apache/lucene/search/DoubleValuesSource.java:
##########
@@ -172,6 +173,52 @@ public LongValuesSource rewrite(IndexSearcher searcher) throws IOException {
}
}
+ /**
+ * Returns a DoubleValues instance for computing the vector similarity score per document against
+ * the byte query vector
+ *
+ * @param ctx the context for which to return the DoubleValues
+ * @param queryVector byte query vector
+ * @param vectorField knn byte field name
+ * @return DoubleValues instance
+ * @throws IOException if an {@link IOException} occurs
+ */
+ public static DoubleValues similarityToQueryVector(
Review Comment:
well I don't think we should rename *all* the existing static methods here - that would just cause confusion. Maybe that's not what you were saying?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Ability to compute vector similarity scores with DoubleValuesSource [lucene]
Posted by "msokolov (via GitHub)" <gi...@apache.org>.
msokolov commented on PR #12548:
URL: https://github.com/apache/lucene/pull/12548#issuecomment-1759887696
oh I missed that @benwtrent had approved -- it's OK w/me to merge as-is, just had some idea about organizing the static methods into a different class...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Ability to compute vector similarity scores with DoubleValuesSource [lucene]
Posted by "benwtrent (via GitHub)" <gi...@apache.org>.
benwtrent merged PR #12548:
URL: https://github.com/apache/lucene/pull/12548
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org