You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "ChrisHegarty (via GitHub)" <gi...@apache.org> on 2026/09/07 09:45:05 UTC
[PR] Add KnnVectorsReader.getVectorCount to read vector counts from metadata [lucene]
ChrisHegarty opened a new pull request, #16647:
URL: https://github.com/apache/lucene/pull/16647
I need to count how many vectors a field has in a segment without opening the vector values. Today the only way to do that is` getFloatVectorValues` or `getByteVectorValues` and then call `size()`. That opens the full vector values object. For sparse fields that builds a `IndexedDISI` and can prefetch index data from disk. Elasticsearch hits this on cluster stats: it walks every dense vector field on every segment just to sum counts, and on remote-backed storage each open can pull a whole cache region. We worked around it by skipping counts on stateless and caching on stateful, but the count is already in the segment metadata next to `vectorDataLength`, which `getOffHeapByteSize` reads for free.
This change adds `KnnVectorsReader.getVectorCount(FieldInfo)`. Standard formats override it to return the size already loaded from the metadata at segment open. No index format change. The default implementation still opens vector values as a fallback so custom codecs stay correct.. I followed the same pattern as `getOffHeapByteSize`: one method on `KnnVectorsReader`, with overrides in Lucene99, Lucene104, backward codecs, SimpleText, Dedup, Faiss, and the usual wrappers.
I added `testVectorCount` in `BaseKnnVectorsFormatTestCase`. It indexes dense fields and sparse fields in the same segment and checks `getVectorCount` matches opening the values. `getNumVectors` in the test base now uses the new API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add KnnVectorsReader.getVectorCount to read vector counts from metadata [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty merged PR #16647:
URL: https://github.com/apache/lucene/pull/16647
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add KnnVectorsReader.getVectorCount to read vector counts from metadata [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on code in PR #16647:
URL: https://github.com/apache/lucene/pull/16647#discussion_r3949474282
##########
lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsReader.java:
##########
@@ -185,6 +185,27 @@ public KnnVectorsReader getMergeInstance() throws IOException {
*/
public void finishMerge() throws IOException {}
+ /**
+ * Returns the number of indexed vectors for the given field in this segment.
+ *
+ * <p>This has the same meaning as {@code get*VectorValues(field).size()}: the number of vector
+ * ordinals stored for {@code fieldInfo} in this segment.
+ *
+ * <p>Standard vector formats read this from segment metadata without opening vector values. The
+ * default implementation opens vector values as a fallback; callers that need to avoid that I/O
+ * should use a reader that overrides this method.
+ *
+ * @param fieldInfo the fieldInfo
+ * @return the number of indexed vectors for the field
+ */
+ public int getVectorCount(FieldInfo fieldInfo) throws IOException {
Review Comment:
Good idea. I updated the Javadoc with @throws IllegalArgumentException and the I/O/prefetch motivation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add KnnVectorsReader.getVectorCount to read vector counts from metadata [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on code in PR #16647:
URL: https://github.com/apache/lucene/pull/16647#discussion_r3949468839
##########
lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene90/Lucene90HnswVectorsReader.java:
##########
@@ -331,6 +331,11 @@ public Map<String, Long> getOffHeapByteSize(FieldInfo fieldInfo) {
return Map.ofEntries(raw, graph);
}
+ @Override
+ public int getVectorCount(FieldInfo fieldInfo) {
+ return getFieldEntry(fieldInfo.name).ordToDoc.length;
Review Comment:
D'oh! done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add KnnVectorsReader.getVectorCount to read vector counts from metadata [lucene]
Posted by "jimczi (via GitHub)" <gi...@apache.org>.
jimczi commented on code in PR #16647:
URL: https://github.com/apache/lucene/pull/16647#discussion_r3949119731
##########
lucene/test-framework/src/java/org/apache/lucene/tests/index/BaseKnnVectorsFormatTestCase.java:
##########
@@ -2499,10 +2588,41 @@ protected void assertOffHeapByteSize(LeafReader r, String fieldName) throws IOEx
protected static int getNumVectors(KnnVectorsReader reader, FieldInfo fieldInfo)
throws IOException {
+ return reader.getVectorCount(fieldInfo);
+ }
+
+ protected void assertVectorCountMatchesVectorValuesSize(LeafReader leafReader)
+ throws IOException {
+ if (leafReader instanceof CodecReader codecReader) {
+ KnnVectorsReader vectorsReader = codecReader.getVectorReader();
+ for (FieldInfo fieldInfo : leafReader.getFieldInfos()) {
+ if (fieldInfo.getVectorDimension() <= 0) {
+ continue;
+ }
+ KnnVectorsReader fieldReader = vectorsReader.unwrapReaderForField(fieldInfo.name);
Review Comment:
Unwrapping here means the `PerFieldKnnVectorsFormat` override is never exercised, and that's the one every caller reaches via `CodecReader.getVectorReader()`. Same for the `SlowCodecReaderWrapper` / `SlowCompositeCodecReaderWrapper` / `SortingCodecReader` overrides. Asserting both the unwrapped and non-unwrapped reader would cover them.
##########
lucene/backward-codecs/src/java/org/apache/lucene/backward_codecs/lucene90/Lucene90HnswVectorsReader.java:
##########
@@ -331,6 +331,11 @@ public Map<String, Long> getOffHeapByteSize(FieldInfo fieldInfo) {
return Map.ofEntries(raw, graph);
}
+ @Override
+ public int getVectorCount(FieldInfo fieldInfo) {
+ return getFieldEntry(fieldInfo.name).ordToDoc.length;
Review Comment:
`FieldEntry.size()` (line 384) returns exactly this - use it, like the 91/92/94/95 overrides do.
##########
lucene/core/src/java/org/apache/lucene/codecs/KnnVectorsReader.java:
##########
@@ -185,6 +185,27 @@ public KnnVectorsReader getMergeInstance() throws IOException {
*/
public void finishMerge() throws IOException {}
+ /**
+ * Returns the number of indexed vectors for the given field in this segment.
+ *
+ * <p>This has the same meaning as {@code get*VectorValues(field).size()}: the number of vector
+ * ordinals stored for {@code fieldInfo} in this segment.
+ *
+ * <p>Standard vector formats read this from segment metadata without opening vector values. The
+ * default implementation opens vector values as a fallback; callers that need to avoid that I/O
+ * should use a reader that overrides this method.
+ *
+ * @param fieldInfo the fieldInfo
+ * @return the number of indexed vectors for the field
+ */
+ public int getVectorCount(FieldInfo fieldInfo) throws IOException {
Review Comment:
Worth pinning down what happens for a field that isn't vector-indexed - implementations disagree today: `IllegalArgumentException` in most, NPE in `PerFieldKnnVectorsFormat`, `0` in `KnnVectorsFormat.EMPTY`. `@throws IllegalArgumentException` to match `getFloatVectorValues` would do it.
Also worth stating here that opening vector values may do I/O and prefetch by design - that's what justifies this method existing, and it's the same reason doc values grew `DocValuesSkipper` instead of a cheaper iterator.
##########
lucene/core/src/java/org/apache/lucene/index/SlowCodecReaderWrapper.java:
##########
@@ -213,6 +213,14 @@ public Map<String, Long> getOffHeapByteSize(FieldInfo fieldInfo) {
return vectorsReader.getOffHeapByteSize(fieldInfo);
}
+ @Override
+ public int getVectorCount(FieldInfo fieldInfo) throws IOException {
Review Comment:
This is only reachable from the non-`CodecReader` branch of `wrap()`, i.e. arbitrary leaf readers, but `segmentReader(reader)` throws on anything that doesn't unwrap to a `SegmentReader`. With a `ParallelLeafReader` over a segment:
```
values.size() -> 10
getVectorCount(fi) -> java.lang.AssertionError: unexpected reader [ParallelLeafReader(_1(11.0.0):c10)]
```
The inherited default delegates to `reader.getXVectorValues(field).size()`, which is correct here and can't fail. Can we drop the override, or fall back to `super.getVectorCount(fieldInfo)` when the leaf doesn't unwrap? (`getOffHeapByteSize` above has the same issue - pre-existing, separate.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add KnnVectorsReader.getVectorCount to read vector counts from metadata [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on code in PR #16647:
URL: https://github.com/apache/lucene/pull/16647#discussion_r3949454102
##########
lucene/core/src/java/org/apache/lucene/index/SlowCodecReaderWrapper.java:
##########
@@ -213,6 +213,14 @@ public Map<String, Long> getOffHeapByteSize(FieldInfo fieldInfo) {
return vectorsReader.getOffHeapByteSize(fieldInfo);
}
+ @Override
+ public int getVectorCount(FieldInfo fieldInfo) throws IOException {
Review Comment:
Good catch. Added unwrapSegmentReader() that returns null when the leaf doesn't unwrap to a SegmentReader, and getVectorCount now falls back to super.getVectorCount(fieldInfo) in that case. The fast metadata path is still used when unwrap succeeds.
I left getOffHeapByteSize as is for now, but will follow up separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
Re: [PR] Add KnnVectorsReader.getVectorCount to read vector counts from metadata [lucene]
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on code in PR #16647:
URL: https://github.com/apache/lucene/pull/16647#discussion_r3949467036
##########
lucene/test-framework/src/java/org/apache/lucene/tests/index/BaseKnnVectorsFormatTestCase.java:
##########
@@ -2499,10 +2588,41 @@ protected void assertOffHeapByteSize(LeafReader r, String fieldName) throws IOEx
protected static int getNumVectors(KnnVectorsReader reader, FieldInfo fieldInfo)
throws IOException {
+ return reader.getVectorCount(fieldInfo);
+ }
+
+ protected void assertVectorCountMatchesVectorValuesSize(LeafReader leafReader)
+ throws IOException {
+ if (leafReader instanceof CodecReader codecReader) {
+ KnnVectorsReader vectorsReader = codecReader.getVectorReader();
+ for (FieldInfo fieldInfo : leafReader.getFieldInfos()) {
+ if (fieldInfo.getVectorDimension() <= 0) {
+ continue;
+ }
+ KnnVectorsReader fieldReader = vectorsReader.unwrapReaderForField(fieldInfo.name);
Review Comment:
I updated assertVectorCountMatchesVectorValuesSize to assert both paths for each field. And testVectorCount now also runs the assertion on SlowCodecReaderWrapper.wrap(...) and SortingCodecReader.wrap(...).
I also added TestGetVectorCountCodecWrappers, to cover the package-private SlowCompositeCodecReaderWrapper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org