[GitHub] [lucene] uschindler commented on a change in pull request #601: LUCENE-10375: Write merged vectors to file before building graph

[GitBox <gi...@apache.org>]

uschindler commented on a change in pull request #601:
URL: https://github.com/apache/lucene/pull/601#discussion_r784344280



##########
File path: lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90HnswVectorsWriter.java
##########
@@ -155,6 +207,37 @@ public void writeField(FieldInfo fieldInfo, KnnVectorsReader knnVectorsReader)
         count,
         docIds);
     writeGraphOffsets(meta, offsets);
+    if (mergeState.infoStream.isEnabled("VV")) {
+      mergeState.infoStream.message("VV", "merge done " + mergeState.segmentInfo);
+    }
+  }
+
+  private void writeVectorDataPadding() throws IOException {
+    long pos = vectorData.getFilePointer();
+    // write floats aligned at 4 bytes. This will not survive CFS, but it shows a small benefit when
+    // CFS is not used, eg for larger indexes

Review comment:
       Thanks. I spent much time in finding a good API. But replacing getFilePointer by alignFilePointer looked like a beautiful and easy readable idea. Now that CFS files are also aligned and I am happy that we start to align file formats where fixed length data types are used.
   With MMapDirectory version 2 we will have to disable JDK alignment checks explicitly, because by default memory segment reads are enforced to be aligned (the reason here is of course volatile reads to be atomic, not speed).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org