[GitHub] [lucene-solr] jpountz commented on a change in pull request #758: LUCENE-8833: Add a #load() method to IndexInput to allow preloading file content into physical memory

[GitBox <gi...@apache.org>]

jpountz commented on a change in pull request #758: LUCENE-8833: Add a #load() method to IndexInput to allow preloading file content into physical memory
URL: https://github.com/apache/lucene-solr/pull/758#discussion_r299462597
 
 

 ##########
 File path: lucene/core/src/java/org/apache/lucene/store/ByteBufferIndexInput.java
 ##########
 @@ -491,4 +492,14 @@ protected ByteBufferIndexInput buildSlice(String sliceDescription, long ofs, lon
       return super.buildSlice(sliceDescription, this.offset + ofs, length);
     }
   }
+
+  @Override
+  public boolean load() {
+    for (ByteBuffer buffer : buffers) {
+      if (buffer instanceof MappedByteBuffer) {
+        ((MappedByteBuffer) buffer).load();
 
 Review comment:
   This would load more than necessary in the case of a slice. If you look at ByteBufferIndexInput#buildSlice, the offset of the first ByteBuffer might not be updated, so calling load() this way might load up to 1GB of data in main memory that we don't need. This might be especially problematic in the case of CFS files where all index inputs are slices of a single large file. We should maybe override this is MultiBufferImpl to ignore the first `offset` bytes of the first slice?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org