You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "ChrisHegarty (via GitHub)" <gi...@apache.org> on 2023/08/01 15:57:56 UTC

[GitHub] [lucene] ChrisHegarty opened a new issue, #12482: Examine the potential of loading vector data directly from the memory segment

ChrisHegarty opened a new issue, #12482:
URL: https://github.com/apache/lucene/issues/12482

   ### Description
   
   Just a crazy idea! Thanks @jpountz ;-)
   
   The Panama Vector API supports for loading directly from a memory segment. If we could do this, then for vector similarity purposes we could load the underlying vector data directly from the MemorySegment (rather than the current copy from the segment into a float[] ).
   
   A quick experiment, if I did it correctly, and only with float dotProduct, shows approx a 2x performance improvement. ( the experiment builds upon @rmuir 's vectorbench, which we used for the original work that integrated the Panama Vector API )
   
   ```
   Linux, AVX 512
   Benchmark                                             (size)   Mode  Cnt   Score   Error   Units
   FloatDotProductBenchmark.dotProductCopyFromArray        1024  thrpt    5   9.714 ± 0.002  ops/us
   FloatDotProductBenchmark.dotProductFromMemorySegment    1024  thrpt    5  18.111 ± 0.008  ops/us
   ```
   https://github.com/ChrisHegarty/vectorbench/blob/5380ebf01df77e1c56d9f392a391df780c19d35e/src/main/java/testing/FloatDotProductBenchmark.java
   
   The benchmark method, `dotProductCopyFromArray`, is what we have in Lucene today, but I added a copy from the memory segment into the two float[]'s to simulate `MemorySegmentIndexInput::readFloats`.
   
   The benchmark method, `dotProductFromMemorySegment`, has the exact same shape of vector operations, just that the values are loaded directly from the memory segment using `FloatVector::fromMemorySegment`. A single segment holds both vectors. I suspect we'd want similar in Lucene too is this were ever to proceed.
   
   In practice I don't see any straightforward way that we could approach this, and there is the complication of using preview / Incubating JDK APIs. That aside, I think that the general idea is sound - avoid the extra intermediate copy to the float[].
   
   I filed this issue mostly to raise awareness of `fromMemorySegment`, and also to provoke a discussion and get thoughts from others. Is it worth trying to pursue? Maybe just crudely hack something and get rough luceneutil bench results to see if / how much it could affect things? `MemorySegmentIndexInput` could expose a way to return XXVector's from a given position? Something else?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] ChrisHegarty commented on issue #12482: Examine the potential of loading vector data directly from the memory segment

Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on issue #12482:
URL: https://github.com/apache/lucene/issues/12482#issuecomment-1660636619

   Note: there is memory segment -> byte buffer interop, which we could use for `MappedByteBufferIndexInputProvider`, so we could do something there too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] uschindler commented on issue #12482: Examine the potential of loading vector data directly from the memory segment

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #12482:
URL: https://github.com/apache/lucene/issues/12482#issuecomment-1660657815

   I have something like this on the agenda but it's currently impossible with non released API and Java 11.
   
   There is another solution: ByteBuffer is still there and you can create vector as view on ByteBuffer. So we could add some DataInput method which returns a byte buffer view on a segment (if the vector is on buffer boundary the copy would of course take place).
   
   Please take in mind that I don't trust the benchmark fully. There are more complications not tested here: you have to make a slice first and so on.
   
   If someone uses another indexinput the default impl would use readBytes or readFloat and wrap it with a heap buffer.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] uschindler commented on issue #12482: Examine the potential of loading vector data directly from the memory segment

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #12482:
URL: https://github.com/apache/lucene/issues/12482#issuecomment-1660738848

   Hi,
   
   > > The problem with older panama vector apis (around java 17) was that there was no real byte/float buffer interop available. Implementations at that time still copied to heap.
   > 
   > Ah ok. I think I'm still missing something. We don't want a ByteBuffer or MemorySegment per vector, right? This would be too much garbage. We just want to return a VectorXX loaded with the values from a given position and length.
   
   It creates garbage in all cases. It does not matter if it returns a slice of a MemorySegment or a Buffer. Both of them will be killed by escape analysis. I see no problem with this. We can't fully remove all instances, because vectors are non mutable. So the DataInput must return a new instance anyways. It does not matter if it is a FloatBuffer or FloatVector.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] ChrisHegarty commented on issue #12482: Examine the potential of loading vector data directly from the memory segment

Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on issue #12482:
URL: https://github.com/apache/lucene/issues/12482#issuecomment-1660728819

   Thanks for engaging :-)
   
   > * boundary conditions: if the file is ginormous, we don't just map it to a single memory segment but multiple segments i think, how can any new api prevent this complexity from working its way into the vector formulas (which are already complicated enough, this would be too much)
   
   Yeah, as @uschindler said, depending on how this was structured we likely fallback to a copy. I don't expect this to be too common as the default is 16GB. But of course it has to work correctly.
   
   > * alignment issues: afaik, with heap arrays there is some amount of alignment by the jvm, but with mapped files this may not be the case unless we do extra work. do we care? what is the penalty (on common arm/x86 cpus)?
   
   You are spot on, alignment does affect things. As it already does - when reading from an unaligned location in the segment to a float[]. I quickly updated the micro benchmark to read from an offset of one byte in the map file. The readFloats / memory segment copy hurts a lot! While the vector load fromMemorySegment hurts a lot less. This by itself deserves some investigation.
   
   ```
   Benchmark                                             (size)   Mode  Cnt   Score   Error   Units
   FloatDotProductBenchmark.dotProductCopyFromArray        1024  thrpt    5   0.697 ± 0.002  ops/us
   FloatDotProductBenchmark.dotProductFromMemorySegment    1024  thrpt    5  15.864 ± 0.005  ops/us
   ```
   
   > Please note: the idea for this issue was already discussed in the original mmapdir issues. I proposed this already there.
   
   Ok. So it is not new, and not crazy. Cool!
   
   >The problem with older panama vector apis (around java 17) was that there was no real byte/float buffer interop available. Implementations at that time still copied to heap.
   
   Ah ok. I think I'm still missing something. We don't want a ByteBuffer or MemorySegment per vector, right? This would be too much garbage. We just want to return a VectorXX loaded with the values from a given position and length.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] uschindler commented on issue #12482: Examine the potential of loading vector data directly from the memory segment

Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #12482:
URL: https://github.com/apache/lucene/issues/12482#issuecomment-1660707500

   Please note: the idea for this issue was already discussed in the original mmapdir issues. I proposed this already there.
   The problem with older panama vector apis (around java 17) was that there was no real byte/float buffer interop available.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] ChrisHegarty commented on issue #12482: Examine the potential of loading vector data directly from the memory segment

Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on issue #12482:
URL: https://github.com/apache/lucene/issues/12482#issuecomment-1661909114

   While the benchmark is just a micro, it can offer some insights, especially into the aligned / unaligned aspect of this. I updated it with aligned / unaligned variants, [here][1]. TLDR there is little difference between aligned and unaligned.
   
   ```
   Mac m1
   Benchmark                                                      (size)   Mode  Cnt  Score   Error   Units
   FloatDotProductBenchmark.dotProductCopyFromArrayAligned          1024  thrpt    5  5.592 ± 0.241  ops/us
   FloatDotProductBenchmark.dotProductCopyFromArrayUnaligned        1024  thrpt    5  4.871 ± 0.059  ops/us
   FloatDotProductBenchmark.dotProductFromMemorySegmentAligned      1024  thrpt    5  6.792 ± 0.119  ops/us
   FloatDotProductBenchmark.dotProductFromMemorySegmentUnaligned    1024  thrpt    5  6.789 ± 0.178  ops/us
   ```
   
   ```
   Linux rocket lake
   Benchmark                                                      (size)   Mode  Cnt   Score   Error   Units
   FloatDotProductBenchmark.dotProductCopyFromArrayAligned          1024  thrpt    5  10.513 ± 0.035  ops/us
   FloatDotProductBenchmark.dotProductCopyFromArrayUnaligned        1024  thrpt    5  10.389 ± 0.005  ops/us
   FloatDotProductBenchmark.dotProductFromMemorySegmentAligned      1024  thrpt    5  15.765 ± 0.022  ops/us
   FloatDotProductBenchmark.dotProductFromMemorySegmentUnaligned    1024  thrpt    5  14.877 ± 0.025  ops/us
   ```
   
   [1]: https://github.com/chrishegarty/vectorbench/tree/fromMemorySegment


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


[GitHub] [lucene] rmuir commented on issue #12482: Examine the potential of loading vector data directly from the memory segment

Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on issue #12482:
URL: https://github.com/apache/lucene/issues/12482#issuecomment-1660636776

   makes sense, a few concerns of mine:
   * boundary conditions: if the file is ginormous, we don't just map it to a single memory segment but multiple segments i think, how can any new api prevent this complexity from working its way into the vector formulas (which are already complicated enough, this would be too much)
   * alignment issues: afaik, with heap arrays there is some amount of alignment by the jvm, but with mapped files this may not be the case unless we do extra work. do we care? what is the penalty (on common arm/x86 cpus)?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org